A system for measuring a pulse and respiratory rate from passive thermal video includes contour segmentation and tracking, clustering of informative pixels of interests, and robust dominant frequency component estimation. Contour segmentation is used to locate a blood vessel region to measure, after which all pixels in the nearby region are aligned across frames based on the segmentation's position, and scale in each frame. Spatial filtering is then performed to remove noise not related to heart beat and then non-linear filtering is performed on the temporal signal corresponding to each aligned pixel. The signal spectrum of each pixel is then feed to a clustering algorithm for outlier removal. Pixels in the largest cluster are then used to vote for the dominant frequency, and the median of the dominant frequency is output as the pulse rate.