A video analytic system includes a depth stream sensor, spatial analysis module, temporal analysis module, and analytics module. The spatial analysis module iteratively identifies objects of interest based on local maximum or minimum depth stream values within each frame, removes identified objects of interest, and repeats until all objects of interest have been identified. The temporal analysis module associates each object of interest in the current frame with an object of interest identified in a previous frame, wherein the temporal analysis module utilizes the association between current frame objects of interest and previous frame objects of interest to generate temporal features related to each object of interest. The analytics module detects events based on the received temporal features.