Methods, systems, an apparatus, including computer programs encoded on a storage device, for tracking human movement in video images. A method includes obtaining a first image of a scene captured by a camera; identifying a bounding box around a human detected in the first image; determining a scale amount that corresponds to a size of the bounding box; obtaining a second image of the scene captured by the camera after the first image was captured; and detecting the human in the second image based on both the first image scaled by the scale amount and the second image scaled by the scale amount. Detecting the human in the second image can include identifying a second scaled bounding box around the human detected in the second image scaled by the scale amount.