A computer implemented method tracks 3D positions of an object moving in a scene. A sequence of images is acquired of the scene with a set of cameras such that each time instant a set of images are acquired of the scene, in which each image includes pixels. Each set of images is aggregated into a synthetic aperture image including the pixels, and the pixels in each the set of images are matched corresponding to multiple locations and multiple depths of a target window with an appearance model to determine scores for the multiple locations and multiple depths. A particular location and a particular depth having a maximal score is selected as the 3D position of the moving object.