In a digital video surveillance system, a number of processing stages are employed to identify foreground regions representing moving objects in a video sequence. An object tracking stage (5) is also provided in order to identify a correspondence between candidate objects in a current frame and those that have already been identified in one or more previous frames. In this way, it is possible to calculate the path taken by the or each foreground object and to record this path information in a trajectory database. In order to improve tracking performance, the object tracking stage (5) employs a state transitional object management scheme which determines whether or not a particular object is tracked. The path information generated by the object tracking stage (5) can be displayed on a video monitor (15) of the surveillance system and/or made available to high level applications (7).