A system and a method for cooperative maneuvering and cooperative risk warning of vehicles. The system includes monocular surveillance cameras, local computing devices, and a master server. Each local computing device receives video frames from the camera, detects and tracks vehicles from the video frames, and converts the video frames to bird-view. Each detected vehicle is represented by a detection vector having first dimensions representing two dimensional (2D) parameters of the vehicle and second dimensions representing three dimensional (3D) parameters of the vehicle. Tracking of the vehicles is performed by minimizing loss calculated based on the first dimensions and the second dimensions of the detection vectors. The master server receives bird-views from different computing devices and combines the bird-views into a global bird-view, and performs cooperative maneuvering and cooperative risk warning of the vehicles using the global bird-view.