Described herein are systems and methods for jointly learning geometry and motion with three-dimensional holistic understanding. In embodiments, such approaches enforce the inherent geometrical consistency during the learning process, yielding improved results for both tasks. In embodiments, three parallel networks are adopted to predict the camera motion (e.g., MotionNet), dense depth map (e.g., DepthNet), and per-pixel optical flow between consecutive frames (e.g., FlowNet), respectively. The information of 2D flow, camera pose, and depth maps, are fed into a holistic 3D motion parser (HMP) to disentangle and recover per-pixel 3D motion of both rigid background and moving objects. Various loss terms are formulated to jointly supervise the three networks. Embodiments of an efficient iterative training strategy are disclosed for better performance and more efficient convergence. Performance on depth estimation, optical flow estimation, odometry, moving object segmentation, and scene flow estimation demonstrates the effectiveness of the disclosed systems and methods.