In one embodiment, a method, apparatus, and system may predict behavior of environmental objects using machine learning at an autonomous driving vehicle (ADV). A data processing architecture comprising at least a first neural network and a second neural network is generated, the first and the second neural networks having been trained with a training data set. Behavior of one or more objects in the ADV's environment is predicted using the data processing architecture comprising the trained neural networks. Driving signals are generated based at least in part on the predicted behavior of the one or more objects in the ADV's environment to control operations of the ADV.