A device receives images of a video stream, models for objects in the images, and physical property data for the objects, and maps the models and the physical property data to the objects in the images to generate augmented data sequences. The device applies different physical properties to the objects in the augmented data sequences to generate augmented data sequences with different applied physical properties, and trains a machine learning (ML) model based on the images to generate a first trained ML model. The device trains the ML model, based on the augmented data sequences with the different applied physical properties, to generate a second trained ML model, and compares the first trained ML model and the second trained ML model. The device determines whether the second trained ML model is optimized based on the comparison, and provides the second trained ML model when optimized.