Systems and methods of training vehicles to improve performance, reliability and autonomy of vehicles. Sensors capture a vehicle driver's eye movements, hand grip and contact area on steering wheel, distance of the accelerator and brake pedals from the foot, depression of these pedals by the foot, and ambient sounds. Data from these sensors is used as training data to improve autonomy of vehicles. Detection of events occurring outside a vehicle is made by comparing human sensor data with an associated road map to determine correlation. Signatures corresponding to human reactions and actions are extracted, which also incorporate data acquired using vehicle and outside environment sensors. By scoring driver responses to events outside the vehicle as well as during non-events, expert drivers are identified, whose responses are used as a part of vehicle training data.