A system, an apparatus or a process may be configured to implement an application that applies artificial intelligence and/or machine-learning techniques to predict an optimal course of action (or a subset of courses of action) for an autonomous vehicle system (e.g., one or more of a planner of an autonomous vehicle, a simulator, or a teleoperator) to undertake based on suboptimal autonomous vehicle performance and/or changes in detected sensor data (e.g., new buildings, landmarks, potholes, etc.). The application may determine a subset of trajectories based on a number of decisions and interactions when resolving an anomaly due to an event or condition. The application may use aggregated sensor data from multiple autonomous vehicles to assist in identifying events or conditions that might affect travel (e.g., using semantic scene classification). An optimal subset of trajectories may be formed based on recommendations responsive to semantic changes (e.g., road construction).