A vehicle control device includes a communication unit configured to communicate with a plurality of autonomous driving vehicles configured to perform autonomous traveling, and a processor. The processor is configured to, when an abnormality occurs in or around at least one first autonomous driving vehicle among the plurality of autonomous driving vehicles, determine a travel instruction for controlling traveling of the first autonomous driving vehicle, transmit the travel instruction to the first autonomous driving vehicle via the communication unit, set a priority representing the degree of the priority in which an instruction operator is notified of the travel instruction in the order determined according to the content of the abnormality, notify any one of at least one instruction terminal of the determined travel instruction in the order of highest priority, and receive a result of checking the determined travel instruction from the instruction terminal.