Aspects of the disclosure provide for controlling an autonomous vehicle to respond to queuing behaviors at pickup or drop-off locations. As an example, a request to pick up or drop off a passenger at a location may be received. The location may be determined to likely have a queue for picking up and dropping off passengers. Based on sensor data received from a perception system, whether a queue exists at the location may be determined. Once it is determined that a queue exists, it may be determined whether to join the queue to avoid inconveniencing other road users. Based on the determination to join the queue, the vehicle may be controlled to join the queue.