Users with headsets may share an electronic device such as a portable computer or handheld device. The electronic device may have a connector such as an audio jack for receiving mating audio plugs on headsets. During normal operation with a single user, audio signals may be conveyed through the audio jack to the headset of the single user. When more than one user wishes to share the electronic device, an adapter accessory may be inserted into the audio jack of the electronic device. The headset of each user may be plugged into mating audio jacks in the adapter accessory. Circuitry in the adapter accessory may receive and process user input from each of the users. User input may be used to make local audio adjustments in the adapter accessory. User input may also be provided from the adapter accessory to the electronic device for processing.