A hearing aid system comprises a hearing aid, and a portable auxiliary device’ adapted to establish a communication link between them. The hearing aid comprises a microphone providing an electric input signal, a signal processor, and an output unit. The auxiliary device comprises a microphone providing an auxiliary electric input signal, and a user control interface allowing a user to initiate a specific calibration mode of operation of the hearing aid system. The signal processor of the hearing aid is configured to receive corresponding time segments of said electric input signal and said auxiliary electric input signal to provide an estimate of an acoustic transfer function from said microphone of said auxiliary device to said microphone of said hearing aid. A method of operating a hearing aid system is further disclosed. The invention may e.g. be used in various applications related to own voice detection and estimation.