The application relates to a binaural hearing assistance system comprising left and right hearing assistance devices adapted for being located at or in a left and right ear, respectively, of a user, each of the left and right hearing assistance devices comprising a) an input unit for providing one or more electric input signals based on one or more input sounds of a sound field surrounding the binaural hearing assistance system; b) a source separation unit for separating and localizing one or more sound sources Ss in said sound field relative to the input transducer unit based on said one or more electric input signals, and providing respective separated sound source signals Xs, and localization parameters LPs of said one or more sound sources (s=1, 2, . . . , Ns); c) an antenna and transceiver unit adapted for allowing an exchange of said electric input signals, and/or said separated sound source signals Xs and/or said localization parameters LPs of said one or more sound sources between said left and right hearing assistance devices. The application further relates to a method of operating a binaural hearing assistance system. The object of the present application is to provide an improved binaural hearing assistance system. The problem is solved in that the system further comprises d) a comparison and calculation unit for comparing said electric input signals, and/or said separated sound source signals Xsl, Xsr and/or said localization parameters LPsl, LPsr of said left and right hearing assistance devices, respectively, to estimate a head related transfer function HRTF value for one or more of said sound sources S with said localization parameters LPs at a given point in time; and e) a memory unit for storing and updating a database of said HRTF values and optionally said localization parameters LPs over time. This has the advantage of providing a flexible binaural hearing assistance system that is capable of learning the relevant HRTFs of its user, optionally without being initialized. The invention may e.g. be used in hearing aids, headsets, ear phones, active ear protection systems and combinations thereof.