An audio signal processing device includes a processing unit for convoluting head-related transfer functions with audio signals of a plurality of channels, and the processing unit includes a storage unit for storing data of a double-normalized head-related transfer function by normalizing a normalized head-related transfer function obtained by normalizing a head-related transfer function in a state in which a dummy head or a person is present in a position of the listener with a transfer characteristic in a pristine state in which the dummy head or the person is not present, using a normalized head-related transfer function obtained by normalizing a head-related transfer function in the state in which the dummy head or the person is present with a transfer characteristic in the pristine state, and a convolution unit for reading the data from the storage unit and convoluting the data with the audio signals.