A signal processing apparatus is provided. The apparatus includes an obtaining unit configured to obtain direction sounds in respective directivity directions from audio signals picked up by a plurality of sound pickup units, and a control unit configured to control, in accordance with a frequency of the direction sounds obtained by the obtaining unit, a directivity direction count indicating the number of directivity directions corresponding to the direction sounds obtained by the obtaining unit.