An image processing apparatus that performs filtering by reading out an image from an external storage unit, the image being divided into a plurality of banks by a first interleave method according to a transfer length when the image is read out from the external storage unit is provided. The apparatus including: a plurality of local memories; and a control unit configured to divide, into a plurality of pixel fragments, a pixel of a bank which includes at least one of a plurality of pixels needed for the filtering by a second interleave method according to the transfer length and store each of the pixel fragments obtained as a result of division in one of the plurality of local memories in accordance with the transfer length.