An image processing apparatus includes a data storing portion configured to store coefficient data for reconstructing an optical transfer function of an image pickup optical system in accordance with a type of the image pickup optical system and an imaging condition, a tap number determining portion configured to determine a tap number of the optical transfer function that is reconstructed by using the coefficient data in accordance with a size of one pixel of an image pickup element, and a reconstruction portion configured to reconstruct the optical transfer function in accordance with Nyquist frequency of the image pickup element and the tap number in a frequency space.