There is provided an image processing apparatus. A focal position determining unit determines, for each of a plurality of captured image data, a focal position in a depth direction at a target position. The focal position determining unit determines a position at which a subject at the target position is in focus as the focal position if a first focus mode has been selected by a selecting unit, and determines the focal position for each of the plurality of captured image data such that the focal positions track a position at which a subject at the target position is in focus at a speed that is less than or equal to a threshold value, in an order of generation of the plurality of captured image data, if a second focus mode has been selected by the selecting unit.