The image processing device includes an imaging unit configured to capture a plurality of pupil-divided images obtained from light fluxes passing through different pupil regions of an imaging optical system, and a control unit configured to select images to be used for image processing from the pupil-divided images captured by the imaging unit, according to the position of the imaging area in the imaging unit.