An image processing apparatus includes a first phase difference detector configured to detect two phase differences in a range that contains a phase difference that provides the highest correlation between a pair of image signals, a comparator configured to compare a signal representative of a matching degree when the pair of image signals have a first phase difference among the two phase differences, and a signal representative of a matching degree when the pair of image signals have a second phase difference among the two phase differences, a signal separator configured to separate a pair of signal components relating to a specific object from the pair of image signals, based on a comparison result by the comparator, and a second phase difference detector configured to detect a phase difference that provides the highest correlation between the pair of signal components separated by the signal separator.