An imaging element includes pixel portions each having a first sub-pixel and a second sub-pixel and outputs a phase difference detection-type focus detecting signal. An image signal A includes information for the first sub-pixel and an image signal AB includes information for the second sub-pixel. A level determining unit compares the image signal A with a threshold value SHA and compares the image signal AB with a threshold value SHAB. A correlation calculation processing unit performs correlation calculation for a signal excluding the image signal A having a level exceeding the threshold value SHA and the image signal AB having a level exceeding the threshold value SHAB so as to output the result of calculation to a CPU. The CPU calculates a focal shift amount in accordance with the result of calculation and performs a focus adjusting operation by drive-controlling a focus lens.