A focus adjustment apparatus includes an object detection unit configured to detect feature information of an object, a plurality of signal generation units configured to accumulate charges to generate and output a pair of image signals respectively, and a focus detection unit configured to control charge storage of the plurality of signal generation units and detect a focus based on a phase difference between a pair of image signals output from a first signal generation unit corresponding to the feature information of the object detected by the object detection unit, wherein if reliability of detection by the object detection unit is a first level which is higher than a second level, the focus detection unit sets charge storage time of the first signal generation unit longer than that in a case of the second level.