A focus detection device includes: an imaging unit having a first and second pixel each of which receives light transmitted through an optical system and outputs signal used for focus detection, and a third pixel which receives light transmitted through the optical system and outputs signal used for image generation; an input unit to which information regarding the optical system is input; a selection unit that selects one of the first and second pixel based on the information to the input unit; a readout unit that reads out the signal from one of the first and second pixel based on a selection result at a timing different from reading out the signal from the third pixel to be read out; and a focus detection unit that performs the focus detection based on at least one of the signals of the first and second pixel read out by the readout unit.