A position controller for an optical element of an optical system includes a first movable member supported by a support member to be movable relative thereto in a first direction orthogonal to an optical axis, a second movable member which holds the optical element and is supported by the first movable member to be movable relative thereto in a second direction which is nonparallel to the first direction, an electromagnetic actuator that produces a driving force for moving at least the second movable member in the second direction, and an insertion/removal drive mechanism for moving the second movable member between an anti-shake driving position, at which a position control for the second movable member can be performed using the electromagnetic actuator, and a removed position at which the second movable member is positioned outside a range of the position control that uses the electromagnetic actuator.