An image sensor may include an array of pixels. Each pixel may have one or more photosensitive areas. Incident light may pass through a microlens and a color filter element to reach the one or more photosensitive areas. In some pixels, the microlens may be shifted relative to the photosensitive area to account for the chief ray angle of the incident light. In some cases, the microlenses may be toroidal microlenses with openings. The openings may be shifted relative to the center of the microlens based on the wavelength of light transmitted by the color filter element for the pixel. The greater the wavelength of light transmitted by the color filter element for a given pixel, the more the opening of the microlens may be shifted towards the center of the array of pixels.