Embodiments are disclosed that relate to modifying a display of a portable electronic device to account for ambient light. For example, one disclosed embodiment provides a method comprising determining an ambient light history including a plurality of ambient light color conditions sensed over a duration of device operation by an ambient light sensor, reading a display-ready image having a plurality of pixels from an image source, adjusting a manner in which the display-ready image is displayed by color-shifting at least a subset of the plurality of pixels based on the plurality of ambient light color conditions in the ambient light history to thereby generate a color-modified image, and displaying the color-modified image on the display of the portable electronic device.