Outlier images—those images that differ substantially from other images in a set—can be automatically identified. One or more penalty values can be assigned to each image that quantifies how different that image is from others in the set. A threshold can be determined based on the set of penalty values. Each image whose penalty values are above the threshold is an outlier image. The penalty values can be the sum of per-pixel penalty values multiplied by the number of pixels with nonzero penalty values. A per-pixel penalty value can be the difference between a color value for that pixel and a predetermined range of color values, based on corresponding pixels in other images. The per-pixel penalty value can be determined for each component color and then optionally summed together. The threshold penalty values can be adjusted to provide for greater, or less, sensitivity to differences among the images.