The subject disclosure is directed towards layered image understanding by which a layered scene representation is generated for an image. Providing such a scene representation explains a scene being presenting in the image by defining that scene's semantic structure. To generate the layered scene representation, the subject disclosure recognizes objects within the image by combining objects sampled from annotated image data and determining whether that combination is both semantically well-formed and matches the visual appearance of the image. The objects are transformed and then can be used to modify the query image. The subject disclosure models the objects into semantic segments that form a portion of the scene representation.