An image compression method includes compressing an input image with first and second compression methods to generate first and second (e.g., lossless and lossy) compressed images. First and second residual layers are generated, based on a difference between the first and second compressed images. Connected components in the residual layers are identified. Each connected component includes a group of one or more pixels that, when mapped to the second compressed image is connected, in first and second directions, to pixels in the second compressed image. A compressed image is generated, which includes, for a connected component identified in the first residual layer, removing corresponding pixels from the second compressed image, and for a connected component identified in the second residual layer, adding corresponding pixels to the second compressed image. The system and method can thus provide file size savings associated with lossy compression while avoiding character replacement.