Techniques are described for optimizing deep learning model performance using image harmonization as a pre-processing step. According to an embodiment, a method comprises decomposing, by a system operatively coupled to a processor, an input image into sub-images. The method further comprises harmonizing the sub-images with corresponding reference sub-images of at least one reference image based on two or more different statistical values respectively calculated for the sub-images and the corresponding reference-sub images, resulting in transformation of the sub-images into modified sub-images images. In some implementations, the modified sub-images can be combined into a harmonized image having a more similar appearance to the at least one reference image relative to the input image. In other implementations, harmonized images and/or modified sub-images generated using these techniques can be used as ground-truth training samples for training one or more deep learning model to transform input images with appearance variations into harmonized images.