A system and a method for detecting image forgery through a convolutional neural network are capable of detecting image manipulation of compressed and/or color images. The system comprises a manipulated feature pre-processing unit applying an input image to a high-pass filter to enhance features due to image forgery; a manipulated feature extraction unit extracting image manipulated feature information from the image with the enhanced features through a pre-trained convolutional neural network; a feature refining unit refining the extracted image manipulated feature information; and a manipulation classifying unit determining the image forgery based on the image manipulated feature information refined by the feature refining unit.