A defect pixel value estimation means estimates a pixel value which each pixel in a defect region as a region to be restored in an image may take, based on the pixel value of pixels in a non-defect region as a region in the image not including the defect region. A patch selection means selects a pair of patches in which a defect patch and a reference patch are most similar to each other from pairs of patches including the defect patch as the image of a region including the defect region and the reference patch as the image of a region not including the defect region. The patch selection means selects a pair of patches in which the image of the defect patch and the image of the reference patch are most similar to each other based on a relationship between the pixel value of the defect region estimated in the defect patch and the pixel value of the corresponding reference patch. An image restoration means restores the defect patch based on the reference patch in the selected pair of patches.