An apparatus for processing a defect candidate image, including: a scanning electron microscope for taking an enlarged image of a specimen by irradiating and scanning a converged electron beam onto the specimen and detecting charged particles emanated from the specimen by the irradiation; an image processor for processing the image taken by the scanning electron microscope to detect defect candidates on the specimen and classify the detected defect candidates into one of plural classes; a memory for storing output from the image processor including images of the detected defect candidates; and a display unit which displays information stored in the memory and an indicator, wherein the display unit displays a distribution of the detected and classified defect candidates in a map format by distinguishing by the classified class, and the display unit also displays an image of a defect candidate stored in the memory together with the map which is indicated on the map by the indicator.