A semiconductor defect inspection apparatus using a method of comparing an inspected image with a reference image includes the following: (1) a light source and an illuminating optical system, (2) plural defect optical imaging systems and photo detectors for scattered light detection, (3) a substrate holder and a stage for a scan, (4) means for obtaining the misalignment information on an adjacent die image using the inspection image of a defect optical imaging system with the highest spatial resolution, and means for transmitting the misalignment information to all the defect inspection image processing units, (5) means for correcting misalignment information so that a design and adjustment condition of each optical imaging system may be suited, and means for calculating a difference image between dies based on the corrected misalignment information, and (6) a defect detection and image processing unit for performing defect determination and detection processing based on the difference image between the dies.