A pattern inspection apparatus includes a first unit configured to acquire an optical image of pattern, a second unit configured to generate a reference image to be compared, a third unit configured to calculate elements of a normal matrix for a least-squares method for calculating a displacement amount displaced from a preliminary alignment position, a forth unit configured to estimate a type of the reference image pattern, by using some of the elements of the normal matrix, a fifth unit configured to calculate the displacement amount based on the least-squares method, by using a normal matrix obtained by deleting predetermined elements depending upon the type of the pattern, a sixth unit configured to correct an alignment position between the optical image and the reference image to a position displaced by the displacement amount, and a seventh unit configured to compare the optical image and the reference image.