An environment recognition system includes image taking means for taking a pair of images of an object in a surrounding environment with a pair of cameras and outputting the pair of images, stereo matching means for conducting stereo matching on a plurality of pairs of images that are taken by different image taking methods or that are formed by subjecting the pair of taken images to different image processing methods and forming distance images respectively for the pairs of images, selection means for dividing the distance images into a plurality of sections, calculating representative parallaxes respectively for the sections, and selecting any of the representative parallaxes of the corresponding section as a representative parallax of the section, and detection means for detecting the object in the image on the basis of the representative parallaxes of the sections.