A method for calibrating a vehicular vision system includes providing a camera at a vehicle, with the camera having a field of view. Images are captured with the camera and a set of resultant images are acquired for a classification. Information is extracted related to image features in the set of resultant images, and an appropriate subset of coefficients is determined. For each classification, a classification vector of at least one appropriate weight is stored that corresponds to the determined subset of coefficients.