A face recognition system based on adaptive learning includes a specific person detection and tracking unit for detecting and tracking a specific person from a moving image. A facial feature extraction unit extracts a plurality of facial feature vectors from the detected and tracked specific person. A face recognition unit searches for a given registration model by comparing the extracted facial feature vectors with facial feature vectors of the registration models previously stored in a user registration model database. A learning target selection unit selects a facial feature vector to be added to a record of the given registration model from among the extracted facial feature vectors. A registration model learning unit adds and updates the selected facial feature vector to the record of the given registration model.