The invention provides a face-based automatic demographics classification system that is robust to pose changes of the target faces and to accidental scene variables, by using a pose-independent facial image representation which comprises multiple pose-dependent facial appearance models. Given a sequence of people's faces in a scene, the two-dimensional variations are estimated and corrected using a novel machine learning based method. We estimate the three-dimensional pose of the people, using a machine learning based approach. The face tracking module keeps the identity of the person using geometric and appearance cues, where multiple appearance models are built based on the poses of the faces. Each separately built pose-dependent facial appearance model is fed to the demographics classifier, which is trained using only the faces having the corresponding pose. The classification scores from the set of pose-dependent classifiers are aggregated to determine the final face category, such as gender, age, and ethnicity.