The invention relates to a system (100) for classifying image data on the basis of a model for adapting to an object in the image data, the system comprising a segmentation unit (110) for segmenting the image data by adapting the model to the object in the image data and a classification unit (120) for assigning a class to the image data on the basis of the model adapted to the object in the image data, thereby classifying the image data, wherein the classification unit (120) comprises an attribute unit (122) for computing a value of an attribute of the model on the basis of the model adapted to the object in the image data, and wherein the assigned class is based on the computed value of the attribute. Thus, the system (100) of the invention is capable of classifying the image data without any user input. All inputs required for classifying the image data 10 constitute a model for adapting to an object in the image data. A person skilled in the art will understand however that in some embodiments of the system (100), a limited number of user inputs may be enabled to let the user influence and control the system and the classification process.