A system and method are provided for enabling a user to annotate a medical image. A collection of key-value pairs is provided, in which a key represents an image-observable quantity and a value represents the value of the image-observable quantity. A collection of structured finding objects is provided, wherein each structured finding object represents a set of key-value pairs, each set of key-value pairs representing a different annotation of the medical image. The user is enabled to select one or more of the collection of key-value pairs, thereby obtaining a user-selected structured finding object which represents a preliminary annotation of the medical image by the user. At least one recommended structured finding object is selected by using the user-selected structured finding object as input to a probabilistic recommendation algorithm. Feedback is provided to the user on the basis of the recommended structured finding object. The annotation is well suited for, e.g., pointer-based selection via a graphical user interface, speech recognition, etc. Moreover, machine interpretability may be improved compared to conventional string-based annotation.