A system for knowledge-based image computer aided detection includes a text interpretation system receiving an electronic patient record and outputting an assertion relevant for the electronic patient record, an annotation/detection system detects anatomical and functional structures in input images and interacts with the text interpretation system to receive the assertion and outputting annotated images based on the input images, and an imaging decision support system taking the annotated images and outputting classifications of annotated structures in the annotated images.