Medical ontologies are being developed with some of these specifically for mammographic computer aided diagnosis (CAD) systems. However, to provide full functionality for such mammographic CAD systems it is essential that the ontology information is fully linked to the image information. This linking can be through problem specific image attributes. However, such an approach tends to be non-generic. Here, we propose a framework that will use generic image structures and the topology that links the image structures. In the process we describe a comparison approach which takes the classes, attributes and semantics into account.