An important goal of research in computer vision systems is to develop architectures which are general and robust and at the same time transparent and easily transferable from one domain to another. To this extent this paper discusses and demonstrates the versatility of a multi-object tracking framework based on the so called knowledge hierarchy. The systematic description and analysis of a priori knowledge provides means not only for reducing the complexity of the multi-object tracking problem but also for building modular systems for solving it. The modularity of the framework, an essential ingredient for versatility, allows to replace individual parts of an existing system without altering the rest of the system or the overall architecture. The paper presents the modular framework including the knowledge hierarchy for multi object tracking. In order to demonstrate the transferability of the proposed approach the tracking framework is then applied to three different tracking scenarios (parking lot surveillance, people interaction monitoring, and dining table setup).