One important source of information in scene understanding is given by actions performed either by human actors or robots. In this paper an approach to recognition and low-level interpretation of actions is presented. Since actions are characterized by specific motion patterns of moving objects, recognition is done by detecting such motion patterns as specific constellations of interactions between moving objects. First of all, motion detection and tracking algorithms are applied to extract correspondences between moving objects in consecutive images of a sequence. Subsequently these are represented with a graph data-structure for further analysis. To detect interactions of moving objects robustly a short history of motion of objects is traced using a finite-state automaton. Finally activities are segmented based on detected interactions. Since robust motion data are required consistency checks and corrections of the acquired motion data are performed in parallel.