This paper presents an approach to extract the rigid parts of an observed articulated object. First, a spatio-temporal filtering in a video selects interest points that correspond to rigid parts. This selection is driven by the spatial relationships and the movement of the interest points. Then, a graph pyramid is built, guided by the orientation changes of the object parts in the scene. This leads to a decomposition of the scene into its rigid parts. Each vertex in the top level of the pyramid represents one rigid part in the scene.