This paper presents a framework to track multiple persons in realtime. First, a method with real-time and adaptable capability is proposed to extract face-like regions based on skin, motion, and silhouette features. Then, a two-stage face verification algorithm is proposed to quickly eliminate false faces based on the face geometries and the Support Vector Machine(SVM). In order to overcome the effect of lighting changes, a method of color constancy compensation is applied. Then, a robust tracking scheme is applied to track multiple persons based on a face-status table. With the table, the system has extreme capabilities to track different persons at different statuses, which is quite important in face-related applications. Experimental results show that the proposed method is much robust and powerful than other traditional methods.