In this paper, we present a stereo-based face tracking system which can track the 3D position and orientation of a user in real-time, and the system’s application for interaction with a large display. Our tracking system incorporates dynamic update of template images for tracking facial features so that the system can successfully track a user’s face for a large angle of rotation. Another advantage of our tracking system is that it does not require a user to manually initialize the tracking process, which is critical for natural and intuitive interaction. Based on our face tracking system, we have implemented several prototype applications which change information shown on a large display adaptively according to the location looked at by a user.