In this paper we propose a novel method for head tracking in large range using a scale invariant view-based appearance model. The proposed model is populated online, and it can select key frames while the head undergoes different motions in camera-near field. We propose a robust head detection algorithm to obtain accurate head region, which is used as the view of head, in each intensity image. When the head moves far from camera, the view of head is obtained through the proposed algorithm first, and then a key frame whose view of head is most similar to that of current frame is selected to recover the head pose of current frame by coordinate adjustment. In order to improve the efficiency of the tracking method, a searching algorithm is also proposed to select key frame. The proposed method was evaluated with a stereo camera and observed a robust pose recovery when the head has large motion, even when the movement along the Z axis was about 150 cm.