As an object moves through space, it changes its orientation relative to the viewing camera and relative to light sources which illuminate it. As a consequence, the images of the object produced by the viewing camera may change dramatically. Thus, to successfully track a moving object, image changes due to varying pose and illumination must be accounted for. In this paper, we develop a method for object tracking that can not only accommodate large changes in object pose and illumination, but can recover these parameters as well. To do this, we separately model the image variation of the object produced by changes in pose and illumination. To track the object through each image in the sequences, we then locally search the models to find the best match, recovering the object’s orientation and illumination in the process. Throughout, we present experimental results, achieved in real-time, demonstrating the effectiveness of our methods.