In this paper, a new approach for tracking rigid objects is presented. It builds on and extends on SVD method and Robust estimation techniques. Because of the presence of the difference of the gray value and the distortion between the object image and the reference image, traditional matching method will be degraded. We need more useful characters. Eigenspace approach is a promising candidate for an appearance-based object representation. In this paper, the tested object images are decomposed first, the eigen spaces of the images are obtained, then they are applied as the characters of the object images in the matching process. Another point in this paper is the distance measurement. We think that the least-squares method has a number of problems, so a robust estimation is adopted to make the similarity measurement. These techniques are used to track objects over long image sequences, the experimental results show that this image matching method could bear affine image motion and image distortion to some extent and it is very promising.