This paper mainly researches how to apply the distance information of millimeter wave radars to the object-tracking of infrared image sequences. In connection with the image sequences whose objects are gradually magnified, a template updating strategy in which the template updating cycle is confirmed by the distance between target and missile is proposed after analyzing the effect of image magnification on the correlation characteristics. In combination of the timing updating, template buffer updating with the distance updating, an adaptive template updating algorithm for fusing distance information is presented. By taking a building as experimental target, the effectiveness of the method is verified. It demonstrates that the method is very suitable for the proposed image sequences. As compared with the traditional tracking methods,it has improved the tracking accuracy and tracking stability greatly and its tracking accurateness is more than 90% for different images.