In order to get the most mosaic effect, the transformation matrix between adjacent images be studied for solving the problem, a robust image mosaic algorithm is presented. In this algorithm, SIFT is used to extract feature points. Then, a pseudo-matching set is obtained by comparing local neighborhoods of features through intensity cross correlation method and these pseudo matches are divided into inliers and outliers using robust error threshold set to false matches. In the inliers, the use of the smallest squares optimization algorithm to accurately estimate the point transformation matrix between the two images. In the end, the image color of the overlapping band is smoothed with bilinear interpolation technique. The whole algorithm filters the noisy or wrong input data iteratively, estimates the model parameters through pure data, so it has strong error-tolerant capacity for the image noise and inaccuracy of feature extraction. When estimating the model parameters, the energy function is constructed based on the position errors of features instead of the features’intensity errors, which conquers the original methods’shortcoming of sensitivity to illuminating conditions and makes this algorithm more practical. Experimental results show that the image mosaic effect is wonderful and the algorithm is stable very much. It is high valuable in practice.