An adaptive total least square algorithm was proposed to further improve the attitude calculation accuracy of optical characteristic points, and the total least square algorithm and adaptive total least square algorithm were researched and compared. Firstly , the coordinate system and 3D model among optical characteristic points , image points and the position of camera were established according to the space position relationship , and the matrix equation between optical characteristic points and image points was created. Then, the two algorithms mentioned above were used to get an optimization solution. Finally , the attitude of optical characteristic points relative to the word coordinate system was obtained based on the optimization solution. The simulation comparison experiment was performed in a coordinate measuring machine,and the experimental result indicates that the standard tolerance of attitude coordinate calculated by total least squares is 0.055 7 mm, and that by adaptive total least squares is 0.041 4 mm. The compared results show that the adaptive total least square algorithm has higher convergence rate and precision , and its calculation speed is better than that of total least square algorithm. It satisfies the system requirements for stabilization, reliability and high precision.