To realize the horizontal positioning of a reticle stage with three degrees of freedom and high accuracy, it needs to establish an accurate multi-degree of freedom decoupling measurement model in the reticle-stage-measurement system. In this study, a two-dimensional diffraction planar grating is used to establish a reticle stage degree-of-freedom position measurement system, and the main reasons for the deviation in the position of the reticle stage are analyzed in terms of the multiple installation errors that are caused by the planar grating and read head. Firstly, the displacement model of the three degrees of freedom of the reticle stage is designed according to the installation layout; then, the installation errors of the planar grating and read head are combined to analyze the cause of the Abbe and cosine errors generated by the position of the reticle stage. A compensation algorithm is designed to reduce the position error of the reticle stage and then the model is simulated using MATLAB software, observing the convergence of coupling coefficient. Finally, a calibration error method is proposed. Using the dual-frequency laser interferometer measurement system as the benchmark and using the least squares method, the coupling coefficient of the degree of freedom of the planar grating position model is determined. Experimental results show that the proposed algorithm can effectively compensate the Abbe and cosine errors of the reticle stage; thus, the uncertainty in the measurement system reaches 5 nm.