The six-degree-of-freedom displacement measurement technology based on position sensitive detector (PSD) and corner cube retroreflector plays an important role in the precise measurement of the initial position and attitude of the precision displacement stage in a compact space. Decoupling between the displacement of 6 degree of freedom (6DOF) displacement stage and the displacement of light spot on PSD is the key to realize precise measurement. In order to reduce the complexity of establishing and solving the model, the small angle approximation method or first-order Taylor series expansion can be used to transform nonlinear trigonometric function terms into linear terms, but this method is not universal enough to meet the needs of high-precision measurement in large ranges. In order to meet the demand of high-precision measurement under large angle, it is necessary to establish a more accurate theoretical model and solve it.Aiming at the six-degree-of-freedom displacement measurement system, the theoretical model is accurately described by trigonometric function rather than small angle approximation, and the analytical relationship between the six-degree-of-freedom displacement of 6DOF stage and the change of the spot position on PSD is derived, and a more accurate measurement model is established. When solving the model, the numerical calculation method is used to complete the model solution, which affords remarkably higher accuracy than the traditional small angle approximation method. In the model simulation, the calculation errors introduced by numerical calculation method and small angle approximation method under the single-degree-of-freedom displacement of 6DOF stage are compared. For six-degree-of-freedom displacement, Monte Carlo simulation is used to compare the accuracy of numerical calculation method and small angle approximation method.Through a 6×9 transformation matrix, the relationship between the displacement of 6DOF stage and the change of spot position on the three PSD can be established. Single-degree-of-freedom displacement will introduce calculation errors in all six degrees of freedom. For the translation displacement in the range of ±10 mm, the calculation error introduced by numerical calculation method and small angle approximation method can be ignored. For the rotational displacement in the range of ±10 mrad, the translational displacement errors introduced by the numerical calculation method are all less than 1.48×10^-16 mm, and the rotational displacement errors are all less than 1.73×10^-15 mrad, the maximum error is far less than the sub-micron accuracy requirements of the system. But the maximum error of the translational displacement calculation introduced by the small angle approximation method is 5.39 μm, which does not meet the sub-micron accuracy (Tab.2). For six-degree-of-freedom displacement, the translation displacement errors and rotation displacement errors obtained by the numerical method are less than 1.6×10^-14 mm and 1.1×10^-13 mrad, respectively, and the maximum error is much less than the accuracy requirement of the submicron level of the system. The accuracy of the numerical method depends on the number of iterations set by the computer and the error introduced by the computer in floating-point operation. However, the maximum error of translational displacement obtained by small angle approximation method is about 5.3 μm, which can not ensure the measurement accuracy of submicron level (Fig.6). Therefore, although the small angle approximation method is simple, its accuracy is much lower than that of the numerical method. Because the small angle approximation method provides iterative initial values for the numerical calculation method, the numerical calculation method has natural advantages in solving the displacement of six degrees of freedom. By using the numerical method, the number of iterations can be artificially set and the accuracy of the algorithm can be improved.Aiming at the six-degree-of-freedom displacement measurement system, a more accurate measurement model is established. The numerical calculation method has high accuracy, and the maximum error is far less than the sub-micron accuracy requirement of the system. The decoupling method in this paper is of great significance to the high-precision displacement measurement of 6DOF displacement stage with large rotational displacement.