In order to solve the problems of low accuracy, complex optical machine structure, and long detection cycle in the error detection of the photoelectric encoder, the optical small-angle measurement principle of an autocollimator and a polyhedral prism, and the continuous error detection method of the reciprocal rotation angle of the dual-axis turntable were combined to establish an error detection system for photoelectric encoder based on optical continuous closed-loop. By using the multi-body system theory and the relative pose matrix transformation method, a full error model of the dual-axis turntable was established, and the influence of the fixed error and variable error of the full error model on the system was analyzed. The detection system was calibrated with a calibrated autocollimator and a 23-sided polyhedral prism, and a high-precision photoelectric encoder was used to compare the detection accuracy with the system. The test results show that the rotation accuracy of the dual-axis had met the requirements of the numerical simulation calculations, the system detection accuracy had reached 0.38″, and the measurement uncertainty is 0.2″ (k=2). And for the encoder actually produced, the detection accuracy of the system is basically the same as the accuracy of the factory calibration. Which had been verified the feasibility of the optical continuous closed-loop system to achieve high precision and circumference continuous error detection.