An effective and flexible rotation and compensation scheme is designed to improve the accuracy of rotating inertial navigation system (RINS). The accuracy of single-axial RINS is limited by the errors on the rotating axis. A novel inertial measurement unit (IMU) scheme with error compensation for the rotating axis of fiber optic gyros (FOG) RINS is presented. In the scheme, two couples of inertial sensors with similar error characteristics are mounted oppositely on the rotating axes to compensate the sensors error. Without any change for the rotation cycle, this scheme improves the system’s precision and reliability, and also offers the redundancy for the system. The results of 36 h navigation simulation prove that the accuracy of the system is improved notably compared with normal strapdown INS, besides the heading accuracy is increased by 3 times compared with single-axial RINS, and the position accuracy is improved by 1 order of magnitude.