In order to meet the requirements of miniaturization, compact structure and high resolution of the space optical systems in the fields of Earth remote sensing observation and spaceborne Lidar detection, this paper designs a compact off-axis triple inverse system based on Zernike free-form surface, which simultaneously meets the requirements of long focal length, small distortion and wide working band. The system adopts an off-axis triple inverse optical system with the asymmetric and nearly circular layout, and the third mirror of the system adopts the free-form surface design. By setting appropriate optimization objectives and methods in Zemax software, the design of the optical system is optimized. Finally, the effective focal length of the system is 800 mm, the F-number is 4, the field of view is 12°×6°, and the distortion is less than 1%. The working band covers the visible and near/middle infrared bands, and the ground element resolutions of 1.5 m (visible light) and 2.5 m (near infrared) can be achieved at the orbit height of 400 km, and the ground width is 80 km×40 km. The analysis and verification of system aberration, dot plot, MTF and other performance indexes are carried out. The results show that the design scheme brings the high resolution and improves the information acquisition ability.