Aiming at the precision requirement of machining and assembling for the SM system of the large-aperture grounded telescope, a design method for the structure of SM system based on topology optimization was presented. Based on the variable density idea of topology optimization, the ground structure was restricted to be the Spider and Serrurie trusses, the target was that the deformation along the expected direction was minimized, and the last appearance and dimension were described by whether the material was deleted from the ground structure or not. Firstly, the relative density was taken as design variable, and then by using 1st modal and deformation in gravity direction for the Spider and deformation along the X and Y axis for the truss as a design restraint, a topology optimization model was established. Secondly, on the basis of concept structure obtained by topology optimization, it was iterated and optimized by using Workbench. Finally, the optimized parameter was set, and the static and dynamic stiffness was analyzed and optimized using the finite element method. It shows that the 1st modal in the 4 m telescope SM system is 22.7 Hz, and the deviation in the direction of the gravity is -0.173 mm and -0.195 mm when the optical axis is parallel and vertical to the gravity direction, and the lightweight ratio exceeds 30% for the Spider and Serrurier truss, respectively. The results demonstrate the validity of the proposed approach properly.