Aiming at the problems that the main structure of a mini-optical device in space is too heavy, the ground gravity deformation is too large and the base frequency is too low, the optimization mathematical model was established with the objective of minimum mass and RMS of the random acceleration response, the fundamental frequency and the deformation as the constraint conditions. The topology optimization design of the main structure of the mini-optical device and the engineering analysis of the optimized main structure were carried out. The results show that the mass of the main structure of mini-optical device is 12.5 kg, which is reduced by 68.71%; The fundamental frequency is increased from 11.18 Hz to 268.7 Hz after optimization; The maximum deformation is 0.3 μm. The magnification of the acceleration response of the optical-load installation is 1.2 which is better than system specification 1.5. Mechanics and thermal experiments were carried out to examine the performance of the main structure of mini-optical device. The detection results meet the overall index, which proves the main structure has good performance, the optimization method is effective and feasible.