In the optimization design of the flexible hinge in the fast steering mirror, in order to ensure the accuracy and stability, the rotational flexibility of the working axis and the rotational rigidity of the non-working axis should be increased as much as possible. Taking the elliptic arc flexible hinge in a fast steering mirror as the research object, firstly the least squares method and the integral method were used to derive the flexibility formula of three rotation axes. Compared with the Finite Element Analysis (FEA), the relative error was less than 6.8%, the problem of complex integration calculation in torsional flexibility of rectangular section was solved. Secondly, based on the improved non-dominated sorting genetic algorithm (NSGA-II) algorithm, the elliptic flexible hinge was multi-objectively optimized, met the design target and the optimal solution and solution were efficiency improved significantly. The efficiency of Pareto solution was improved by 14.3% compared with the Multi-Island Genetic Algorithm (MIGA) method, and improved by 25% compared with Particle Swarm Optimization (PSO). Finally, the optimization results were verified by FEA. The relative error was less than 6.5%, which was a good agreement.