For the shortcomings of backlash shaft, friction and clearances in coarse tracking system of satellite optical communication, a scheme of using a flexible shaft to support the terminal was presented. The principle of flexible shaft in satellite optical communication was presented. Then the characteristics of flexibility in working direction and stress in thinnest incision were analyzed. According to the key point deflection, the rotation error angle expression was derived and its influence on capture probability was analyzed. Based on the research of flexible shaft characteristics, the flexible hinge was optimized by genetic algorithm. The results are as follows: the flexibility in working direction is 0.768 rad/N·m, the stress of thinnest incision is 1.768 e+8 Pa and the rotary error angle is 343 μrad. Finally, the finite element method was utilized to verify the design consequences, the relative error of the three indexes was less than 3.5%, which proved that the design was reliable. This study can provide some reference for flexible shaft design in satellite optical communication terminal.