To achieve the best structure performance of the airborne polarization imaging spectrometer lens which meets the optical performance, the optimization model based on the line of sight jitter error was established and calculated in Isight environment. Firstly, the principle that the line of sight jitter error affected the performance of the optical system was elaborated and the initial structure layout was designed according to the optical model and performance. Secondly, the calculation method of the line of sight jitter error was deduced, based on which the optimization model was established. Finally, the optimization model was calculated by applying the Multi-island Genetic Algorithm in Isight environment. The computational result of the integrated optimization model shows that the mass of the lens barrel is reduced by 17.4%, which satisfying the constraint of the optimization model and the value of the optical transfer function is 0.4(77 lp/mm), which meeting the requirement. The line of sight jitter error is applied during the procedure of the structure optimization presented, which expanded the connotation of optomechanic structure optimization and provided a new optimization idea for the same kind of structure.