Calibration method and process for spatial modulating optical structure composed of quarter wave plate, composite birefringent wedge and polarizer was studied, and the calibrating equations of spectrum and radiation were presented. Based on the polarization principle of spatial modulation, linear least square method for calibration of polarization was provided. By simulation analysis, it is indicated that calibrating accuracy of linear least square method is higher than that of standard four-point method, the modulating coefficient deviation of linear least square method has the maximum of about 2×10-4, and moreover, distributes stochastically in the modulating period.