According to the transmission method of polarized light in Mueller ellipsometry method, this paper presented a method for calibrating the optical element parameters in ellipsometry system. By establishing a nonlinear least squares model of outgoing light intensity with respect to orientations of transmission axis of the polarizer and analyzer and orientations and retardation of the rotation compensator, the initial parameters were iterated with Levenberg-Marquardt (LM) algorithm. Accurate values of optical element parameters could be obtained, so as to achieve the calibration of components. Through the simulation experiment, using the SiO₂/Si standard sample with the known Mueller matrix and the calibration value of (24.90 ± 0.30) nm, the residual square sum of the light intensity value was calculated based on LM algorithm. When numbers of iterations accumulate to 50, the sum of squares converges of the residuals of the measurement and calculation was limited to 0.24. Then compared with the traditional multi-point calibration method, the feasibility of solving optical parameters based on LM algorithm was verified. Fitting results were validated by SiO₂/Si standard sample with calibration value of (91.21±0.36) nm. Calculated film thickness was 91.53 nm and the relative error is 0.35%. Results proved that LM algorithm has advantages of rapidly converging and high precision in the parameter calibration of the Mueller ellipsometry system.