The application of frequency modulation(FM) spectroscopy technology in the trace gas detection is greatly restricted by the residual amplitude modulation(RAM) which induced by the birefringence of the electro-optic modulator(EOM) and the misaligned linear polarization direction of laser to the EOM. Based on the interaction between the laser field and crystal, a lineshape expression of the FM spectroscopy with RAM is obtained. The angle between the polarization direction of input light and the principle axis of EOM, θ, the phase difference between the two principle axes of electro-optic crystal, Δ, and the modulation index FM, β, are the major factors to influence the lineshape. The larger θ and larger Δ are, the stronger distortion of the lineshape is. Meanwhile a DC offset exists in the FM dispersion spectroscopy which is influenced by θ and Δ. Finally a servo control of θ and Δ is suggested to reduce the RAM. These phenomenon and the analysis of the lineshape provide a necessary technical support for the fiber components based FM spectroscopy.