To improve the accuracy of frequency measurement in a polarization modulation range-finding system, a method based on dual-directional frequency sweep was proposed. First, the principle of the ranging method and the relationship between frequency and ranging stability were analyzed. Then, factors influencing frequency drift, as well as the changing law of drift in the system, were discussed, and it was proved that modulation depth and thermally induced additional phase delay are important factors affecting frequency drift. Accordingly, a compensation method was proposed to compensate for the frequency drift caused by additional thermal phase delay, which can be realized with low modulation depth. Experimental results show that, for the target at 15.23 m, the standard deviation of frequency measurement decreased from 3.822 9×104 Hz to 5.807 5×103 Hz, the ranging error decreased from 7.513 7 mm to 0.866 7 mm, and the validity of the method was verified.