Based on Mente Carlo method, the modulating effect of the target surface roughness on the optical heterodyne signal was discussed. The quantitative analysis of the decoherence effect caused by the roughness of targets was investigated. The relationship between the optical heterodyne signal and rough parameters of the target was given. It is shown that when the correlation length is 100λ, the optical heterodyne signal increases with the height of the root mean square fluctuation decreasing. When the height of the root mean square fluctuation exceeds 0.2λ, the optical heterodyne signal is very weak. In addition, when the height of the root mean square is 0.1λ, the optical heterodyne signal decreases with the correlation length decreasing. It seriously affects the optical heterodyne detection of a rough target.