The roughness of the material surface affects the THz nondestructive testing results. Roughness of the surface can be ignored in the microwave region, but should be considered in the terahertz frequency domain. The effect of scattering caused by rough surface at terahertz frequency on the reflection spectrum was studied and discussed. By consideration of the reflection model of a single sample, the reflection signal of a rough surface can be correlated with that of a smooth surface by using the Kirchhoff approximation. In addition, the glucose tablets with different roughness were measured by THz-TDS (Terahertz time domain spectroscopy) system and their reflection spectra were analyzed. The reflection spectrum results show that due to the roughness of the surface, diffuse scattering generated by the rough surface weakens the intensity of the reflection spectrum at the receiver. In order to reduce the influence of the roughness on the spectrum, a spectral Gaussian compensation method was proposed, which can restore the spectral characteristics of the smooth surface. The power spectrum of 360 mesh roughness was increased by about 3 dB and 9 dB at 0.5 THz and 1 THz, respectively. Therefore, it can be envisaged that the proposed rough-surface spectral compensation method has a specific reference value in the development of the THz nondestructive testing technology in the future.