For effectively overcoming the influence of rough background including point-clutter, strip wave and highlighted area in single-spectral forward-looking infrared (FLIR) image, a smoothing method for sea surface rough background based on multi-spectral FLIR images fusion is proposed. The method makes full use of the complementarity and difference existing in multi-spectral FLIR images. It aims at combining multiple images into a quality image in which the sea surface rough background is smoothed and the feature information of the ship targets is maintained good. Firstly, the multi-spectral source images were decomposed into low frequency sub-bands and high frequency sub-bands by discrete wavelet transform (DWT). The high frequency sub-band mainly contains the detailed information of the background and the ship target while the low frequency sub-band mainly contains the grayscale information. After obtaining the high frequency fusion image based on the maximum value of the high frequency coefficient, the regional energy of each pixel was calculated to modulate the high frequency fusion image in order to suppress the details of the background and maintain the details of the ship targets simultaneously. Then the low frequency fusion image was obtained by the average strategy and smoothed by the guided filter. Finally, the fusion image was reconstructed based on the high frequency fusion image and the low frequency fusion images by inverse wavelet transform. When the simulation experiment was carried out on the actually collected multi-spectral FLIR images to prove the effectiveness of the proposed method, the proposed method was compared with the other 6 smoothing methods including bilateral filter, guided filter, gradient minimization, relative total variation, bilateral texture filtering and rolling guidance filtering. A large number of experimental results show that the smoothing performance of the proposed method is better than the other 6 methods. The proposed method can effectively smooth the sea surface rough background and maintain the structure, grayscale, contrast of the ship targets, which can greatly enhance the separability of the ship targets. In the future work, the proposed method needs to be optimized to further improve the timeliness.