This study introduces a multiscale sliding window filter (M-SWF) image fusion method to address issues with traditional filter banks in infrared and visible image fusion. First, a multiscale image decomposition method based on SWF is proposed to extract the structural detail layers and base layers of the source image. Second, the L1 norm fusion rule (L1-Fusion, L1F) is used to integrate the structural detail layers, which can extract the structure of the image. Then, to highlight the salient objects, energy attribute fusion (EAF), which is a rule for fusing image energy contributions, is used to integrate the base layers, and the fusion results are obtained by stacking the integrated multiscale structure detail layers and base layers. The energy contribution coefficient was analyzed, and a suitable energy contribution coefficient was obtained for the fusion of infrared and visible images in the M-SWF domain from subjective and objective perspectives. Compared with other fusion methods, the M-SWF not only improves the ability to extract the structural information of the source image but also improves the poor fusion effect and effectively highlights salient targets by integrating the energy attributes of the image.