The present study an improved fusion algorithm was proposed based on the Tetrolet transform. It was used to solve the problems that the infrared and visible light images fusion speed is slow, the contrast of the fused image is low and it is easy to bring artifacts to the fused image. First of all, the visible light image was converted to the lαβ color space to get three irrelevant color channels. Secondly, the component l and infrared image were decomposed by the Tetrolet transform. The neighborhood energy and proximity were introduced to the low-pass coefficients fusion rule. The Tetrolet coefficients were observed by the pseudo-random Fourier matrix. The observation value was weightedly fused. Thirdly, the fused observation value were iterated by the CoSaMP optimization algorithm to get the fused Tetrolet coefficient. The fused gray image was got after the Tetrolet reconstruction. Finally, the final fused image was obtained by mapping the grey image to the RGB color space. The experiment results testified the algorithm validity for the image fusion.