The technology of fusing visible images and infrared thermal images is one of the most important researching fields in terms of high-performance night vision technology home and abroad Using this kind of technology can efficiently improve people’s ability of target-detecting and scene’s comprehension. The commonly used color transfer algorithm belongs to single reference image’s global color transfer, which results in huge influence on the color of fusion image. Thus, this kind of method cannot adapt to different kinds of scenes. In this essay, we focus on environmental adaptation problems of regular color transfer in color image fusion algorithm base on YUV color space. After analyzing the mean and standard deviation of typical scenes’ UV channel, we find that it has obvious classified characteristics. As a result, we propose a method of constructing adaptive reference image based on mean and standard deviation of U and V channels, which makes visible and infrared thermal image fusion algorithm better adapt to environment and fusion images’ color is natural. Furthermore, the handling capacity is small and it exerts little impact on the speed of algorithm real-time hardware processing. To conclude, it is a kind of practical natural color fusion processing algorithm.