Many traditional feature point algorithms can not handle more complex nonlinear brightness changes because the gray between multi-source remote sensing images is nonlinear changes. To cover the shortage, a Scale Invariant Feature Transform(SIFT) feature matching algorithm of multi-source remote sensing images was proposed. First, the approximate linear gray value between multi-source remote sensing images was achieved through linear fitting of the bands of the images. Then, an improved SIFT algorithm was adopted to match the fitted remote sensing images. Finally, the random sample Consensus algorithm was used to remove the false matching point pairs. In comparison with other feature matching algorithms (SIFT, Gradient Location Orientation Hologram(GLOH), RS-SIFT). The experimental results show that the feature matching rate increases by about 4% between ETM+ panchromatic and multispectral images and the number of correct matches of key points increases by about 8 point pairs between CBERS-02B and HJ-1B images. It concludes that the proposed method significantly outperforms many state-of-the-art methods under multi-source remote sensing images.