Non-uniformity correction based on blackbody calibration is widely used in staring infrared imaging system. But in some cases, blackbody calibration method is not applicable. Two kinds of cases were presented: when the IR imaging system facing to the sky background, or observing normal objects through high temperature optical window in high speed aircraft. Based on the analysis of the response characteristics of infrared imaging system, the concept of non-uniformity with response spectral was proposed. It was pointed out that the correction results of blackbody calibration method depended on the spectral distribution of scene. The principle of non-uniformity with response spectrum was analyzed, which included internal and external reasons. The intrinsic factor was that the quantum efficiency of different pixels of infrared detector varies with spectrum, and the external factor was that the radiation spectrum distribution of application scene was quite different from blackbody. Suggestions were given to solve response spectral non-uniformity.