Using a white diffuser to calibrate the imaging spectrometer has been a new technology developed to calibrate the imaging spectrometer recently. It has characteristics of easy realization and higher calibration accuracy. The expression of the collected signal electron number in each spectral channel of the imaging spectrometer detector pixels was deduced with the slit parallel and perpendicular to the meridional plane according to the principal of the calibration of imaging spectrometer using diffuser method in the present paper. The spectral radiometric calibration characteristics of the imaging spectrometer was numerically analyzed under the two special slit directions. The results indicate that the slit direction has significant effect on the spectral radiometric calibration of the imaging spectrometer. The signal electron number of the same spectral channel collected by the different scene pixels is different when the slit parallels to the meridional plane, and when the pixel is closer to the standard lamp, it collects more signal electrons; the signal electron number of the same spectral channel collected by the different scene pixels doesn’t change with the scene pixel position when the slit is perpendicular to the meridional plane.