In order to study the spectral line shift property of prism-dispersive imaging spectrometer, the influencing factors and mechanisms of spectral line shift were presented, and the mathematical model based on linear optics model was established to describe the spectral line shift property. Code V API functions was used, in Matlab environment, to verify the validity of mathematical model, and the sensitivity coefficient of spectral line shift was analyzed. Results indicate that rigid body motion of optical mirror surface generated by environmental variation is the key causation of spectral line shift. When the decenter of mirror surface is no more than 0.2 mm and the tilt is less than 0.02°, the value of spectral line shift of different wavelengths at different fields is equivalent, and the error is less than 0.1 pixel. Spectral line shift due to mirror rigid body motion is linear and independent, and the total shift of the spectral line is the algebraic sum of values produced by the single freedom of motion (DOF) of single mirror surface. The mathematical model based on linear optics model can be used to study the spectral line shift property of the prism-dispersive imaging spectrometer. It will provide some guidance for spectral calibration and spectral property analysis under complex work condition.