In terms of the application of spectroscopy in-situ for soil quality monitoring from grassland, this paper takes the soil spectrum of Hulunbeier’s typical grassland as the research object. Verification by indoor simulated spectroscopy experiment and field spectrum measurement, and reveal the influence of plant litter on soil spectrum by analyzing the characteristics of mixed spectra. The blind source separation (BSS) independent component analysis (ICA) algorithm is used to separate the mixed spectra. Furthermore, spectral similarity value (SSV) is calculated to optimize BSS- ICA for unmixing soil spectra. The accuracy of the SOC prediction model before and after unmixing is compared to valid applicability of BSS-ICA algorithm. The results show that, (1) the cellulose absorption index (CAI) based on the characteristics of mixed spectra could effectively detect the extent of plant litter cover in the mixed spectra. CAI index would increase with the increasing of plant litter cover in quadratic regression; (2) It is found that a steep slope occurs at the transition band of 700 nm and weak lignin absorption characteristics in 1 680 and 1 754 nm, strong cellulose absorption occurs at 2 100 nm from mixed spectra; The SOC would be overestimated by about 11.94% using SVM prediction model once soil surface covered by only 5% plant litter. (3) The unmixing method of BSS-ICA can reduce the spectral characteristic from plant litter effect, and using partial least squares (PLSR), support vector machine (SVM) and random forest (RF) to model the prediction of organic carbon before and after unmixing. SVM has the highest accuracy among the three methods. The accuracy of SOC prediction was improved from R2 of 0.71 before unmixing to 0.75 after unmixing, RMSE of 4.82 g·kg-1 before unmixing to 4.50 g·kg-1 after unmixing. The optimized BSS-ICA algorithm can effectively separate soil from mixed spectra with litter and might improve the accuracy of SOC estimation by field spectra. This experimental study of reducing the external factors on soil spectra provides a theoretical basis for SOC prediction based on in-situ measurement of soil spectra.