Using visible/near infrared spectroscopy to model soil properties is very important in current soil sensing research. It can be applied to rapidly access soil information and precision management. In the present study, paddy soil in Zhejiang Province is treated as the research samples. The nonlinear models such as random forests (RF), supported vector machines (SVM) and artificial neural networks (ANN) were used respectively to build models to predict soil organic matter based on different selection of calibration and validation datasets. The results show that there is a certain impact on prediction results under the division of different sample modes. Compared to the commonly used linear model PLSR, the nonlinear model RF and SVM have comparable prediction accuracy, especially predictions by SVM using all Vis-NIR wavelengths produced the smallest RMSE values. It shows that the model constructed by SVM method has a good predictive ability. In addition, a combined method, PLSR-ANN (with the introduction of ANN into PLSR), significantly improves the predictive ability of PLSR. Even though ANNs are “black box” systems the combination of PLSR and nonliner modelling helps achieve good predictions and interpretability.