In recent years, although the nutrient content in the soil can be quickly obtained with the emergence of hyperspectral technology, but different soil types have great differences in the accuracy of estimation. The soil type of coastal wetland is greatly affected by the marine environment, and its hyperspectral reflectance and inland soil type will be different. This will reduce the precision in the same estimation model when inverting the nutrient content of coastal wetland soil types. With the development of marine resources and the ecological restoration of coastal wetlands in recent years, it is urgent to explore a suitable estimation model to quickly and accurately obtain nutrient content in soil. This study aimed to verify the use of visible-near infrared hyperspectral reflectivity to construct a nonlinear model so as to invert the feasibility of organic matter (SOM) in coastal wetland soils. The topsoil in the third core area of Dafeng Elk National Nature Reserve in Yancheng, Jiangsu province was taken as the investigated object. The sensitive bands corresponding to Soil Organic Matter (SOM) content were retrieved based on correlation coefficient after 5 point S-G filtering and four differential transformations of R′, (1/R)′, (1/R)″, (lgR)′ by spectral reflectance of soil samples. The estimation models of SOM by Support Vector Machine (SVM) and BP neural network were determined, and the prediction accuracy of the model was verified by using the decision coefficient R2 and the root mean square error RMSE. The research results indicated that the effective bands can be identified by S-G filtering, differential transformation and correlation coefficient method based on the original spectra of soil samples. The characteristic bands of SOM based on transformations (1/R)′ were 498~501, 1 180~1 182, 1 946, 1 947, 2 323～2 326 nm. Estimation accuracy of SVM was better than that of BP neural network for SOM in Yancheng coastal wetland. The estimation model of SOM by SVM based on (1/R)′ spectra had the highest precision, with the determination coefficients (R2) and root mean square error (RMSE) of 0.93 and 0.23. Therefore, it is suitable to use hyperspectral remote sensing to quickly estimate the nutrient contents of topsoil in coastal wetland.