In recent decades, reflectance spectroscopy technology has developed rapidly and has been widely used in soil science, especially soil property estimation. It can greatly reduce the manpower and material resources consumed by traditional chemical measurement methods as an effective method to estimate total phosphorus content in soil. In this paper, 147 soil samples were collected from 30 sampling sites in coastal soil of Jiangsu Province, China. The spectral data and total phosphorus content of the soil were measured, respectively. Three different sample set partitioning methods were performed on the original spectral data and six different spectral transformation results, including Random Sampling (RS), Kennard-Stone (KS) and Sample Set Partitioning Based on Joint X-Y Distance Algorithm (SPXY). In order to compare and analyze the influence of three sample set partitioning methods on the accuracy of estimation results, partial least square regression(PLSR) and support vector machine(SVM) methods were used to establish the estimation models of total phosphorus content in the soil. The results are as follows: (1) Under the condition of original spectral data, the RS method can obtain better results and more stable model accuracy in most cases for PLSR, which is superior to KS and SPXY. In the SVM model, the result obtained by SPXY method is the best, KS is the second, RS is the worst. (2) The appropriate spectral transformation methods for different sample set partitioning methods are also different. Among the three sample set partitioning methods, the optimal spectral transformations of PLSR and SVM are respectively the reciprocal of logarithm and the first derivative (KS method), the original spectrum and the first derivative (RS method), the first derivative and multiple scattering correction (SPXY method). Using the KS method to divide the sample set, PLSR and SVM model can obtain the optimal prediction results. Not all spectral transformation methods can improve the model accuracy. The prediction accuracy of the PLSR model is significantly reduced after partial spectral transformation. (3)Among all sample set partitioning methods, SVM has a better modeling effect than PLSR. Using the RS method to divide the sample set, the prediction accuracy of PLSR is higher than that of SVM. The results were reversed when KS and SPXY were used. According to the comprehensive results, the best estimation model for the study area was obtained using the KS sample set partitioning method, and the first derivative transformation method, combined with the SVM method, the R² of the prediction result was 0.82. This study shows that reflectance spectroscopy can effectively predict the total phosphorus content of the soil in coastal areas and have a certain guiding significance for the efficient and rapid inversion of soil phosphorus.