This study proposes a quantitative analysis method combining sensitivity dimensionality reduction and Bayesian optimization algorithm support vector regression (BOA-SVR) to improve quantitative analysis accuracy of soil elements. The X-ray fluorescence (XRF) spectrum of the soil is obtained using a portable XRF analyzer, and the background is subtracted by iterative discrete wavelet transform. Furthermore, the calculated net peak area of each element is used as the model input feature. The influence of different input feature sets on the prediction accuracy is studied using sensitivity analysis to achieve feature dimensionality reduction. The samples are divided into training and test sets, and prediction accuracy of the model is evaluated using the root mean square error and coefficient of determination. Based on Cu and As elements, the prediction results of the BOA-SVR model under full feature input, the BOA-SVR model after feature dimension reduction, and the single-parameter partial least squares model are compared. The experimental results show that BOA-SVR model after feature dimension reduction achieves the best prediction result in both Cu and As elements.