The current qualitative classification models of soil heavy metal content based hyperspectral remote sensing technology mostly use indoor measured spectral data from the same area for model training and testing. However, the indoor spectrum measurement requires a complicated processing process with high cost and low efficiency, and thus cannot obtain the spatially continuous spectral information in the target area quickly. Moreover, whether this kind of model can be transferred to the outdoor measured spectral data in different test areas is still unclear. In order to answer this question, two lead-zinc mining areas in Chenzhou City and Hengyang City of Hunan Province were selected as research areas. Support Vector Machine was used as classifier. Then 83 sample data from indoor sampling in Zhangzhou experimental area and 46 sample data from indoor sampling in Hengyang experimental area were used for classifier training, and 46 sample data from field sampling in Hengyang area were used for classification testing. The difference of spectral distribution between the indoor and outdoor measured spectral data was reduced by the transfer learning method based on joint distribution adaptation (JDA), and then the domain adaption model for two research areas was constructed. The experimental results show that:(1) The spectral data measured by outdoor samples may be affected by factors such as solar radiation and differences in extracted soil components, leading to the significantly spectral difference for indoor and outdoor samples. As a result, it is difficult to directly transfer the qualitative classification model of soil heavy metal pollution trained by indoor samples to the outdoor samples from the same area. However, after the reduction of indoor and outdoor distribution differences by JDA transformation, the transfer ability of the model has been significantly improved, and the classification accuracy of three heavy metals As, Pb and Zn has reached over 84%. The accuracy of classification of Zn elements exceeding the standard even reached 89%. (2) Due to seasonal influences, regional component interference, and spectral noise, there are even more significant differences in the distribution of spectral data in different areas. This further increases the difficulty of soil heavy metal pollution monitoring in different areas, and it is difficult to directly transfer the qualitative classification model of soil heavy metals based on indoor sampling spectral data to field sampling data in other areas (with an average classification accuracy of about 50%). After the indoor and outdoor spectral transformation processing by JDA, the transfer ability of the model has been greatly improved. Therefore, the outdoor spectral sampled can be directly used to investigate the pollution situation of heavy metals (As, Pb and Zn) in different test areas.