Compared with the traditional multispectral imaging detection, polarized multispectral imaging detection can detect more information of the detected object surface such as roughness and moisture content, which brings great convenience to target detection. However, at present, it is mainly used for target detection and not widely used in target classification. BP neural network is a typical neural network commonly used at present. Neural network can establish the start-to-end mapping. On the premise that the training sample set is large enough, the trained neural network with good consequences is an efficient, accurate and high-speed tool. Firstly, the polarized multispectral images of typical ground objects were obtained by using the polarized multispectral imaging detection system based on rotating polarizer and filter, and after the images were preprocessed, the data set could be established; Secondly, the neural network was trained on this data set. The trained neural network could process the unknown polarized spectrum images and realize the classification of several typical ground objects; Finally, the effect of neural network classification was evaluated and compared with several other typical classification methods. It was found that the neural network method has better classification accuracy and effect. Compared with the typical maximum likelihood classification algorithm, its overall classification accuracy could be improved from 91.7% to 94.2%, and the Kappa coefficient could be improved from 0.851 to 0.898. The results show that the polarized multispectral image classification method based on neural network has certain research significance for improving and optimizing the existing data processing methods of polarized multispectral images.