The feature difference among the images of fine-grained vehicle models is small and there exist many factors disturbing recognition under complex image background. To improve the feature extraction ability and the recognition accuracy of images under complex background, a classifier named Softmax-SVM is proposed based on deep convolutional neural network (DCNN) and support vector machine(SVM), in which the cross-entropy cost function is combined with the hinge loss function to replace the Softmax function layer, so that the over-fitting is avoided. Meanwhile, a 10-layer DCNN is designed to extract features automatically and the problem of manual extraction of features is also avoided. The experimental dataset consists of the images of 27 types of fine-gained vehicle models under complex background, especially of the similar models from the same car manufacturer. The experimental results show that the Softmax-SVM classifier can be used to recognize the 269 sample images without much emphasis on the pre-processing stages, and in the identification process, the accuracy rate is 97.78% and the time to identity each image is 0.759 s. The above model performs more efficiently than the traditional recognition methods and the unimproved DCNN models. In consequence, the Softmax-SVM classifier based on DCNN can adapt to the complex changes of environment and give consideration to both the recognition accuracy and efficiency, which provides practical reference value in the classification field of fine-gained vehicle models under complex background.