In this study, a classification method of chopped strand mat defects based on convolutional neural network is proposed. In the proposed method, the rotation, translation, and inversion are employed to expand the dataset for solving the overfitting problem caused by the small data samples in the deep convolutional neural networks. Transfer learning is employed to improve the convergence speed and generalization ability of the network. Further, the different network structures are compared, and the most optimal network structure is used to verify the database. The experimental results demonstrate that the proposed method can effectively classify the chopped strand mat defects with an accuracy rate of 93%.