In view of the serious impact and threat to public security of UAV''s undocumented flight and random flight, an anti UAV system was proposed. Recognition of UAV is one of the key points in the realization of anti UAV system. An image recognition method based on convolutional neural network was proposed. The self-made optical system was used to collect images of different UAVs and birds, and convolutional neural network and support vector machine for UAV small sample recognition were designed. The convolution neural network was used to identify MNIST data set, UAV image and bird image respectively. At the same time, support vector machine was used to identify UAV and bird image, and the experiment was carried out. The experimental results show that the recognition accuracy of the convolutional neural network is 91.3% in MNIST data set, 95.9% in UAV recognition and 88.4% in support vector machine (SVM). The experimental results show that the proposed method can identify UAVs, birds and different types of UAVs, and the recognition result is better than that of SVM. It can be used for the identification of UAVs in anti UAV system, which provides reference for similar research.