The eating and drinking behavior of pigs is the most direct evidence to evaluate the health degree of pigs. Therefore, it is of great significance to use real-time monitoring of the eating and drinking status of pigs by computer vision technology for improving the welfare of pig breeding. This paper proposes a recognition method of pig eating and drinking behavior based on visible spectrum and improved YOLOv2 neural network. The method builds head-neck model on the pig visible spectrum image sequence, making use of improved YOLOv2 neural network to realize target detection in the scene of the real pigsty, then utilizing the output of the position information for preliminary judgment of eating and drinking behavior. Then using traditional image processing methods to make an accurate judgment of pig eating and drinking behavior. First, the head-neck model is constructed in the sequence of pig images, and the unblocked head and neck were used as the detection target. This model can effectively solve the occlusion problem in the pig target detection processing, and can also accurately locate the head of the pig, providing assistance for the subsequent identification of eating and drinking behaviors. Then this paper adopted the international mainstream neural network YOLOv2 as the basic network model for target detection, and improve the activation function to achieve fast and accurate target detection of live pigs. Before network training, the K-means algorithm is used to cluster the target frame of the homemade pig data set. Compared with the initial performance of YOLOv2, the mAP value and the Recall value was improved by 3.94% and 5.3%. In order to increase the robustness of the network-facing input changes or noise, this paper compared the performance of the three activation functions of ReLU, Leaky-ReLU and ELU, and found that the performance of the ELU was significantly different from the former two. Compared with the original YOLOv2 and Faster R-CNN, the target detection model in this paper has a mAP value of 90.24% and a recall value of 84.56%, both of which are better than the latter two. Finally, the pig head-neck position information gets from target detection was used to make the preliminary judgment of eating and drinking behavior. When pigs appeared in the eating and drinking area of the picture, background difference method, morphological calculation and other image processing methods are performed on the picture, and the pig eating and drinking behavior is judged more accurately by combining with the residence time of the drinking area. The experiment shows that: the method used in this paper can be used to judge the eating and drinking behavior with an accuracy of 94.59% and 96.49%, which are better than the results judged by the traditional method directly, and can be applied to assist the management of breeding personnel in the actual breeding process.