Infrared image has the advantages of long detection distance and wide selectable working time, and plays an important role in infrared target detection in the field of ship image detection. Due to the existence of a large amount of interference information, ship target detection in a complex sea and sky background is facing enormous challenges. The target detection algorithm based on deep learning has strong ability to extract features, strong adaptability of the model to the environment, and good detection effect and stability. YOLOv5 algorithm is a widely used target detection algorithm based on deep learning, but there are still shortcomings in the process of infrared ship target detection. To solve the gradient explosion problem of YOLOv5 algorithm in marine infrared ship target detection, the border regression loss function based on CIoU is improved, the regression process is optimized, the convergence effect of the model is improved, and the gradient explosion problem is solved in this paper. To address the problem of inconsistency between the size of the target dataset and the anchor frame, the K-means algorithm is improved to obtain an anchor frame suitable for the infrared ship dataset used in this algorithm, which improves the algorithm's detection ability for infrared ship targets.The K-means clustering algorithm is improved, median is used instead of the average as the selection criteria for the clustering center to reduce the impact of discrete points on the clustering results. By improving the penalty term of the aspect ratio in the frame regression loss function, a regression loss function named MIoU (Multivariate intersection over union) is proposed, which optimizes the regression process, improves the convergence speed and detection accuracy, and avoids false detection and missed detection of similar targets.Using the anchor frame generated from the infrared ship dataset to train the YOLOv5 algorithm model, the experimental results show that it improves by 0.7% compared to the standard YOLOv5 algorithm on the mAP (Fig.5). Comparative experiments are conducted using different border regression loss functions in the YOLOv5. The border regression loss functions include the border regression loss function Smooth L1 based on center distance and the border regression loss function based on the overlapping area of IoU, GIoU, DIoU, and CIoU. The results show that except for CIoU and MIoU, other loss functions can detect two small targets that are relatively close to each other as a single target. Only CIoU and MIoU border regression function can accurately detect the target and avoid false detection (Fig.8). Compared to other frame loss functions, MIoU frame regression loss functions can detect more targets in the image and avoid missing detection of some targets (Fig.9). Comparing MIoU and CIoU with better test results in terms of frame loss (Fig.10) shows that the MIoU loss function Box-loss decreases by 1.5%. Comparative experiments are conducted to compare the improved YOLOv5 algorithm with other improved YOLOv5 algorithms. The experimental results are given (Tab.2). The ablation experiments are conducted on the improved method, and the results (Tab.4) and the recognition results (Fig.11) of various comparison algorithms are given.The infrared ship target detection algorithm based on improved YOLOv5 is proposed to address the issues of inaccurate detection and poor boundary regression performance of the anchor frame and dataset target sizes that do not match the YOLOv5 algorithm. Firstly, the K-means anchor frame clustering algorithm is improved, the median is used instead of the average to select the cluster center, the impact of discrete points is avoided, and the intersection ratio of the border represented by two points to replace the distance between the two points is used. The anchor box is made more compatible with the dataset target. At the same time, the aspect ratio penalty term of the border regression function is improved, which effectively avoids gradient explosion in the regression process, and optimizes the border regression process. The ablation and contrast experiments are carried out on the infrared ship data set. The experiment results show that the average detection accuracy of the improved algorithm is 1.1% higher than the standard YOLOv5 algorithm and has higher average detection accuracy than other improved YOLOv5 algorithms, which verifies the superiority of the improved algorithm and improves the detection effect of the infrared ship target.