Infrared ship target detection and recognition technology is the key technology of infrared imaging guidance for anti-ship missile, which is of great significance to the guidance performance of weapon equipment. For the accuracy and speed in complex environment, an infrared ship target detection algorithm based on YOLOX-S was proposed. Firstly, the depthwise separable convolution (DSC) was introduced to replace the traditional convolution in the feature pyramid network (FPN) and YOLOHead residual structures, which could reduce the amount of parameters of the model. Secondly, the ECANet channel attention mechanism was introduced to improve the attention of network, which could reduce the false detection rate and missed detection rate of ship targets. Finally, the CIoU loss function was used to further improve the detection accuracy of the network. The experimental results show that the average precision (AP) of the optimized algorithm reaches 98% and the detection speed is 56 frame/s. Compared with the previous YOLOX-S algorithm, the detection speed and the average precision are improved by 6 frame/s and 3%, respectively, and the model is more lightweight. The experimental results fully prove that the proposed algorithm can effectively complete the infrared ship target detection task.