Aiming at the problem that it is difficult to detect infrared dim and small target accurately and quickly in complex background, a lightweight real-time network model YOLO-IDSTD for infrared dim and small target detection was proposed. Firstly, in order to improve the detection speed, the network structure of the feature extraction part was redesigned, and the Focus module was used to reduce the reasoning time after the input layer. Secondly, in order to enhance the detection ability, the path aggregation network was adopted in the feature fusion part and an improved receptive field block was added. Finally, four-scales detection was increased in the target detection part. Compared with the classical lightweight model YOLOv3-tiny on the infrared dim and small target data set, the recall is increased by 7.57%, the average pricision is increased by 1.92%, and the CPU reasoning speed is increased by 36.1%. The model can balance accuracy and speed, and the amount of calculation and parameters are significantly reduced. The size of the model is compressed to 7.27 MB, which reduces the dependence on the computing power of the hardware platform and realizes the accurate and fast detection of infrared dim and small targets.