To address the practical issues of few pixels, limited information, detection difficulties, and misalignment of small objects in remote sensing images, this paper aims to improve the YOLOv5 and proposes a technique of boosting residual connections and cross-layer attention to improve the model's detection capability for small objects in remote sensing images. To effectively improve the detection capability of YOLOv5 for small objects in remote sensing images, the method employs residual linking for feature maps and the addition of detection heads. Furthermore, using cross-layer attention, this paper attaches semantic informations to the features of different network layers, improving the model's ability to suppress complex background informations in remote sensing images. In the experiments on the Detection in Optical Remote (DIOR) remote sensing dataset, the proposed approach achieves a mean accuracy precision (mAP) of 86.4% and a small object detection accuracy evaluation metrics (APs) of 23.4%, which is 5.9 percentage points higher than the benchmark network. The experimental results show that the method proposed in this research performs well in small object detection problems in remote sensing images, and it also confirms that the bottom feature map and attention mechanism in the feature pyramid are critical for improving small object detection performance.