An improved helmet detection algorithm for YOLOv4 (SMD-YOLOv4) is proposed to effectively detect whether construction workers are wearing helmets in complex scenes and reduce safety hazards. First, the SE-Net attention module is used to improve the ability of the model backbone network to extract effective features. Next, a dense atrous space pyramid pooling (DenseASPP) is used instead of spatial pyramid pooling (SPP) in the network to reduce information loss and optimize the extraction of global contextual information. Finally, the scale of feature fusion is increased in the PANet part and deep separable convolution is introduced to obtain detailed information about small targets in complex contexts without slowing down the network inference speed. The experimental results show that the mean average precision (mAP) of SMD-YOLOv4 algorithm reaches 97.34% on the self-built experimental dataset, which is 26.41 percentage points, 6.44 percentage points, 3.25 percentage points, 1.49 percentage points, and 3.19 percentage points higher than that of the current representative Faster R-CNN, SSD, YOLOv5, YOLOx, and original YOLOv4 algorithms, respectively, and can meet the real-time detection requirements.