Insulator defect detection often faces challenges such as large model parameters, significant interference from complex backgrounds, and suboptimal performance in detecting small target defects. To address these issues, in this study, an improved YOLOv8n-based insulator defect detection algorithm is proposed. The approach introduces a novel C2f-RE module applied to the backbone network, which reduces model parameter redundancy, suppresses background interference, and enhances feature extraction capabilities. Additionally, the neck structure is redesigned, incorporating the focusing diffusion pyramid network (FDPN). This modification aims to improve small target feature extraction by integrating multiscale feature information across high, medium, and low dimensions. Furthermore, a lightweight shared detail-enhanced convolutional detection head (LSDECD) is proposed to facilitate multiscale feature information interaction while minimizing the number of model parameters. Additionally, an improved Inner-WIoU loss function is employed to effectively focus on small target samples and ordinary quality annotation samples. Experimental results demonstrate that the proposed algorithm realizes a mean average precision P_mA of 97.1%, representing a 2.6% increase in P_mA and a 4% boost in recall R when compared to the original YOLOv8n algorithm. Finally, the number of parameters and computational costs are reduced by 35.5% and 23.5% respectively, and the model size is reduced by 1.8 MB. The method effectively balances detection accuracy with lightweight characteristics, and it exhibits strong robustness against complex weather interference.