Electroluminescence (EL) inspection technology is widely used as an important means for solar cell and module defect detection. However, defect screening in EL inspection is still a major challenge. Herein, to overcome the limitations in previous studies, such as few types of defects to be identified, the inability to locate defects, the large size of model parameters, and slow detection speed, the upgraded YOLOv5 network is used to detect and classify the four types of defects that are commonly found in electroluminescent images, including crack, finger interruption, break, and black zone. This shows that the improved Ghost module extracts ordinary convolutional modules in the network to reduce number of network model parameters compared with the YOLOv5 backbone. Additionally, to ensure good detection performance, the Squeeze-and-Excitation (SE) attention module is added to the tail of the backbone network to improve the algorithm' target detection ability. In the neck part, the bidirectional feature pyramid network (BiFPN) structure is used to further strengthens the feature fusion capability of the network. Experimental results show that the proposed model successfully identifies and locates the four common defects, has a reduced volume by 21% compared with the YOLOv5 algorithm and achieves an improved single image detection speed by 17.4% without GPU acceleration.