To address the issues of low accuracy, slow speed, poor robustness, and missed detections while utilizing traditional image processing algorithms for identification of contact lens surface defects, an algorithm based on an improved Faster R-CNN is proposed. First, ResNet50 was chosen as the backbone network based on the performance of three feature extraction networks. Second, a feature pyramid network (FPN) is introduced to enhance the multi-scale detection capability of Faster R-CNN by fusing multi-level feature information. Finally, based on the dataset of contact lens surface defects, K-means++ algorithm is used to improve the size and number of anchors. According to the experimental findings, the updated Faster R-CNN algorithm's mean average precision (mAP) on the test dataset is 86.95%, 9.45 percentage points greater than the original Faster R-CNN algorithm. The improved algorithm can efficiently identify several typical defects of contact lens, such as the bubble, turning point, scratch, and mold point.