Surface defects of optical components seriously affect the performance of optical systems. The existing surface defects detection technology of optical components has problems such as slow detection speed and low accuracy. This paper studies the digital image processing method of polarization imaging of defects based on the polarization detection technology of micro-defects of optical components. Firstly, we used the polarization property of light to capture the image with high contrast of defects, and then pre-processed the image by filtering and denoising, threshold segmentation, morphological processing. Next, different feature parameters are selected according to different types of defects so as to achieve defect detection and classification, meanwhile accurately measuring the size of defects. The results showed that the total defect detection rate of this method was 95.90%, and the detection time of a single image was less than 50 ms.