Considering the non-effectiveness of traditional convolution neural networks in detecting pattern defects in yarn-dyed fabrics, a defect detection method based on a U-shaped Swin Transformer reconstruction model and residual analysis is proposed. This method uses the Transformer model to improve the extraction of global image features and enhance reconstruction while solving for the small number and unbalanced types of defective samples during the actual production process. First, the training process of the reconstructed model is completed for a certain pattern using the non-defective samples after adding noise. Subsequently, the test image is inputted into the model to obtain the reconstructed image, and its residual image and reconstructed image are calculated. Finally, the defect areas are detected and located via threshold segmentation and mathematical morphology processing. The results indicate that this method can be effectively used for the detection and location of defect areas on multiple yarn-dyed fabric patterns without requiring the marking of the defective samples.