Particles size detection is an important link in production, and the use of cameras to capture and process images is a commonly-used non-contact detection method. To meet the requirements of identification and size detection of particles, the sand particles were selected as the detection object, and a Mask R-CNN model with the improved boundary mask of particles was proposed. Combined with the classical edge detection technology, the deep learning model was used to predict the mask, and the mask with higher precision was obtained according to the results of edge segmentation. The DenseNet was used as the backbone network of the network detection to reduce the number of network parameters, and the channel attention mechanism was used to strengthen the feature extraction ability of the network. The experiments show that the improved network can improve the detection accuracy, and the combination of image processing can improve the accuracy of mask size detection, which provides a meaningful method for industrial detection of particles.