Powder properties are of great importance in powder bed fusion(PBF), one of the popular metal additive manufacturing (also called 3D printing) methods currently^[1-2]. Typically, spherical particles with proper size distribution support high flowability and density of the powder. A dense powder layer with uniform thickness can significantly improve the dimension accuracy during the melting process of PBF^[3].

Commonly, preparation method of the spherical powder includes plasma rotating electrode process(PREP), gas atomization (GA) and plasma spheroidization(PS)^[4-5]. Properties of the prepared powder based on the above methods can be characterized by the particle size distribution(PSD), degree of sphericity(DS, also called degree of circularity in 2-dimentional), and spheroidization ratio(SR)^[6-7].

One of the most used methods to measure the PSD of metal powder is laser diffraction(LD), which detects and analyses the angular distribution of the scattered light produced by a laser beam passing through a diluted powder layer^[6]. However, all the particles are assumed to perfectly spherical in the LD method, which is usually not the truth. Thus, shape information of the particles cannot be provided. Another method is by image analysis^[8], where the morphology of particles can be clearly observed by scanning electron microscopy(SEM), before the calculations of the degree of sphericity and PSD with assistant software^[9]. However, this method demands all particles to be spread sparsely, where the number of particles in the visual scope is limited. Meanwhile, the overlapped particle targets cannot be analyzed, resulting in the deviation in statistics^[10]. Moreover, it is very time-consuming to count the non- spheres among total particles manually, which makes a challenge to the measurement of the spheroidization ratio^[11-12].

With the advance in SEM technology, abundant particle information can be withdrawn from the microscopy image with the existing image processing tools, such as ImageJ^[13] and cisTEM^[14]. However, these tools, mostly based on conventional edge-based and thresholding algorithms, are hard to discriminate overlapped particles and require considerable human working, which results in inconsistent processing results.

In this work, the Mask R-CNN^[15], one of the remarkable instance segmentation convolutional neural networks, was employed to implement auto-statistics and the measurement of microscopy images of the spherical powder. In the past few years, the Mask R-CNN has been used in many complex vision tasks^[16-20]. Compared with traditional algorithms, the proposed model here is powerful in detecting the morphology of different particles, even though they are overlapped by upper particles.

Figure 1 depicts flow chart of the developed system, based on the instance segmentation results from the Mask R-CNN algorithm, which consists of particle size distribution, degree of sphericity and spheroidization ratio modules.

To train the Mask R-CNN model, powder microscopy images are collected with the requirement that powders with varied size and shape should be contained. The un-sifted Ti-6Al-3Nb-2Zr-1Mo alloy powder(provided by High Performance Powder Synthesis Lab, Fujian Innovation Academy, Chinese Academy of Sciences), prepared by radio frequency plasma spheroidization, was selected to construct the dataset. Before SEM, the powder sample is dispersed on the conductive tape and then blown to remove the unstuck powder by rubber suction bulb. Each SEM(using Phenom XL) image is magnified by 300 times to allow sufficient particles in a clear image with view field of 895 µm. The images were saved with a size of 2048×2048 pixel.

To increase the detection accuracy and facilitate the use of Mask R-CNN, the original SEM image, is then cropped into 16 sub-images with equal-size of 512×512 pixel (Fig.2(a)). LabelMe is selected as the tool to manual label the sub-images^[21]. Region of each particle is marked by a polygon with all vertex coordinates recorded in pixel dimension.

For ease of post-processing and statistical counting, 4 kinds of labels are adopted (Fig.1(f)): “ins_sphere” is for more than half occluded spherical particle, “inc_sphere” for less than half occluded spherical particle, “com_sphere” for complete spherical particle, and “non_sphere” for non-spherical particle. Characteristic image labeled by LabelMe is illustrated in Fig.2(b). 100 sub-images(Fig.1(d)), with total 14835 labeled particles (“com_sphere”: 7073, “inc_sphere”: 4396, “ins_sphere”: 1973, “non_sphere”: 1393), was applied for training. Figure2(c) shows one mask image, which generated from corresponding labeled image, as the input data to training process.

The Mask R-CNN, extended from Faster R-CNN^[22], is a state-of-the-art two stage instance segmentation architecture (Fig.1(e)). Firstly, based on the input image, the proposals about the regions where there might be an object is generated. Secondly, Mask R-CNN predicts the class of the object, refines the bounding box, and generates a mask in pixel level of the object based on the proposals.

At the first stage, the input images are processed by a feature extraction network, which also called backbone, to construct feature maps containing spatial semantic information at different scales. ResNet-101^[23] was used in this stage, which offers high accuracies at comparably low computational costs, without facing the vanishment of gradients^[24]. Then, a set of regions of interest (RoI) that may contain objects, are proposed by the regional proposal network(RPN), based on the feature maps.

At the second stage, feature maps for each RoI that proposed by RPN, are cropped by the RoI alignment, and resized with the same size for the following convolution networks. The RoI alignment also fixes misaligned features with low-resolution in the feature maps. Next, cropped feature maps that contain objects are fed into a classifier, which conducts classification and bounding box regression. After that, one object is enclosed by each bounding box. Finally, the original feature maps are cropped again using these bounding boxes, after being resized, the newly cropped feature maps are fed into a fully convolution network to conduct semantic segmentation and predict the binary mask.

After the prepared training image dataset is applied to train model parameters by Mask R-CNN, a set of pre-trained model weights on MS COCO dataset^[25] is adopted to fine-tune the developed model instead of training from scratch. During the training, the kernel weights and bias values are automatically altered to minimize the train loss, which is the difference between the input labeled masks and the network output masks. The batch size(the number of samples that will be propagated through the network) is set to 1, and the epoch(iteration times over all training data) is 100. The epoch & train loss curve is shown in Fig.3. This model was trained on an NVIDIA Quadro RTX 4000 GPU, which took 123 hours in total.

The raw input image size is 2048×2048 pixel, it is cropped into 28 sub-images (Fig.1(b)), in which 16 sub-images are with equal size (512×512 pixel) and 12 sub-images are with border strips among the 16 sub-images (Fig.2(a)), the green one is with 256×1024 pixel and the blue one with 1024×256 pixel. The width of border strips can be adjusted according to the maximum size of the powder in one image.

Before entering the 28 sub-images into the trained model, each sub-image is transferred into 4 images by maintaining itself, making a 180° rotation, flipping horizontally, and flipping vertically. Next, 4 output images are roughly merged into one sub-image. The purpose of this step is to improve recognition rate of the particle and reduce the wrong classification. Figure4 illustrates the transferring and merging process. In the rough merging process, one particle’s 4 masks(maybe less than 4) are merged simply depending on the intersection-over-union(IoU) and the intersection-over-self(IoS) of their circumscribed rectangle(Fig.5(a)). The usage of IoS here plays the role of precenting the occurrence of mis-merging, as shown in Fig.5(c). Each two masks satisfying the condition of IoU_Rec>0.7 ∩IoS_Rec-A>0.7 ∩IoS_Rec-B>0.7 will be merged. Several unrecognized particles inFig.4(c) are recognized after rough merging. The merged image contains some un-merged small masks, which are supposed to belong to one same particle, will be merged correctly in the next precise merging process.

The next step is to merge all the 28 predicted sub-images(Fig.1(f)) via precisely merging. The IoU and IoS of two masks instead of their circumscribed rectangles are adopted to conduct more precisely merging(Fig.5(b)), which consumes much more computation resources than the IoU & IoS with circumscribed rectangles, especially when thousands of particles are in one input image. In this process, each two masks that satisfy the condition of $Io{U_{Mask}} > 0.4 \cup Io{S_{Mask - A}}> 0.6\cup Io{S_{Mask - B}} > 0.6$ will be merged. 16 sub-images were stunk by 12 border strips stick in a tape-like style to reunite back the separated half particles at the border of two adjacent sub-images.

The error of the particle measurement comes from two facts. One is the deviation of boundary calculation due to the aliasing effect by the square pixels tessellation^[26]. Another is the residual prediction result and ground truth, comprising errors of the labeled region and the predict model. The B-spline curve is introduced to smooth boundary of the particle mask and reduce the former error, which can be described as the following parameterized function^[27]:

$Sp(t) = \sum\limits_{i = 0}^{n + k} {{p_i}N_i^k(t)} $ (1)

where $k$ is the degree of spline curve, ${p_i}$ is the coordinate of $i$th control point, $N_i^k(t)$ is the $i$th B-spline basis function with degree of $k$, it can be computed as follows:

$$$\begin{array}{l}{N}_i^0(t)=\{\begin{array}{c}0,t_i⩽t⩽t_{i+1}\\ 1,\mathrm{o}\mathrm{t}\mathrm{h}\mathrm{e}\mathrm{r}\mathrm{w}\mathrm{i}\mathrm{s}\mathrm{e}\end{array}\\ N_i^k(t)={\displaystyle \frac{t-t_i}{t_{i+k}-t_i}}{N}_i^{k-1}(t)+{\displaystyle \frac{t_{i+k+1}-t}{t_{i+k+1}-t_{i+1}}}{N}_{i+1}^{k-1}(t)\end{array}$$ $ (2)

where ${t_i}$ is the $i$th element in a uniformly distributed knot sequence that ranges from 0 to 1. During the smoothing process, the contour points of the mask are extracted and each 5 points of them is selected to be the control points. As shown in Fig.6(a), the green points are the contour points, the red one the control points, and the red curve the B-spline curve.

To compensate the second residual error, a set of standard circles with evenly spaced diameter from 5 µm to 100 µm are predicted before predicting the input image. The deviation of output result and ground truth of these standard circles can be calculated, and the residual function can be fitted using the scattered deviation values. Figure6(b) shows the fitted residual function curves of perimeter and area. The residual value(perimeter or area) will be compensated during the statistical process for corresponding mask’s equivalent projected area diameter according to the residual function.

Except for the particles at the edge of the image, each particle in the original image (2 048×2 048 pixel) is classified, and their masks’ information is extracted. There are two methods to calculate the size of a particular particle. One is to calculate the equivalent projected area diameter, corresponding to the diameter of a circle with the same projected area as the particle. However, as nearly 30% particles are occluded in one image, it’s inaccurate to use equivalently projected area diameter. Another is to calculate the minimum circumscribed circle diameter. This descriptor can provide the diameter for the “inc_sphere” particles, which account for 90% of the occluded particles. The “ins_sphere” particles, accounting for 3% in all particles, are abandoned in the statistics of PSD due to the limitation of 2-dimensional images.

The particle’s degree of sphericity is calculated using the following formula (also called root of form factor)^[28]:

$DS = \frac{{2\sqrt {\pi A} }}{P}$ (3)

where $A$ is the projected area of the particle, and $P$ is the perimeter of the particle periphery. This index, sensitive to boundary irregularity, represents how the shape of the particle deviates from a standard circle.

The spheroidization ratio is defined as follow:

$SR = \left(1 - \dfrac{{{N_{non}}}}{{{N_{all}}}}\right) \times 100 {\text{%}} $ (4)

where ${N_{non}}$ is the number of “non_phere” particles, and ${N_{all}}$ is the number of all particles.

The output image of our model is shown in Fig.7(d). Each recognized particle is labeled and colored, where the label describes the class and probability that the particle belongs to. The color represents the class intuitively. Here, green is for “com_sphere”, blue for “inc_sphere”, orange for “ins_sphere”, and red for “non_sphere”, respectively. We use the Phenom ProSuite Software Particlemetric, a professional microscopy image processing software, to compare our proposed method with traditional image segmentation method (Fig.7(c) and 7(e)). Recognition accuracy of the proposed method is 96.95%, higher than that of 78.44% by Particlemetric.

As the non-sphere particles have complicated random surface texture and shape, traditional method shows poor recognition ability on them. In comparison, the proposed model can recognize these non-sphere particles correctly via learning deeply the complex feature, where some small spherical particles adhering to the non-sphere particle can also be detected. Moreover, it is hard to separate two particles with position closed to or overlapped to each other by the traditional method (Fig.7(c)), which can be easily segmented by our system and tell which one is un-occluded.

The statistics results of 8 raw images in PSD and DSD are shown in Fig.8. Total 9374 particles were detected by the Particlemetric, less than that of 12192 by our methods. Most of the small particles are attached and detected as one part of the big ones in the traditional method (Fig.7(e)), which can be solved by our method. Non-sphere particles were recognized as many tiny small particles with less than 5 µm by the traditional method (Fig.7(c)), which was not shown in in the corresponding PSD results (Fig.8(a)). This inconsistence is due to the fact that the particle with smaller size will be considered as noise, according to the image resolution of the software.

Obvious difference in PSD is shown between methods by laser diffraction (using LS 13 320 Tornado) and microscopy image particle segmentation (Fig.8(a)). This is attributed to the satellite particle, observed in the formation of small particles^[29], which can adhere to the larger particles. However, all separated particles are treated as the perfect sphere by the laser diffraction method, even though the separated particles are with many tiny appendages. In this respect, satellite ratio of the powder can be evaluated according to the deviation of PSD measured by the proposed segmentation method and laser diffraction, since that there is no good method to characterize the satellite ratio now. As the in-complete sphere particles are not calculated during the statistical process of degree of sphericity, significant difference in the statistical results of degree of sphericity are shown between the traditional method and our model (Fig.8(b)).

Unlike the other two methods, spheroidization ratio can be provided by the proposed model, where 646 non-sphere particles among total 12192 particles in the 8 raw images can be calculated, corresponding to a SR value of 94.70%.

In this study, a spherical particle image segmentation and auto-statistics system is proposed by employing deep learning and mask merging techniques. The proposed method can recognize particles with four typical shapes and extract their feature and size information, before providing the particle size distribution, degree of sphericity and spheroidization ratio of the powder. Superior to the existing method of image analysis and laser diffraction, the proposed method can also detect overlapped spherical particles with high accuracy, automatically calculating the spheroidization ratio of powder, and providing an orientation in the measurement of satellite ratio of the spherical powder. Besides providing accurate particle size and shape information during the production process of spherical power, the proposed method can also be extended to a large variety of particles.