As a versatile digital holography technique, optical scanning holography (OSH) has been widely used in microscopy^[1], remote sensing^[2], image encryption, etc^[3,4]. In OSH, the object is scanned by the heterodyne beams, which are launched from the same light source with frequency difference generated by the frequency shifter. Unlike traditional digital holography, OSH can record 3D objects into 2D holograms by single-pixel 2D scanning. The hologram can preserve the amplitude as well as the phase information of the object. One can obtain this object’s information using hologram reconstruction.

In the image reconstruction, there are two important issues. The first one is auto-focusing, i.e., finding the accurate reconstruction distance, and the other is choosing an applicable reconstruction method with increasing depth resolution as well as less defocus noise. A great deal of research have proposed to retrieve the reconstruction distance automatically, such as the extended focused imaging^[5], structure tensor^[6], and connected domain^[7]. Researchers also used the time-reversal (TR) technique to get the depth information by calculating the pseudo-spectrum of the TR matrix generated from the hologram^[8]. To further improve the depth resolution, methods based on double measurements have also been proposed, including the use of a dual-wavelength laser^[9], double-location detection^[10], and the reconfigurable pupil^[11].

The out-of-focus haze, also known as the defocus noise, is undesired residual signals from other sections. Many methods have been presented to conduct image reconstruction and to suppress the defocus noise, such as inverse imaging^[12,13], Wiener filtering^[14], and 3D imaging^[15]. For example, Zhou et al. used a random phase pupil to transfer the defocus noise into speckle-like patterns^[16]. This noise can be further suppressed by average^[16], connected component^[17], and image fusion^[18].

In recent years, deep learning has undergone rapid development and has found wide applications in some areas, such as language processing, image processing, biomedical and machine visions, as well as digital holography^[19–22]. Ren et al. presented a convolution neural network (CNN) based on the regression method to achieve fast auto-focusing^[23]. The CNN has been trained by a set of holograms a priori. Pitkäaho et al. showed that CNNs can also predict the in-focus depth by learning from half a million hologram amplitude images in advance^[24]. Compared with traditional methods, their work showed better precision and efficiency. Rivenson et al. used deep learning to rapidly perform phase recovery and image reconstruction simultaneously. The calculation was based on only one hologram, and could reconstruct both the phase and amplitude images of the objects^[25]. Nguyen et al. presented a phase aberration compensation method based on deep learning CNN^[26]. It could perform automatic background region detection for most aberrations. Deep learning has also proved an effective tool in molecular diagnostics^[27], as Kim et al. trained the neural networks to classify the holograms without reconstruction. The captured holograms of the cells were used as raw holograms to train the neural networks, which were able to classify individual cells afterwards.

In this paper, we present for the first time and to the best of our knowledge, a reconstruction method based on a U-shaped convolutional neural network (U-net) to remove the speckle-like defocus noise in a OSH system. The U-net approach is adopted to learn the mapping between various holograms and the corresponding sectional images. Unlike other CNN methods, which require large training data sets, U-net can work with very few training images and yields more precise results. The proposed method can eliminate the speckle-like noise generated by the random phase pupil. Simulation results show that the algorithm works well with both simple and complex graphics. It also outperforms the traditional reconstruction methods in terms of better sectional image quality and significantly faster processing speed.

This paper is organized as follows. In Section 2, we first introduce the OSH system and the principle of random phase pupil system. The theory of U-net deep learning is also explained in this section. Simulation results are presented and discussed in Section 3 to demonstrate the visibility of the proposed method. The conclusion remarks are given in Section 4.

The holographic system is illustrated in Fig. 1. A He–Ne laser is set at the starting position of the system to launch a cluster of planar waves. The waves will be split into two beams by beam splitter BS1. An acousto-optic frequency shifter (AOFS) is used to shift one of the frequencies from ${\omega }_0$ to ${\omega }_0+\mathrm{\Omega }$. The two beams will pass though different mirrors, pupils, and lenses, respectively, and then converge at BS2. After that, the combined beam will be used to scan the object via the $X–Y$ scanner. In the meantime, lens L3 will collect the light from the object. The generated electrical signal from the photo detector (PD) will be transformed into a hologram via a band-pass filter and further demodulation process.

In a random phase pupil holographic system, the two pupils are set as $p_1(x,y)=\exp [j2\pi r(x,y)]$ and $p_2(x,y)=1$, where $r(x,y)$ is a random function, and its value is chosen from a uniform distribution between (0, 1). The random phase pupil is widely used in OSH to disperse the defocus noise from other sections into speckle-like patterns^[16]. If we discretize the object into $N$ sections along the $z$-axis, then the hologram can be expressed as $$\begin{gathered} g(x,y)=\sum _{i=1}^N{\mathcal{F}}^{-1}\{\mathcal{F}[\mathcal{O}(x,y;z_i)]\times \exp [-j\frac{z_i}{2k_0}(k_x^2+k_y^2)] \\ \times P_1^{*}(-\frac{z_i{k}_x}{f},-\frac{z_i{k}_y}{f})\}, \end{gathered}$$(1)where $\mathcal{O}(x,y;z_i)$ is the complex amplitude function of the object, $x$ and $y$ are space coordinates of the object, and $z_i$ indicates the axial coordinates of the $i$th section. $k_x$ and $k_y$ are frequency domain coordinates, and $k_0=\frac{2\pi }{\lambda }$ is the wavenumber, where $\lambda$ represents the wavelength of the laser. $\mathcal{F}$ and ${\mathcal{F}}^{-1}$ indicate the Fourier transformation and inverse Fourier transformation, respectively. $P_1$ is the frequency domain expression of $p_1(x,y)$, $*$ represents the conjugate operation, and $f$ is the focus distance of the lens.

To recover the $j$th section, we can set the decoding pupils as $p_{1d}(x,y)=1$ and $p_1^{*}(-x,-y)p_{2d}(x,y)=1$. The reconstruction distance is set as $z_j$. In this case, the recovered image becomes $$\begin{gathered} I_{\mathrm{out}}(x,y;z_j)={\mathcal{F}}^{-1}\{\mathcal{F}\{g(x,y)\}\times \exp [j\frac{z_j}{2k_0}(k_x^2+k_y^2)] \\ \times P_{2d}(\frac{z_j{k}_x}{f},\frac{z_j{k}_y}{f})\} \\ =\mathcal{O}(x,y;z_j)+N(x,y\text{:}{z}_j), \end{gathered}$$(2)where $I_{\mathrm{out}}(x,y;z_j)$ is the reconstructed sectional image at $z_j$, which contains useful signal $\mathcal{O}(x,y;z_j)$ and the speckle-like noise signal $N(x,y;z_j)$. The speckle noise can be derived by substituting Eq. (1) into Eq. (2) and can be expressed as^[16]$$\begin{gathered} N(x,y\text{:}{z}_j)=\sum _{i\ne j}^N{\mathcal{F}}^{-1}\{\mathcal{F}\{\mathcal{O}(x,j;z_i)\times \exp [j\frac{z_j-z_i}{2k_0}(k_x^2+k_y^2)\left]\right\} \\ \times P_1^{*}(-\frac{z_i{k}_x}{f},-\frac{z_i{k}_y}{f})\times P_{2d}(\frac{z_j{k}_x}{f},\frac{z_j{k}_y}{f})\}. \end{gathered}$$(3)

For a better sectioning effect, this noise should be further eliminated. One can suppress the speckle haze by averaging multiple section images or using the connected component methods^[16,17]. These methods succeed in suppressing overriding noise. However, they all require multiple frames to solve the problem. This would greatly reduce the efficiency. Here, we present a special CNN-based method, U-net, to suppress the speckle-like noise. In addition to having no requirement of prior information, the U-net method also features a simpler, shorter operation time and is more robust than the conventional methods mentioned above.

Unlike other CNN methods, which require large training data sets, U-net can work with very few training images and yield more precise results^[29,30]. This is the main advantage, especially in the situation where it is difficult to retrieve large data sets, such as biological applications.

The neural network is organized by a contracting path as well as an expansive path. As the appearance is very similar to the letter ‘U’, this neural network has been named U-net. U-net was first presented to segment the biomedical image and has proved to be a very effective end-to-end image processing tool^[30,31].

Figure 2 illustrates the architecture of the U-net, which contains two paths: the contracting one (convolution + downsampling) and the expansive one (deconvolution + upsampling). The contracting path has many layers. Each layer consists of two $3\times 3$ convolutions, followed by a rectified linear unit (ReLU). For downsampling, a $2\times 2$ max pooling operation with stride 2 is used. While for the expansive path, the pattern of deconvolution with upsampling is repeated. A $2\times 2$ convolution is used for upsampling, and two $3\times 3$ convolutions followed by a ReLU activation function are included in the deconvolution layers. In the final layer, the required features are retained while those structures related to speckle noise are discarded.

It is worth noting that the number of feature channels are copied and cropped after each downsampling process, as denoted by the white boxes in Fig. 2. These contracted high resolution features are then merged and combined with the upsampled output to generate a more precise output.

For the $\mathcal{L}$-th convolutional layer, we assume there are $N_{\mathcal{L}}$ feature maps with a size $k\times k$, and it can be expressed as $h_j^{\mathcal{L}}$, $j=\mathrm{1,2,\dots ,}{N}_{\mathcal{L}}$. The next convolutional layer $h_j^{\mathcal{L}+1}$ can be expressed as $$h_j^{\mathcal{L}+1}=\mathrm{ReLU}(\sum _{i=1}^{N_{\mathcal{L}}}{h}_i^{\mathcal{L}}\otimes w_{ij}^{\mathcal{L}+1}+b_j^{\mathcal{L}+1}),$$(4)where $\otimes$ is the convolution operation, and $w_{ij}$ and $b_j$ are the weight and bias that need to be learned via training, respectively. ReLU is an activation function, which can be denoted as $\mathrm{ReLU}(x)=\max (0,x)$. In a U-net model, when the inputs are transmitted between neurons, the weights are applied to the inputs and passed into an activation function ReLU along with the bias. The weights are essentially reflecting how important an input is, while the value of bias controls when the activation function is activated. In the process of downsampling, we need to pad the feature maps with zero first. This process can make the next convolutional layer feature maps have the same size after convolution. In the next step, we utilize the max pooling to choose the maximum value as the representation among a small region. In this way, the size of the feature maps will be smaller. The process of upsampling is similar with the downsampling, except that the upsampling needs to extend the size of the feature maps with zero. It is worth noting that lots of useful information will be dropped out in the process of downsampling. To preserve enough significant information, U-net provides an approach to merge the symmetric parts of convolution and deconvolution.

To suppress the defocus noise, a sufficient training set should be collected. The training set contains the encoded hologram with speckle-like noise and the labeled image without noise. In the training process, the reconstructed images with noise are set as input images, which would propagate forward to obtain the predicted images. The loss function $L$ is defined as a function between the predicted image and the labeled standard image without noise by mean square error (MSE), $$L=\frac{1}{M}\sum _{i=1}^M{|y_i-{\widehat{y}}_i|}^2,$$(5)where $M$ is the total count of pixels, $y_i$ is the labeled standard image, and ${\widehat{y}}_i$ represents the predicted image.

The proposed method was demonstrated via simulation. The optical process was simulated with Matlab, while the reconstruction results based on the proposed U-net method were generated with Pytorch. The GPU used in the simulation was NVIDIA GTX 1080Ti with 16 GB memory. A He–Ne laser centered at 632.8 nm was used as the source. The focal length of the lens L1 and L2 was $f=75\mathrm{mm}$. An object with two sections located at $z_1=30\mathrm{mm}$ and $z_2=30.3\mathrm{mm}$ was used in the simulation. The size of each section was $2\mathrm{mm}\times 2\mathrm{mm}$, which was sampled to $256\times 256$ pixels.

The U-net method was first verified with simple graphics, such as the English alphabet. In the training process, it is important to generate enough data sets. In the simulation, 386 original images were used, with each one passing through 27 different random phase pupils in the OSH system as shown in Fig. 1. In this way, 10,422 data sets were produced for training. Some of the sample images with speckle-like noise are shown in Fig. 3, which are used as the input images of the U-net model. The speckle-like noises are generated from the other section based on Eqs. (1) and (2). The sectional images are in the database as mentioned above. One can observe from this figure that different speckle-like noise was added according to Eq. (2). The corresponding noise-free images, also denoted as the standard images or reference images of the U-net, are shown in Fig. 4.

To accelerate the convergence, we chosed the method for stochastic optimization with a learning rate equal to 0.0001^[32]. The parameters less than 0.5 were dropped out to prevent over-fitting^[33]. The relationship between the training loss and the iteration times is shown as the blue curve in Fig. 5, while the orange one represents the relationship between the validation loss and the iteration times. It can be seen from this figure that the loss of the training data as well as the validation data both decrease with the iteration times.

The first simulation results proved that the U-net architecture really did a good job of learning the characteristics of speckle-like noise. Here, we demonstrate the reconstruction results under different random phase pupils in Fig. 6. Three different test images were used to verify the proposed method. The image with the letters ‘ABC’ was in the training data sets, while the ones with letter ‘$XYZ$’ and a Chinese character were not included. One can observe from this figure that the speckle-like noise has been eliminated successfully.

To evaluate the reconstruction effect with the U-net, we compare the results with two important factors: the peak signal-to-noise ratio (PSNR) and the structural similarity (SSIM). The PSNR can be defined as^[34]$$\mathrm{MSE}=\frac{1}{M_1{N}_1}\sum _{i=1}^{M1}\sum _{j=1}^{N_1}{[X(i,j)-Y(i,j)]}^2,$$(6)$$\mathrm{PSNR}=10\lg \left[\frac{{{(2}^n-1)}^2}{\mathrm{MSE}}\right],$$(7)where $X(i,j)$ is the output image, while $Y(i,j)$ is the corresponding reference image. $M_1$ and $N_1$ are the number of columns and rows, respectively, and MSE stands for mean square error.

The SSIM is used to quantify the visibility of differences between the output image and the corresponding reference image. The quality assessment is based on the degradation of structural information and can be expressed as^[35]$$\mathrm{SSIM}=l(X,Y)\times c(X,Y)\times s(X,Y),$$(8)where $$l(X,Y)=\frac{2{\mu }_X{\mu }_Y+C_1}{{\mu }_X^2+{\mu }_Y^2+C_1},$$$$c(X,Y)=\frac{2{\sigma }_X{\mu }_Y+C_2}{{\sigma }_X^2+{\sigma }_Y^2+C_2},$$$$s(X,Y)=\frac{{\sigma }_{XY}+C_3}{{\sigma }_X{\sigma }_Y+C_3},$$where $X$ and $Y$ represent the output image and the reference image, respectively. ${\mu }_X$ and ${\mu }_Y$ are the means of $X$ and $Y$, ${\sigma }_X$ and ${\sigma }_Y$ are the variances of $X$ and $Y$, ${\sigma }_{XY}$ is the covariance of $X$ and $Y$, respectively, and $C_1$, $C_2$, and $C_3$ are constants. In general, $C_1=(K_1\times L{)}^2$, $C_2=(K_2\times L{)}^2$, and $C_3=C_2/2$. Here, we assume that $K_1=0.01$, $K_2=0.03$, and $L=255$.

The quantified assessment results are shown in Table 1. One can observe from the row ‘Input VS Original’ that, the PSNR for all cases is all around 15 dB, and the SSIM is quite small at around 0.06. This means that the speckle-like noise has greatly degraded the signal-to-noise ratio, and the similarity between the noisy images and the original ones has also been reduced to a rather low degree. While for the scenario of ‘Output VS Original’, the PSNR for all cases is increased above 32 dB, and the SSIM is also raised close to 1. This indicates that speckle haze has been eliminated successfully, and the output images of the U-net have small distortion and high similarity with the original ones.

It is worth noting that the computation time for all cases are around 33.3 ms, and the computer is configured as NVIDIA GTX 1080Ti with 16 GB memory.

We have also measured the sectioning results of the Chinese character in Fig. 6(h) with different noise ratios. The noise is generated from different sections with the traditional method^[16], as is shown in Figs. 7(a)–7(c). The sectional images generated by the U-net method are shown in Figs. 7(d)–7(f), respectively. Table 2 presents the corresponding quantified results. It can be seen from Fig. 6 and Table 2 that for sectional images with different noise ratios, the values of the PSNR and the SSIM all increase significantly (with the PSNR around 32 dB and the SSIM close to 1). This result shows that the U-net method can successfully remove the defocus noise with different noise ratios.

In this subsection, the situation of complex graphics is tested. Some samples of the complex graphics are shown in Fig. 8, in which some standard digital image processing images are included, such as Barbara, Cameraman, and Peppers. These images were used in the OSH system to generate holograms with complex graphics in order to test the noise-suppressing ability of the proposed method.

To generate the training data sets, 364 original images were used, with each one passing through 30 different random phase pupils. In this way, 10,920 data sets were produced for the U-net model. It is important to mention that the speckle-like noises are generated from the other section based on Eqs. (1) and (2). The sectional images are in the database as mentioned above.

The loss function for both the training data and the validation data were calculated in the training process. The results are shown in Fig. 9. As can be seen from this figure, the loss function decreases with the iteration times. One can expect that the defocus noise can be suppressed after 400 iterations. The generalization gap between training loss and validation loss is measured to be around 0.002 in this case.

The reconstruction results using the U-net with complex graphics are shown in Fig. 10. Two complex graphics named ‘Monkey’ and ‘Rice’ were used to test the U-net method. It can be seen from this figure that most of the speckle haze has been eliminated.

The quantified evaluation results are listed in Table 3. By comparing the data in the row ‘Input VS Original’ with that in the row ‘Output VS Original’, one can observe that the values of the PSNR and the SSIM have both increased after the U-net processing. The increased PSNR represents the improvement of the signal-to-noise ratio, which means that the noise has been decreased. While the variation of the SSIM suggests higher similarity between the output images of the U-net and the original ones.

It can also be concluded from Tables 1 and 3 that as the input image becomes more complex, the improvement of the PSNR would degrade. This indicates that it is harder for U-net to distinguish the features between the image and the speckle noise when the complexity of the image increases.

To verify the feasibility of eliminating the defocus noise in OSH, we have also evaluated the reconstruction results of two different 3D objects in this subsection. The performances between the conventional reconstruction method and the proposed one are also analyzed.

The first 3D object used in the simulation contains two slices, as is shown in Fig. 11(a). Each slice has a size of $1\mathrm{mm}\times 1\mathrm{mm}$ and are sampled to $256\times 256$ pixels. The focal length of the lens in the optical system is set as 75 mm. The locations of the two slices are $z_1=9\mathrm{mm}$ and $z_2=10\mathrm{mm}$, respectively. The wavelength of the laser is 632.8 nm. Figure 11(b) shows the recorded hologram.

Figures 12(a) and 12(b) show the retrieved sectional images using the conventional algorithm^[16], with reconstruction distance at $z_1=9\mathrm{mm}$ and $z_2=10\mathrm{mm}$, respectively. One can see from this figure that at the front plane ($z_1=9\mathrm{mm}$), the slice “UESTC” is in focus, and the image from the other section has become defocused noise spreading around the figure. This speckle-like noise obviously degrades the image quality, which results in a poor 3D reconstruction effect. While the corresponding results of the proposed U-net based method are shown in Figs. 12(c) and 12(d), in which each section has been recovered clearly with degraded defocus noise.

We have also tested the U-net method with a simulated hologram of a rocket. The semi-transparent 3D rocket is shown in Fig. 13. It has been divided into six uniformly separated sections along the $z$-axis. The $i$th section is located $z_i$ away from the focal plane, with $z_1=31\mathrm{mm}$ and $z_6=36\mathrm{mm}$. Each slice has a size of $1\mathrm{mm}\times 1\mathrm{mm}$, and are sampled to $256\times 256$ pixels, as can be seen in Figs. 14(a)–14(f).

The reconstructed images with the traditional method are shown in Figs. 14(g)–14(l), while the ones with the U-net-based method are shown in Figs. 14(m)–14(r). The corresponding reconstruction distances are set as $z_1=31\mathrm{mm},z_2=32\mathrm{mm},\dots$, and $z_6=36\mathrm{mm}$. It can be seen from these images that the proposed method outperforms the traditional method in suppressing the defocus noise.

The quantified assessment results of each section are also analyzed. The results are shown in Figs. 15 and 16, in which number of section denotes the $i$th sectional image. One can observe from Fig. 15 that with the traditional method, the PSNR drops from 16 dB at Section-1 to 11 dB at Section-6. This is caused by the defocus noise from other sections. While with the U-net method, the output PSNR have been upgraded to around 35 dB, which indicates the improvement of the image quality. The measured SSIM results of each method are shown in Fig. 16. One can find that the value of the SSIM are all above 0.9 with the proposed U-net method. While for the case with traditional method, the SSIM decreases from 0.36 at Section-1 to 0.28 at Section-6. This also implies better sectioning results with the U-net method.

In conclusion, the U-net based method can be adapted to 3D objects with multiple sections. It outperforms the traditional method in removing defocus noise as well as recovering multiple sections. One can also deduce from the tables and figures in section 3 that the improvement of the PSNR ranges from around 5 dB to 20 dB, which indicates better sectioning results over the conditional method. However, the scenario of the object with complex sectional images still needs further investigation. This can be done either by increasing the training data sets or adjusting the training mode in the deep learning algorithm. These are all the future tasks to be done.

Speckle-like noise is generated by the random phase pupil in an OSH system, which is hard to reduce. This work verifies the feasibility of the U-net neural network for this task, which provides a new way for us to realize fast and effective defocus noise suppressing in OSH. Simulation results show that the proposed method works well both in simple and complex graphics. We believe that the proposed method can also be applied to other digital holography systems, especially for biomedical applications where it is hard to get enough training data sets.