In practical outdoor free-space optical (FSO) communication systems, wavefront aberration is usually observed due to the atmospheric turbulence that causes random fluctuations in the air’s refractive index^[1–3]. At a lens-based optical receiver, extra loss is induced by low coupling efficiency of the aberrated light beam, which is coupled into a standard single mode fiber (SMF) followed by an optical detector/amplifier. To reduce the turbulence-induced power loss, the traditional adaptive optics (AO) is applied to compensate the wavefront aberration^[4–6], and this normally requires a wavefront sensor (WFS) for detection of the wavefront aberration. However, wavefront measurement becomes a challenge in the strong turbulence, as the method requires many sensing and control channels in the WFS for an aperture larger than the Fried parameter^[7].

To address this challenge, the WFS-less AO was investigated with blind optimization algorithms including simulated annealing (SA) and stochastic parallel gradient descent (SPGD)^[6–8]. In such WFS-less AO systems, the wavefront aberration is compensated iteratively to obtain a target performance metric, which is related to the laser beam quality, such as the far-field intensity distribution^[6]. Thousands of iterations and performance metric calculations are usually needed for converged performance.

As an emerging technology, deep learning has been recently studied for wavefront sensing/compensation in biomedicine, astronomy, and FSO systems with Gaussian or vortex beams^[9–16]. A WFS may not be required, as deep learning was applied to estimate the wavefront aberration directly from intensity images captured by a camera^{[11–13,15,16]}. These deep learning-based solutions include the convolutional neural network (CNN)^[11–13], back propagation (BP) artificial neural network^[15], and reinforcement learning with the Markov decision process^[16]. The CNN was applied on a point-spread function to achieve a good preliminary estimate, which improved the convergence performance (rate)^[10]. For the FSO applications, an optical feedback network making use of a CNN and a gradient descent optimizer was designed to mitigate the turbulence effect by adjusting the optical mode profiles at the transmitter^[17]. However, such a solution may confront the challenge of large latency in BP and image processing. In the above literature, most investigations were made by simulation irrespective of the practical components’ limitations, including image quality and coupling efficiency.

We experimentally demonstrated a solution of wavefront compensation with an AlexNet-based CNN without discussions about key limitation factors in FSO systems^[18]. To further study the effectiveness of the CNN-based solution for FSO system, in this Letter, an extended investigation is made on the impact of practical limitation factors on the aberration compensation performance. These factors include quantization resolution or noise of received intensity images, the number of Zernike modes (polynomials), and structure of the CNN. We construct the wavefront aberrations in a statistical way such that the light beams at the receiver can be modeled with Zernike modes in the Kolmogorov turbulence. To evaluate the CNN-based solution, the power penalty of the received light beam coupled into an SMF is studied in experiment.

Figure 1 depicts a block diagram of an AO system with deep learning for FSO communication. There is atmospheric turbulence-induced wavefront aberration for the received light beam $\varphi (\rho ,\theta )$. $\rho$ and $\theta$ are the normalized polar radius and angle, respectively. Without compensation of the wavefront aberration, the aberrated light beam after the lens may not be efficiently coupled into an SMF-pigtailed optical detector (photodiode) or amplifier, thus causing a large power penalty/loss. Therefore, a wavefront corrector is applied to compensate the wavefront aberration estimated by the deep learning solution at the receiver. Residual wavefront may exist after the aberration compensation due to inaccurate estimation of the aberration. With beam splitters (BSs), a fraction of the light beam is captured by a camera, whilst the remainder of the light beam is coupled into an optical fiber of the optical detector. At the initial stage of the FSO link establishment, a deep learning network is trained with intensity images and phase, which are detected by the camera and WFS, respectively. After network training, the WFS may not be necessary, so only the camera is required.

The wavefront aberration is usually modeled with Zernike modes/polynomials in the atmospheric turbulence based on the Kolmogorov theory^[19,20]. Orthogonal Zernike modes/polynomials are used to form an aberrated wavefront, $\varphi$: $$\varphi (\rho ,\theta )=\sum _{k=1}^K{a}_k{Z}_k(\rho ,\theta ),$$(1)$$Z_k=\{\begin{array}{lll}\sqrt{2(n+1)}{R}_n^m(\rho )\cos m\theta ,& m\ne 0,& k\text{is even},\\ \sqrt{2(n+1)}{R}_n^m(\rho )\sin m\theta ,& m\ne 0,& k\text{is odd},\\ \sqrt{(n+1)}{R}_n^0(\rho ),& m=0& ,& \end{array}$$(2)$$R_n^m(\rho )=\sum _{s=0}^{(n-m)/2}\frac{{(-1)}^s(n-s)!{\rho }^{n-2s}}{s![(n+m)/2-s]![(n-m)/2-s]!},$$(3)where $a_k$ is the $k$th Zernike mode coefficient. $m$ and $n$ are integers ($m\le n$, $n-|m|=\mathrm{even}$). $K$ is the number of modes. The Zernike coefficients are normally treated as zero-mean Gaussian random variables. However, the Zernike coefficients are not independent. To statistically model the atmospheric turbulence with correlated random variables, the covariance between coefficients $a_i$ and $a_j$ can be written as below^[21]: $$\begin{gathered} E(a_i,a_j)=\frac{2.2698·{(-1)}^{(n+n^{\prime }-2m)/2}{\delta }_{m{m}^{\prime }}}{\mathrm{\Gamma }[(n-n^{\prime }+17/3)/2]}·{\left(\frac{D}{r_0}\right)}^{5/3} \\ ·\frac{\sqrt{(n+1)(n^{\prime }+1)}·\mathrm{\Gamma }[(n+n^{\prime }-5/3)/2]}{\mathrm{\Gamma }[(n^{\prime }-n+17/3)/2]·\mathrm{\Gamma }[(n+n^{\prime }+23/3)/2]}, \end{gathered}$$(4)$${\delta }_{m{m}^{\prime }}=(m=m^{\prime })\wedge [\text{parity}(i,j)\vee (m=0)],$$(5)where $m$ ($m^{\prime }$) and $n$ ($n^{\prime }$) denote the azimuthal degrees and radial frequencies of $Z_i$ ($Z_j$), respectively. ${\delta }_{m{m}^{\prime }}$ is a logical Kronecker symbol. $\mathrm{\Gamma }(·)$ is a gamma function. $D/r_0$ represents atmospheric turbulence strength. $D$ is receiver aperture diameter, and $r_0$ denotes atmospheric coherent length. parity$(i,j)$ is one if the same parity occurs for indices $i$ and $j$, otherwise, it is zero. The ‘$\wedge$’ and ‘$\vee$’ are logical operators of ‘and’ and ‘or’, respectively. The singular value decomposition (SVD) of the covariance matrix from Eq. (4) is applied to calculate a unitary matrix and a diagonal matrix for the independent Gaussian random variables generation. As a result, a random atmospheric wavefront is generated.

By applying the CNN, the aforementioned wavefront aberration can be estimated with intensity images of the received light beam, ${|\mathcal{F}\{E_r\exp [j\varphi (\rho ,\theta )]\}|}^2$. $E_r$ is the electric field of the arriving beam. $\mathcal{F}(·)$ represents two-dimensional Fourier transform. A typical CNN with a structure of AlexNet in Refs. [22,23] is applied in Fig. 1. There are eight layers in the AlexNet structure, including the first five convolutional layers and the last three fully connected layers. The role of the convolutional layers is to extract features of the input images. The output of the convolution operation is connected to the activation functions (rectified linear unit) for the final output feature map. The task of the AlexNet-based CNN is to search the mapping between Zernike coefficients vector $\mathit{A}=[a_1,a_2,\dots ,a_K]$ and intensity images. In order to accurately estimate the wavefront aberration, a number of images and the corresponding Zernike coefficients are used to train the CNN. In the training stage, the network weights of the CNN are updated by the Adam algorithm for the gradient-based stochastic optimization [10]. The loss function is written as $L_C=E(|\mathit{A}-{\mathit{A}}_e|)$. $E(·)$ stands for expectation, and ${\mathit{A}}_e$ means the estimated Zernike coefficients vector.

To evaluate key factors affecting the aberration compensation performance, investigations are made on the impacts of the number of Zernike modes, quantization resolution (noise) of images, and CNN structure on power penalty in the simulation. As the first three modes ($Z_1–Z_3$) can be easily corrected, Zernike modes ($Z_4–Z_K$) are used to simulate the turbulence-induced wavefront aberration^[24,25]. During the training stage, the turbulence strength ($D/r_0$) is 0–16 for the mixed weak/strong turbulence. The power penalty is defined as the optical power difference measured at the optical fiber’s end with and without the turbulence. The number of estimated Zernike vectors is 50 for statistical study of the power penalty. These parameter values are set by default unless specified explicitly. As shown in Figs. 2(a) and 2(b), the power distribution is plotted as a function of the number of Zernike modes. Even though the first three modes are not considered, the first 10 and 21 modes account for 85.15% and 93.40% of the total energy, respectively. The main energy is distributed on the low-order modes. The power starts to converge towards one for $K>10$.

The wavefront compensation performance with different numbers of Zernike modes is analyzed in the strong turbulence ($D/r_0=16$). To analyze the CNN-based estimation performance for different mode counts, the number of Zernike modes estimated with CNN is no larger than 45 randomly generated Zernike modes. It is seen from Figs. 2(c) and 2(d) that the test error with the AlexNet-based CNN increases with the number of Zernike modes. The average power penalty for $K=10$ is slightly higher than that for $K=45$. This indicates that the test error may not be able to directly represent the wavefront compensation performance. The power penalty is essential to indicate power loss/efficiency at the receiver, which is the focus of this study. In addition, Fig. 2(d) shows a small variation in mean and standard deviation of power penalty estimated with the CNN even when some Zernike modes are not estimated. Such small variation is because of a majority of the power on the low-order modes. This reveals that a small number of Zernike modes can be used for the low-complexity purpose. Therefore, $K$ is set to 10 in the following study.

The impact of quantization bits on the wavefront compensation performance with the CNN is presented in Figs. 2(e) and 2(f) when $D/r_0=16$. The trends of the varied test error in the CNN for different quantization resolutions are almost the same. The average power penalty slightly decreases with increasing quantization bits. The variance in the power penalty for 4 bit resolution is slightly larger than that for 8/12/16 bit resolutions, which have almost the same performance. This is because the CNN has an advantage of combating quantization noise with a large amount of input data. Therefore, the optimum number of quantization bits of eight is suggested.

The CNN structure varies for different applications. We analyzed three different network structures: 4-layer LeNet, 8-layer AlexNet, and 19-layer VGG19. It is shown in Figs. 2(g) and 2(h) that AlexNet and VGG19 significantly reduce the power penalty for a number of Zernike modes up to 45 in the mixed weak/strong turbulence case ($D/r_0=0–16$). The average and deviation of power penalty increase with the number of Zernike modes, whilst the difference between AlexNet and VGG19 is very small. Before the CNN processing, the average and deviation remain relatively high. It is noted that for a large number of Zernike modes the simple structure of LeNet fails to estimate the wavefront aberration, which causes a large power penalty comparable to that before CNN. Considering the high complexity of VGG19, AlexNet with relatively low complexity has almost the same performance as VGG19. Therefore, the AlexNet-based CNN is chosen as a suitable solution in our experimental study.

Apart from the comparison among different CNN structures, traditional solutions of wavefront compensation (SPGD or SA)^[6–8] were utilized for performance comparison. As shown in Fig. 3, the power penalty with SPGD and SA sharply increases with the turbulence strength factor ($D/r_0$). For the weak turbulence case when $D/r_0\le 6$, the power penalty with the AlexNet-based CNN is very close to that with SPGD/SA solutions. However, the difference in power penalty increases for large $D/r_0$ in the relatively strong turbulence case ($D/r_0>6$). This suggests that in the strong turbulence case the CNN outperforms the SPGD and SA. As there is no requirement of iterations, the CNN solution can be used to compensate wavefront aberration faster than the traditional solutions in the especially strong turbulence case after the CNN is trained.

Figure 4 depicts an experimental setup of a CNN-based AO system and its corresponding block diagram for FSO applications. A standard fiber-pigtailed distributed feedback (DFB) laser operating at a wavelength of 1550 nm is used to emit a Gaussian beam. The DFB laser was connected to a collimator, half-wave plate, and polarizer such that vertically polarized transmission was realized as required by a phase-only spatial light modulator (SLM). The propagation beam diameter is 7 mm. Random phase patterns produced with Zernike modes were introduced by the SLM with a pixel count of $1920\times 1080$ and a pixel pitch of 8.0 µm (Holoeye PLUTO-TELCO-013) in order to generate wavefront aberration for different atmospheric turbulence cases. The SLM can also be used for compensation of the wavefront aberration that is estimated with the CNN. A BS with a split ratio of 50:50 was utilized to split the received light into two beams, each of which reaches an infrared charge-coupled device (CCD) camera ($640\times 512$ pixels, Bobcat-640-GigE) or an SMF. The laser’s launch power was 11.8 dBm. The batch size is 16.

For training the AlexNet-based CNN, 16,000 datasets of wavefront aberrations together with a number of epochs were produced with different atmospheric turbulence strength values. The intensity images corresponding to the wavefront aberrations were captured by the CCD camera. The training dataset consisting of wavefront aberrations and corresponding intensity images were sent to the CNN for the TensorFlow-based training. Another 2000 datasets were used to calculate test errors. Once the AlexNet-based CNN is trained, we can estimate the Zernike coefficients for the wavefront aberrations, the conjugation of which was used for the wavefront compensation at the SLM.

In order to evaluate the loss performance in the training stage, the curves of the loss versus epochs are shown in Fig. 5(a), where both experimental and numerical (simulation) results are presented for comparison. An epoch means one pass of a full training set. As seen in the figure, the simulated loss performance converges faster than experimentally measured loss. A minimum of 1 and 5 epochs are observed for converged performance in the simulation and experiment, respectively. This may be due to the background noise of the CCD camera with limited amount of pixels in the experiment.

As an example, a random Zernike coefficient vector is used for generation of a wavefront aberration ($D/r_0=16$) in Fig. 5(c). We observed a dispersed speckle in the camera, as shown in Fig. 5(d). In order to understand the experimental phenomenon, the CNN-based wavefront compensation was also numerically investigated for comparison in Fig. 5(b). The Zernike coefficients estimated in simulation and experiment are almost identical to the theoretical values. The calculated absolute errors of Zernike coefficients are smaller than 0.25. A circular speckle was thus observed after the wavefront compensation in Fig. 5(d).

The performance of the CNN-based wavefront compensation was further investigated statistically with 50 random Zernike vectors for different wavefront aberrations in experiment. The relatively weak or strong ($D/r_0=4$ or 16) turbulence cases were considered in Fig. 6, where the box plot of the measured power penalty denotes its range and mean. Thanks to the CNN-based wavefront compensation, the mean/variance (power penalty) significantly decreases. The mean of the power penalty reduces to 1.8 (0.8) dB from 12.4 (4.4) dB for $D/r_0=16$ (4) in the strong (weak) turbulence. In the inset of the figure, the constellation of power penalty versus the root mean square (RMS) of wavefront errors is given to further understand the result. Although the constellation points are dispersed, the power penalty is nearly proportional to the RMS wavefront errors. However, the variance of the power penalty increases for large RMS. The dispersed constellation suggests that the performance of the CNN-based wavefront compensation may not be well indicated with the RMS for FSO applications. Given the difficulty in measuring wavefront aberrations, the optical power can be monitored as a simple way for indicating the wavefront compensation performance.

An AlexNet-based CNN for the atmospheric turbulence-induced wavefront aberration compensation for FSO applications has been investigated in experiment and simulation. Statistically constructed wavefront aberrations were used for the aberrated light beams with the Zernike modes (polynomials) theory. To explore the effectiveness of the AlexNet-based CNN solution, we have analyzed three key factors affecting the compensation performance, which include CNN structures, Zernike mode count, and quantization resolution (noise) of intensity images. AlexNet was identified as an appropriate CNN structure by considering a trade-off between the power penalty and complexity. Experimental results indicate that the power penalty reduces to 1.8 dB from 12.4 dB for the strong turbulence case. As one of the key challenges for outdoor FSO applications, the background noise and/or inference will be studied in the future.