Nowadays, vortex beams (VBs), carrying orbital angular momentum (OAM), have been known as the hot spots and shown great potential in various fields such as optical micro-manipulation, optical sensors, optical transmission, optical lasers, and amplifiers^[1]. In 1992, Allen et al. indicated that for beams with helical phase fronts of $\exp (il\varphi )$ the OAM in the propagation direction has the discrete values of $lh$ per photon, where $h$ is the reduced Planck constant, and $l$ is usually an integer that represents the topological charge (TC), whose sign indicates the direction of the phase spiral^[2]. In theory, the VBs with different OAM modes are mutually orthogonal, and their intensity distribution presents unique “doughnut” shapes due to phase uncertainty at the beam cross section^[3]. The discovery of optical vortices carrying OAM has accelerated progress in many research areas^[4]. Especially in optical communications, VBs are of great potential in increasing capacity and modulation ability by providing additional physical dimension^[5]. By multiplexing VBs, the transmission rate of Tbit/s is realized in free-space optical (FSO) communication^[6,7]. Also, further studies have shown that hybrid multiplexing of OAM and other dimensions can achieve communication capacity close to Pbit/s^[8]. However, the OAM shift keying communication, which encodes digital signals to OAM modes, is severely hampered because the locking effective detection methods can hardly adapt to the fast switching modes. Moreover, the VBs are susceptible to atmospheric turbulence (AT), which may cause wavefront distortion and lead to mode diffusion^[9]. Hence, fast and high-accuracy OAM mode recognition is a challenge in the practical application of VBs.

Many schemes have been proposed to detect OAM modes. The most common schemes are interference and diffraction methods^[10]. Through the interference of VBs with the Gaussian beams or planar wave, the TC information can be extracted from the interference fringes. There are two main schemes for using machines to recognize interference fringes: traditional machine learning and deep learning technology^[11]. Unfortunately, the effects of AT undermine the performance of simple image recognition technologies, which causes poor recognition accuracy. Meanwhile, most machine learning schemes are based on convolutional neural network (CNN)^[12], which usually requires complex network architecture and huge computational cost. Therefore, the combination of reduction in computational complexity and anti-turbulence capability has become top priority.

In this article, we proposed and designed an OAM mode recognition method based on an interferogram processed by a Gaussian smoothing filter (GSF) to identify the OAM modes of VBs. This method is called vortex interferogram recognition (VIR) based on GSF, which is abbreviated as VIR-GSF in this article. Performing simple image feature extraction on the VB light field, this scheme realizes mode recognition. Without a complex convolution and deconvolution process, the computational cost is greatly reduced. Since GSF can strengthen the stability of optical field distribution and reduce the variance of intensity fluctuations, by adding GSF to process the interferogram between the Gaussian beam and the VB, high identification accuracy for the distorted image disturbed by turbulence can be obtained. The calculation results show that this scheme has high accuracy for 1 km transmission in turbulent channels. Specifically, in no-turbulence, weak-turbulence ($C_n^2=1\times {10}^{-16}{\mathrm{m}}^{-2/3}$), and medium-turbulence ($C_n^2=1\times {10}^{-15}{\mathrm{m}}^{-2/3}$) conditions, the identification rate is up to 100%. Under strong-turbulence ($C_n^2=1\times {10}^{-14}{\mathrm{m}}^{-2/3}$) conditions, the recognition accuracy can reach 91.24%. Moreover, the proposed scheme is compared with two reference schemes, i.e., the traditional CNN algorithm and the CNN with Gerchberg–Saxton (GS) algorithm^[13], and our scheme can achieve better performance.

At the receiver end of the OAM communication system, various OAM modes of the VB need to be demodulated. High-accuracy recognition of OAM modes is a necessary prerequisite for accurately distinguishing different modes. In the proposed scheme, mode recognition is accomplished based on the phase information carried by the VBs^[14]. Firstly, the VBs interfere with the Gaussian beam to convert the phase information into intensity distribution, and they use a photon detector to detect the intensity signals^[15]. In this process, we need to collect both intensity images and interferograms. The principle of interference between the VB and Gaussian beam is shown in Fig. 1, and the expression of the interference optical field $E$ is^[16]$$E=E_{\mathrm{LG}}+E_{\mathrm{TEM}00}.$$(1)

Among them, the first item $E_{\mathrm{LG}}$ is used to generate the Laguerre–Gaussian beam, which is one of the classic VBs. The cylindrically symmetric solutions $E_{\mathrm{LG}}(r,\phi ,z)$ are given by^[2,17]$$\begin{gathered} E_{\mathrm{LG}}(r,\phi ,z)=\sqrt{\frac{2p!}{\pi (p+|l|)!}\frac{1}{{\omega }^2(z)}}{\left[\frac{\sqrt{2}r}{{\omega }^2(z)}\right]}^{|l|}{L}_p^{|l|}\left[\frac{2r^2}{{\omega }^2(z)}\right] \\ \times \exp [i(2p+|l|+1)\arctan \left(\frac{z}{z_R}\right)] \\ \times \exp [-\frac{r^2}{\omega (z)}-\frac{ik{r}^{2}z}{2R(z)}]\exp (il\phi ). \end{gathered}$$(2)

The second item $E_{\mathrm{TEM}00}$ is the optical field distribution of the fundamental mode Gaussian beam, and its expression is as follows: $$\begin{gathered} E_{\mathrm{TEM}00}(r,\phi ,z)=\sqrt{\frac{1}{{\omega }^2(z)}}\exp [-\frac{r^2}{\omega (z)}-\frac{ik{r}^{2}z}{2R(z)}] \\ \times \exp \left[i\arctan \left(\frac{z}{z_R}\right)\right]. \end{gathered}$$(3)

In the above expressions, $l$ and $p$ are the order of the OAM mode and radial index, respectively, the wavenumber $k=2\pi /\lambda$ with $\lambda$ as the wavelength, and $L_p^{|l|}(·)$ is the generalized Laguerre polynomial. ${\omega }_0$ is the size of the beam waist, $\omega (z)={\omega }_0\sqrt{1+{(z/z_R)}^2}$ is the radius of the beam, and $z_R=\pi {\omega }_0^2/\lambda$ is the Rayleigh range. $\phi$ is the azimuth angle, and $R(z)$ describes the curvature of the spherical wave near the optical axis.

After the realization of Eqs. (1) and (2), the intensity images $F_1$ and interferograms $F_2$ of VBs are obtained. Further, we adopt an intensity map to calculate the average amplitude value of light intensity on the ring with different radii from the center outward. Then, the $\mathrm{MAX}\{·\}$ function is used to get the $r$ value corresponding to the largest average amplitude. Each pair of interference and intensity images is from the same VB, so their intensity and phase have a one-to-one correspondence. We utilize $r$ of the maximum amplitude in the interference image to find a circle with a radius of $r$ and extract the intensity value on the circle corresponding to the radius $r$ to create a one-dimensional matrix $M_r^{\prime }$. The waveform is plotted along a one-dimensional matrix, and the number of fluctuations of the waveform represents the absolute value of the OAM mode $|l|$. Specifically, $|l|$ is equal to the number of intersections between the average intensity $A_{(r)}$ and the falling edge of the waveform $M_r^{\prime }$. The detailed recognition principle of this scheme is explained in Fig. 2. Next, we need to determine the sign of the OAM modes. We extract another one-dimensional matrix $M_{r+\mathrm{\Delta }r}^{\prime }$ from the ring with radius $r+\mathrm{\Delta }r$ in the same interferogram, where $\mathrm{\Delta }r$ needs to satisfy the following conditions: $$0<\mathrm{\Delta }r<\sqrt{2\pi ·R(z)/(k·z)}.$$(4)

Equation (4) guarantees that the phase difference $\phi$ between matrices $M_r^{\prime }$ and $M_{r+\mathrm{\Delta }r}^{\prime }$ satisfies $-\pi /2<\phi <\pi /2$. If $\phi >0$, the sign of the OAM mode is positive. If $\phi <0$, the sign is negative. Combining the above two steps, the OAM state can be determined jointly by the absolute value and the sign. However, the waveforms will be influenced by AT. So, we simulated turbulence and added this artificial turbulence into the beam propagation process. The influence of AT on the OAM beam can be approximate to the effect caused by a series of random phase screens along the propagation path. Among them, every random phase screen is a thin sheet that can change the phase distribution of the transmitted beam, and its perturbation of the refractive index is approximately represented by the Kolmogorov spectrum^[18]. The power spectral models of refractive-index fluctuations in each phase screen can be expressed as $\varphi (k_x,k_y)$^[17]: $$\begin{gathered} \varphi (k_x,k_y)=0.033C_n^2[1+1.802\sqrt{\frac{k_x^2+k_y^2}{k_l^2}} \\ -0.254{\left(\frac{k_x^2+k_y^2}{k_l^2}\right)}^{7/12}] \\ \times \exp \left(\frac{k_x^2+k_y^2}{k_l^2}\right){\left(k_x^2+k_y^2+\frac{1}{L_0^2}\right)}^{-11/6}, \end{gathered}$$(5)where $C_n^2$ is the structure constant of the refractive index (in units of ${\mathrm{m}}^{-2/3}$), which is used to describe the AT intensity. $k_x$ and $k_y$ are the wavenumber in the $x$ and $y$ directions, respectively. The $L_0$ and $l_0$ are the outer scale and inner scale of AT, $k_l=3.3/l_0$. Further, the fluctuation spectrum will affect the wavefront phase of the OAM beam. The variance of the phase spectrum can be written as $${\psi }^2(k_x,k_y)=2\pi \mathrm{\Delta }z·(k_x^2+k_y^2)·\varphi (k_x,k_y)·{\left(\frac{2\pi }{N\mathrm{\Delta }x}\right)}^2,$$(6)where $N$ and $\mathrm{\Delta }x$ represent the size and grid spacing of the random phase screens, respectively. What is more, $\mathrm{\Delta }z$ is the interval distance between adjacent phase screens. To facilitate the calculation, the random distribution of phase perturbation is described in the Cartesian coordinate system, and the phase screen is then expressed in the frequency domain through the fast Fourier transform (FFT) operation. $\theta (x,y)$ is an $N\times N$ phase screen ($N=500$), which can be expressed as follows: $$\theta (x,y)=\mathcal{F}\{C\psi (k_x,k_y)\},$$(7)where $\mathcal{F}\{·\}$ is FFT, $C$ is a complex random matrix with a mean of zero and a variance of one. Through the above turbulence simulation model, the intensity maps after AT disturbance are obtained. Next, after the VB interferes with the Gaussian beam, the interferogram is eventually obtained. The principle of the VB generation and interference under turbulence is shown in Fig. 3.

In order to reduce the influence of random scattering on the VBs, we added GSF in the recognition process to optimize the waveform jitter of the matrices $M_r^{\prime }$ and $M_{r+\mathrm{\Delta }r}^{\prime }$. It should be emphasized that executing GSF has great significance for extracting waveforms, which can effectively improve the recognition accuracy. The specific operation of GSF is calculating the weighted average of the pixel values within the neighborhood and then saving the value to new matrices ${\overline{M}}_r^{\prime }$ and ${\overline{M}}_{r+\mathrm{\Delta }r}^{\prime }$. The Gaussian smoothing can be described as $${\overline{y}}_j=\frac{1}{k}\sum _{i=-m}^m\exp [-\frac{{(x_i-x_j)}^2}{2{\sigma }^2}]y_i,0<j<n,2m<n,$$(8)with $$k=\sum _{i=-m}^m\exp [-\frac{{(x_i-x_j)}^2}{2{\sigma }^2}],$$(9)where $\sigma$ is the smoothing factor, and ($x_i,y_i$) are elements in matrix $M_r^{\prime }$. $n$ is the size of matrix $M_r^{\prime }$, and $m$ represents the smoothness level. When $m$ increases, the GSF’s ability to filter noise will become stronger, but the loss of features that are helpful with identifying the waveform will also be more obvious. Therefore, choosing an appropriate $m$ determines the success rate of mode recognition. In this study, we took two cases of $m=15$ and $m=25$ as examples and compared their effectiveness. The comparison results of smoothness are shown in Fig. 4. From the results, we find that the original waveform is unstable because the received beam additionally carries some random disturbance and noise. By performing GSF, sawtooth waves and noise signals are reduced. The waveform of matrix ${\overline{M}}_r^{\prime }$ becomes smoother and more stable. However, an unlimited increase in $m$ is disadvantageous, and a larger $m$ will bring greater loss of effective information. So, we finally chose $m=25$ as the smoothness level.

Combining our proposed VIR scheme and using GSF can realize OAM mode recognition under AT environment. As a brief summary, we outline the proposed VIR-GSF scheme in Algorithm 1.

In the numerical simulation, we used five levels of AT intensity: no turbulence, weak turbulence, medium turbulence, medium-strong turbulence ($C_n^2=5\times {10}^{-15}{\mathrm{m}}^{-2/3}$), and strong turbulence. The transmission distance used two cases of 1 km and 2 km. Figure 5 shows the image sets at 1 km, and the number of images in each mode is 1000. These images will be used to test the accuracy of the proposed scheme.

When the transmission distance is 1 km, the recognition accuracy of the VIR-GSF scheme is shown in Fig. 6. Under the conditions of no turbulence, since the beam is stable, and its phase distribution is uniform, the mode identification accuracy can reach 100%. When the AT intensity is increased to weak and medium levels, recognition accuracy can be maintained at 100%, and the results are shown in Fig. 6(a). Further increasing the AT intensity to medium-strong and strong levels, the average accuracy can still reach 98.02% and 91.24%, as shown in Fig. 6(b).

When the transmission distance is 2 km, the test results are shown in Figs. 7(a)–7(c). The results show that the accuracy can still be maintained at 100% under no-turbulence, weak-turbulence, and medium-turbulence conditions. Upgrading the turbulence level to medium-strong and strong degrees, the accuracy can reach 96.36% and 87.73%, respectively.

In addition, the proposed VIR-GSF scheme is compared with two reference methods of CNN and CNN-GS. The traditional CNN algorithm includes the input layer ($N\times N$), convolutional layer (CL) 1 ($28\times 28$), pooling layer (PL) 1 (max pooling, $12\times 12$), CL 2 ($8\times 8$), PL 2 ($4\times 4$), fully connected layer (FCL) 1 (50 nodes), FCL 2 (50 nodes), and output layer. The CNN-GS scheme adds the GS algorithm basis on the traditional CNN scheme. The comparison results of average accuracy are shown in Fig. 7(d). In this case of weak turbulence, their recognition accuracy can reach 100%. When increasing turbulence intensity from the weak to the medium level, the accuracy of the CNN scheme drops to 95.6%. In a strong-turbulence environment, the recognition accuracies of CNN, CNN-GS, and VIR-GSF are 74%, 81.34%, and 87.73%. Therefore, our proposed VIR-GSF scheme has better performance than the CNN and CNN-GS schemes.

In this Letter, an OAM modes recognition method based on an interferogram was proposed. This method realizes the OAM modes recognition by computing the number of intersections between the average value and the falling edges of the waveform. Additionally, the GSF algorithm is adopted to reduce the influence of AT on OAM-FSO communication. The VIR-GSF scheme can achieve higher recognition accuracy in turbulent environments without using neural networks. This scheme can greatly save computational resources and reduce system complexity. It is anticipated that our work might be helpful for further improving the reliability of the OAM-FSO communication system.