Polarization division multiplexing plays an important role to double the spectral efficiency for high speed coherent optical fiber transmissions. The polarization de-multiplexing algorithms can be divided into blind equalization and data-aided multi-input multi-output (MIMO) algorithms. The blind algorithms are mainly based on the characteristic of constellation points, such as constant modulus algorithm (CMA), multi-modulus algorithm (MMA), and cascaded MMA (CMMA), together with their variants^[1–3]. These kinds of blind algorithms are based on the constant modulus or certain radii, which enables de-multiplexing in the presence of laser phase noise, frame asynchronism, and inter-symbol interference^[4]. However, these blind algorithms have low convergence speed, singularity problem, and step-length sensitivity^[5] due to the intrinsic shortcoming of such modulus-based cost functions. Besides, the independent component algorithm (ICA) is also effective for blind source separation^[6,7], which solves the singularity^[8] but has the inherent polarization alignment problem^[9]. The singularity problem can be solved by Stokes space estimation or as the initialization of the elements in the Jones matrix for the CMA-based blind equalizations. However, the fast rotation of state of polarization (RSOP), whose speed may exceed several hundred mega radians per second in some extreme cases^[10], may disable the conventional MIMO fiber system and bring enormous challenges to existing digital signal processing.

The Kalman filtering (KF) has been widely used in system control, which has been introduced into optical communications in recent years. KF has both blind^[11,12] and data-aided two-stage equalization forms^[13] for both coherent and direct detection schemes^[14,15]. Blind KF avoids the decision-error-induced divergence during iteration, but is modulation format dependent. The two-stage data-aided KF uses the training sequence at the first stage and decision-directed tracking at the second stage, enabling general de-multiplexing under all kinds of modulation formats without singularity or channel logical confusion at the first stage. However, the performance deterioration of the second stage may occur in the presence of decision error, especially for low SNR cases. In our previous work, we have proposed modified KF (MKF) with higher speed of convergence and stability by modifying the measurement equation of the conventional KF (CKF)^[14]. However, the improvements from measurement equation modification have not been analyzed, which will be beneficial for different kinds of KF.

In this paper, we analyze the noise tolerance of CKF and MKF in theory by introducing conditional variance of measurement residual and process noise. We find that the enhanced noise tolerance of MKF compared with CKF results from the decrease of the mean square error in each iteration by numerical simulations. Besides, the covariance reduction indicates the noise tolerance improvement of MKF, which results in better tracking ability and covariance initialization flexibility. Furthermore, we find the noise covariance matching method can be smoothly combined with MKF to adaptively update the covariance matrices of measurement noise and process noise. We call it adaptive MKF corresponding to adaptive CKF. The simulation comparison between adaptive CKF and adaptive MKF is conducted in virtual photonics instrument (VPI)-MATLAB co-simulation on a single carrier quadrature amplitude modulation (QAM)-16 coherent optical fiber transmission system. Compared with adaptive CKF, adaptive MKF has $>3\mathrm{dB}$ optical SNR (OSNR) improvement at the scrambling rate of 10 MHz, and around 73.5% scrambling rate improvement at the OSNR of 24 dB. Particularly, the adaptive MKF retains the successful joint tracking of the fast RSOP and laser phase noise in the lower OSNR cases. For higher OSNR, either the MKF or adaptive MKF outperforms CKF or adaptive CKF in initialization flexibility.

Before the comparison, we denote the channel model and variables of the transmission system. The polarization diversity multiplexing (PDM) coherent optical transmission is assumed to be linear and the discrete-time form of the received signal $y(n)$ after dispersion compensation can be expressed as $$y(n)=H_{\mathrm{CD}}^{-1}\{H_{\mathrm{CD}}[\mathit{H}x(n)+N]\},$$(1)where $x(n)$ is discrete-time form of the transmitted signal, $N$ is the additive white Gaussian noise (AWGN), and $H_{\mathrm{CD}}(·)$ and $H_{\mathrm{CD}}^{-1}(·)$ denote the fiber dispersion and its digital-domain compensation. The channel response matrix $\mathit{H}$ representing laser phase noise and RSOP can be expressed as $$\mathit{H}=\left[\begin{array}{cc}{H}_{\mathit{xx}}& H_{\mathit{xy}}\\ H_{\mathit{yx}}& H_{\mathit{yy}}\end{array}\right]=\left[\begin{array}{c}\cos \kappa (n){\mathrm{e}}^{j\xi (n)}-\sin \kappa (n){\mathrm{e}}^{j\eta (n)}\\ \sin \kappa (n){\mathrm{e}}^{-j\eta (n)}\cos \kappa (n){\mathrm{e}}^{-j\xi (n)}\end{array}\right]{\mathrm{e}}^{j\varphi (n)},$$(2)where $\varphi (n)$ is the Wiener laser phase noise, and the RSOP can be characterized with a three-parameter model denoted by the amplitude ratio angle $\kappa$ and the phase angles $\eta$ and $\xi$^[16–18]. We can merge the five-parameter estimation of laser phase noise and Jones matrix in Eq. (2) into four-parameter estimation by multiplying the laser phase noise into the matrix.

The model of CKF and MKF is defined as follows. During the iteration process of KF, the convergence can be achieved by minimizing the difference between the predicted signal and the observed signal. It is assumed that ${\widehat{\mathit{z}}}_{\mathrm{CKF}}$ and ${\widehat{\mathit{z}}}_{\mathrm{MKF}}$ are signals predicted by CKF and MKF. In CKF, ${\widehat{\mathit{z}}}_{\mathrm{CKF}}$ can be derived by $${\widehat{\mathit{z}}}_{\mathrm{CKF}}=\left[\begin{array}{c}{\widehat{x}}_x(n)\\ {\widehat{x}}_y(n)\end{array}\right]=\widehat{\mathit{G}}\left[\begin{array}{c}{y}_x(n)\\ y_y(n)\end{array}\right]=\widehat{\mathit{G}}\mathit{y},$$(3)whereas in MKF, ${\widehat{\mathit{z}}}_{\mathrm{MKF}}$ can be derived by $${\widehat{\mathit{z}}}_{\mathrm{MKF}}=\left[\begin{array}{c}{\widehat{y}}_x(n)\\ {\widehat{y}}_y(n)\end{array}\right]=\widehat{\mathit{H}}\left[\begin{array}{c}{x}_x(n)\\ x_y(n)\end{array}\right]=\widehat{\mathit{H}}\mathit{x},$$(4)where the subscripts $x$ and $y$ in Eqs. (3) and (4) stand for the signal on two orthogonal polarization states, i.e., $x$-Pol. and $y$-Pol, and $\widehat{\mathit{H}}$ and $\widehat{\mathit{G}}$ are the estimated channel response matrix and the estimated inverse channel response matrix, respectively. Then, the measurement residual ${\mathit{\delta }}_{\mathrm{CKF}}$ and ${\mathit{\delta }}_{\mathrm{MKF}}$ for CKF and MKF can be expressed as $${\mathit{\delta }}_{\mathrm{CKF}}={\mathit{z}}_{\mathrm{CKF}}-{\widehat{\mathit{z}}}_{\mathrm{CKF}}=\left[\begin{array}{c}{x}_x(n)\\ x_y(n)\end{array}\right]-\widehat{\mathit{G}}\left[\begin{array}{c}{y}_x(n)\\ y_y(n)\end{array}\right],$$(5)$${\mathit{\delta }}_{\mathrm{MKF}}={\mathit{z}}_{\mathrm{MKF}}-{\widehat{\mathit{z}}}_{\mathrm{MKF}}=\left[\begin{array}{c}{y}_x(n)\\ y_y(n)\end{array}\right]-\widehat{\mathit{H}}\left[\begin{array}{c}{x}_x(n)\\ x_y(n)\end{array}\right].$$(6)

The difference in measurement residual between CKF and MKF is illustrated in Fig. 1. For CKF, the noisy terms $\widehat{\mathit{G}}$ and the predicted measurement $y(n)$ are multiplied together to influence the measurement residual. While MKF differs from CKF, only one noise-related factor contributes to the prediction ${\widehat{\mathit{z}}}_{\mathrm{MKF}}$, and the two noisy terms $y(n)$ and $\widehat{\mathit{H}}$ additively act on the measurement residual. We show the equalization model and state estimation process in Fig. 2.

Apart from the measurement residual, the KF algorithm also minimizes the process noise. $\mathit{G}$ and $\mathit{H}$ are defined as the state matrix of CKF and MKF, respectively. The process noises ${\mathrm{\Delta }}_G$ and ${\mathrm{\Delta }}_H$ in each iteration for CKF and MKF can be expressed in Eqs. (7) and (8): $$\widehat{\mathit{G}}=\mathit{G}+{\mathrm{\Delta }}_G={\mathit{H}}^{-1}+{\mathrm{\Delta }}_G,$$(7)$$\widehat{\mathit{H}}=\mathit{H}+{\mathrm{\Delta }}_H.$$(8)

To concentrate on the noise tolerance analysis of the two schemes, we degenerate the theoretical analysis into single polarization. Substituting $\widehat{\mathit{G}}$ in Eq. (5) with Eq. (7), we derive the measurement residual of CKF as $$\begin{gathered} {\delta }_{\mathrm{CKF}}=x-\widehat{x}=x-\widehat{G}(H·x+N) \\ =x-(H^{-1}+{\mathrm{\Delta }}_G)(H·x+N) \\ =-(G+{\mathrm{\Delta }}_G)N-{\mathrm{\Delta }}_{G}Hx=-\widehat{G}N-{\mathrm{\Delta }}_{G}Hx. \end{gathered}$$(9)

Similarly, we have the measurement residual ${\delta }_{\mathrm{MKF}}$, $$\begin{gathered} {\delta }_{\mathrm{MKF}}=y-\widehat{y}=y-\widehat{H}x=(Hx+N)-\widehat{H}x \\ =(Hx+N)-(H+{\mathrm{\Delta }}_H)x=N-{\mathrm{\Delta }}_{H}x. \end{gathered}$$(10)

According to Eqs. (9) and (10), the process noise for CKF and MKF versus the measurement residual can be expressed as $$\begin{gathered} {\mathrm{\Delta }}_G=-({\delta }_{\mathrm{CKF}}+H^{-1}N)/y \\ =-({\delta }_{\mathrm{CKF}}+H^{-1}N)/(Hx+N), \end{gathered}$$(11)$${\mathrm{\Delta }}_H=(N-{\delta }_{\mathrm{MKF}})/x.$$(12)

We introduce the conditional variance of measurement residuals and process noises for CKF and MKF to evaluate the noise tolerance and to explore their interaction during each prediction and innovation iteration. We define the conditional variance of ${\mathit{\delta }}_{\mathrm{CKF}}$ when sequence $x$ is transmitted as $$D({\delta }_{\mathrm{CKF}}|x)=E_{{\mathrm{\Delta }}_G}[D(\widehat{G}N+{\mathrm{\Delta }}_{G}Hx)].$$(13)

Similarly, we have $$D({\delta }_{\mathrm{MKF}}|x)=E_{{\mathrm{\Delta }}_H}[D(N-{\mathrm{\Delta }}_{H}x)],$$(14)$$D({\mathrm{\Delta }}_G|x)=E_{{\delta }_{\mathrm{CKF}}}\{D[({\delta }_{\mathrm{CKF}}+H^{-1}N)/(Hx+N)]\},$$(15)$$D({\mathrm{\Delta }}_H|x)=E_{{\delta }_{\mathrm{MKF}}}\{D[(N-{\delta }_{\mathrm{MKF}})/x]\},$$(16)where $E(·)$ and $D(·)$ are the expectation and the second central moment. The conditional variances are derived from the property of $$D(A|C)=E_B[D(A|B,C)]+D_B[E(A|B,C)],$$where the second term on the right hand is small enough and can be assumed to be zero since the process noise and the measurement residual of a converged equalizer in a certain iteration can be regarded as a zero-mean random variable. Larger conditional variance leads to worse noise tolerance particularly at the hard decision stage, and hence the filter has higher probability suffering from the noise-dominated divergence.

To avoid the complicated analytical derivation in variance calculation, numerical simulations are conducted to compare the noise tolerance between CKF and MKF based on Eqs. (13)–(16), where a 16-QAM sequence with $2^{16}$ complex symbols is transmitted through the AWGN channel. For simplicity, we compare the process noises by setting the channel-scaling factor, $H=1$, which leads to $G=H^{-1}=1$. Figures 3(a) and 3(b) show the conditional variance curves of measurement error and process noise varying with process noise and measurement error at different SNRs. To fully compare the conditional variance, we set the $x$ axis in logarithmic scale, which covers the cases with both slighter and stronger noise. We take specific values of two variables to illustrate the rationality of the abscissa of Figs. 3(a) and 3(b). Since the I- or Q-component of the transmitted QAM signal ranges from $-3$ to 3, ${\delta }_{\mathrm{KF}}=6$ in Fig. 3(a) indicates that the measurement residual is on the same order of magnitude as the transmitted signal, which is large enough for a convergent Kalman filter. In Fig. 3(b), ${\mathrm{\Delta }}_{\mathrm{KF}}={10}^{[10{\log }_{10}({\mathrm{\Delta }}_{\mathrm{KF}})]/10}={10}^{3/10}=2$ also provides the conditional variance under the situation with larger process noise. As shown in Figs. 3(a) and 3(b), both the conditional variances of process noise and measurement error for MKF are always smaller than those of CKF. The gaps between MKFs and CKFs occur in practical SNRs and tend to be more obvious for lower SNR cases.

To compare the SNR robustness between MKF and CKF, we further depict the ratio of $D({\delta }_{\mathrm{CKF}}|{\mathrm{\Delta }}_G,x)$ and $D({\delta }_{\mathrm{MKF}}|{\mathrm{\Delta }}_H,x)$ versus SNRs at different process noise levels, as shown in Fig. 4. All of the conditional variance ratios are larger than one, which indicates that MKF always introduces a smaller measurement residual, especially for channels with higher noise power.

Therefore, for the general SNR region, we always have $$D({\delta }_{\mathrm{MKF}}|{\mathrm{\Delta }}_H,x)\le D({\delta }_{\mathrm{CKF}}|{\mathrm{\Delta }}_G,x),$$(17)$$D({\mathrm{\Delta }}_H|{\delta }_{\mathrm{MKF}},x)\le D({\mathrm{\Delta }}_G|{\delta }_{\mathrm{CKF}},x),$$(18)which means that MKF has less equalization enhanced instability than CKF. By averaging each case over ${\mathrm{\Delta }}_{H(G)}$ and ${\delta }_{(\mathrm{MKF})\mathrm{CKF}}$ in Fig. 2, for the given SNRs, we have $$D({\mathrm{\Delta }}_H|x)<D({\mathrm{\Delta }}_G|x),$$(19)$$D({\delta }_{\mathrm{MKF}}|x)<D({\delta }_{\mathrm{CKF}}|x).$$(20)

Inequalities (19) and (20) also hold in all practical channel noise powers, as shown in Fig. 3, which indicates MKF outperforms CKF in noise tolerance, especially for large noise. This also explains the better noise tolerance for MKF compared with CKF in our previous experiment^[19].

For stability and real implementation, we introduce the noise covariance matching method into MKF, since it has lower complexity and is more practical to implement than other adaptive algorithms^[20,21]. The adaptive estimations of covariance matrices of process and measurement noise ${\mathit{Q}}_k$ and ${\mathit{R}}_k$ are expressed in a compact form as Eqs. (21) and (22): $${\mathit{R}}_k={\alpha }_R{\mathit{R}}_{k-1}+(1-{\alpha }_R)\frac{\mathrm{tr}({\mathit{\delta ̆}}_{\mathrm{MKF},k}{\mathit{\delta ̆}}_{\mathrm{MKF},k}^H+{\mathit{M}}_n{\mathit{P}}_k{\mathit{M}}_n^H)}{\mathrm{tr}({\mathit{R}}_{k-1})}{\mathit{R}}_{k-1},$$(21)$${\mathit{Q}}_k={\alpha }_Q{\mathit{Q}}_{k-1}+(1-{\alpha }_Q)\frac{\mathrm{tr}({\mathit{K}}_k{\mathit{\delta ̆}}_{\mathrm{MKF},k}{\mathit{\delta ̆}}_{\mathrm{MKF},k}^H{\mathit{K}}_k^H)}{\mathrm{tr}({\mathit{Q}}_{k-1})}{\mathit{Q}}_{k-1},$$(22)where $${\mathit{M}}_n=\left[\begin{array}{cccc}{x}_x(n)& x_y(n)& 0& 0\\ 0& 0& x_x(n)& x_y(n)\end{array}\right],$$(23)$${\mathit{s}}_k=[{\widehat{H}}_{xx}(k),{\widehat{H}}_{xy}(k),{\widehat{H}}_{yx}(k),{\widehat{H}}_{yy}(k)],$$(24)and ${\mathit{\delta ̆}}_{\mathrm{MKF},k}={[y_x,y_y]}^T-{\mathit{M}}_n{\mathit{s}}_k^T$ is the measurement residual after the update of ${\mathit{s}}_k$. “tr” represents the trace of the matrix. $\mathit{K}$ and $\mathit{P}$ are the Kalman gain matrix and the error autocorrelation matrix. ${\alpha }_R$ and ${\alpha }_Q$ are forgetting factors that are helpful to balance the tracking speed and estimation accuracy.

When the channel is memorial, the state vector can be extended to multi-tap. When obtaining the multi-tap state vector $\mathit{s}$, we can reshape the state vector to the channel response matrix $\mathit{A}$ and determine its pseudo-inverse matrix as $\mathit{G}={\mathit{A}}^{†}={({\mathit{A}}^H\mathit{A})}^{-1}{\mathit{A}}^H$ if the columns of $\mathit{A}$ are linearly dependent. Besides, the inter-symbol interference can also be mitigated by conventional adaptive equalizers with the least mean squares (LMS) or recursive least squares (RLS) algorithms.

In this section, we conduct numerical simulations to testify the proposed modified adaptive Kalman filter for joint polarization and phase noise tracking. The simulation system is a conventional polarization-multiplexed coherent 16-QAM optical transmission system established in VPI 9.1. System parameters are shown in Table 1. The block of RSOP is set based on Eq. (2), where the angle $\kappa =2\pi ft$ varies in a sinusoidal form with the scrambling frequency $f$. At the receiver, frequency-domain group velocity dispersion compensation, fourth power FFT-based frequency offset compensation, and match filter are applied before the Kalman filter. The adaptive KF is adopted for polarization de-multiplexing and phase recovery. Both CKF and MKF adopt the covariance matching adaptive algorithm with the initial covariance of 0.5.

We first compare the tracking performance between adaptive CKF and adaptive MKF at different OSNRs, as shown in Fig. 5, where we also compare the performance of the extended CKF (ECKF) and pilot-aided averaging window method (AWM) as a reference, where the ECKF still regards the measurement variable as the transmitted symbols but estimates the Jones matrix and laser phase noise $\varphi (n)$ separately. The AWM inserts three pilot symbols between every 17 symbols and smooths the estimation noise by time averaging. The pilot overhead of AWM is 15%, while for KF the number of training symbols is 100. The overhead of KF is much less than AWM. The scrambling rate of RSOP is 10 MHz and 20 MHz, corresponding to Figs. 5(a) and 5(b), both of which have the linewidth of 100 kHz. The performances are evaluated at the 20% forward error correction (FEC) limit, i.e., the bit error rate (BER) of $2.5\times {10}^{-2}$.

When the OSNR is below 18 dB, the divergence of KF mainly results from the noise sensitivity of the second stage of Kalman filter. The BER performance of MKF and CKF mainly differs from each other within 18 to 24 dB since the noise tolerance of KF differs in this range. We find MKF outperforms CKF and ECKF for both cases and has $>3\mathrm{dB}$ OSNR improvement over CKF at the 20% FEC limit. When the scrambling rate increases to 20 MHz, 3.5 dB OSNR gain is observed compared with CKF, while 1.87 dB OSNR gain is observed compared with ECKF. AWM works better than KF at extremely low OSNRs, which benefits from its very large overhead. Once KF converges, the BERs rapidly drop below those of AWM.

We then compare the tracking ability of the adaptive CKF and MKF with lower and larger noise powers. Figures 6(a) and 6(b) show the BER performances for CKF and MKF versus scrambling rate at the OSNRs of 24 dB and 20 dB. The trends can be divided into three segments, i.e., the optimum stage (the scrambling rate is lower than 10 MHz), the transition stage (the scrambling rate from 20 MHz to 35 MHz at 24 dB OSNR, from 10 MHz to 20 MHz at 20 dB OSNR), and the divergence stage (the higher scrambling rate).

Focusing on the transition stage, i.e., the extreme tracking performance for CKF and MKF, we find that CKF and ECKF diverge sharply at the OSNR of 24 dB, whereas the performance of MKF deteriorates gradually. From Fig. 6(a), we find that MKF can converge under the scrambling rate up to 35 MHz, around 73.5% enhancement over CKF, i.e., 14.83 MHz. When the OSNR is 20 dB, CKF diverges, but MKF still has $>15\mathrm{MHz}$ scrambling rate tolerance, as shown in Fig. 6(b). This is because the enhanced channel noise has higher probability to induce cycle-slip or algorithm divergence.

At the optimum stage for the OSNR of 24 dB, the CKF and MKF have similar performances. For the OSNR of 20 dB, MKF can tolerate higher scrambling rate than ECKF at 20% FEC limit, while CKF fails to track the fast RSOP in the megahertz (MHz) level. MKF and ECKF outperform AWM in both cases, even though MKF is a kind of linear KF. The results are consistent with Fig. 5, which shows that MKF can work stably in relatively lower OSNR.

Finally, we investigate the parameter initialization tolerance with and without the Q/R adaptive algorithm. We show the BERs versus the initial Q and R with hot maps in Fig. 7, where the OSNR is 30 dB. In Fig. 7, the required R is proportional to but larger than Q when the Q/R adaptive algorithm is not enabled. Comparing the BER performance of MKF with CKF, we find MKF outperforms CKF, as shown in the blue regions of the hot maps in Figs. 7(a) and 7(c), which indicates a higher flexibility of MKF in Q/R initialization that is mainly brought from the improved noise tolerance of MKF.

When the adaptive algorithm is applied to the Kalman filters, the convergence regions are only dependent on the initial R. We find that the Q/R adaptive algorithm can also effectively improve the flexibility of MKF, since both the convergence regions of CKF and MKF are expanded significantly. This indicates that the adaptive algorithm can improve the robustness of MKF at higher OSNRs. While for lower OSNRs, adaptive MKF succeeds in tracking the fast RSOP but adaptive CKF fails, as discussed in Section 3.2.

The simulation results in Section 3 are concluded in Table 2 for a clear comparison. Based on the discussion above, our findings are as follows. (1) MKF has higher noise tolerance than CKF especially with lower OSNRs. (2) The modification improves the tracking speed and initialization flexibility, which benefits from the improved noise tolerance. (3) The noise covariance matching approach is effective for improving the initialization flexibility of MKF.

Both theoretical analysis and simulation results have shown that the MKF has higher robustness to noise-induced decision error and covariance matrices initialization that are related to the tracking ability in the presence of fast RSOP. Higher tracking speed (up to 50% improvement from CKF), better OSNR tolerance ($>3\mathrm{dB}$ improvement from CKF), and superior initialization flexibility are observed from MKF. The MKF combined with the covariance matching method outperforms the adaptive CKF, especially in the case of lower OSNR.