With the advent of emerging technologies such as metaverse, 6G, virtual reality, and augmented reality, enhancing the network capacity of short-range fiber optic communication systems has become increasingly pressing. Compared to coherent optical communications, intensity modulation/direct detection (IM/DD) systems offer the advantages like low cost and power consumption, making them widely deployed in short-haul communications^[1,2]. Additionally, carrierless amplitude and phase modulation (CAP) stands out for its high spectral efficiency, low error rate, robust resistance to nonlinear effects, low power consumption, and high system integration. Therefore, CAP is widely adopted in IM/DD systems^[3,4].

To mitigate the impact of signal noise, geometric shaping (GS) has garnered research interest. By increasing the minimum Euclidean distance (MED) while maintaining constant power, the resilience of the signal to noise can be further enhanced^[5,6]. However, the proposed GS methods may struggle to fully adapt to fluctuating real-world channel conditions. In recent years, a learning model known as the autoencoder (AE) has seen extensive applications in communication systems^[7]. By minimizing reconstruction error, the AE can jointly train the encoder and decoder, thereby improving the performance of two-dimensional constellations in IM/DD systems^[8]. However, the exploration of the AE’s performance in 3D-GS remains insufficient. Three-dimensional (3D) constellations require consideration of the spatial distribution of constellation points, significantly increasing complexity. Therefore, it is necessary to conduct in-depth research on AE’s optimization capabilities in 3D-GS to understand its potential in optimizing constellation point distribution and increasing channel capacity in high-dimensional space, thereby providing new technical support for future high-performance high-reliability communication systems. The issue of communication security has garnered significant research attention. Chaotic systems, known for their ergodicity, pseudo-randomness, and sensitivity to initial conditions, have become a popular choice for communication encryption. However, existing digital chaotic encryption techniques have inherent limitations^[9]. Achieving signal encryption without compromising transmission performance remains a major challenge for optical fiber communication systems.

In this paper, a constellation optimization based on the AE is proposed, encompassing a four-level masking scheme for high-security 3D-CAP modulation. This scheme aims to optimize the geometrical distribution of the constellation points end-to-end based on the embedded channel model of the AE, thereby enhancing the transmission performance of the system. Subsequently, a chaotic sequence is generated by a 3D chaotic oscillator model^[10]. Before transmission, chaotic selection mapping rules are constructed based on the chaotic sequence. The subcarriers are masked using the masking factors derived from the chaotic sequences, and the constellation maps are encrypted in conjunction with class-k order Fibonacci polynomials^[11]. The four-stage masking scheme driven by class-k Fibonacci polynomials boasts a large key space of ${10}^{103}$ and is experimentally verified on a 2 km seven-core fiber transmission system, achieving a receiver sensitivity gain of 1.0 dB compared to the original face-centered cubic constellation, thereby confirming the feasibility of the proposed scheme.

Figure 1 depicts the principle of high-security 3D-CAP employing four-level masking, where the encoder emulates the transmitter, the decoder emulates the receiver, and an additive white Gaussian noise (AWGN) fiber channel model is embedded between them. Through end-to-end optimization, a geometrically shaped constellation is obtained. Subsequently, chaotic selective mapping and k-order Fibonacci polynomial masking^[11] of the constellation data are realized by leveraging chaotic selective mapping factors generated from a 3D chaotic oscillator model^[10] and Fibonacci polynomial masking factors. Furthermore, constellation rotation and subcarrier masking are identified based on the rotation and subcarrier masking factors. Prior to transmission, the data-masked 3D-CAP signal undergoes oversampling, shaping, filtering, and summation. Conversely, the receiver performs the reverse process, demodulating and decrypting the received signal.

Now we examine the principle of end-to-end 3D constellation geometry shaping, $Y=G\{C[F(X)]\}$, where $F(·)$ denotes the encoder function and $G(·)$ denotes the decoder function. Since the AWGN channel model exhibits a channel response akin to the seven-core channel, we opt to utilize the AWGN channel as the function $C(·)$. The mathematical model of the AWGN channel can be expressed as $C(x)=x+n$, where $C(x)$ represents the signal received at the receiver end, $x$ is the signal transmitted by the sender, and $n$ denotes the Gaussian white noise, comprising independent and identically distributed Gaussian random variables with a mean of 0 and a variance of $N_0/2$. $N_0$ is the noise power spectral density.

The whole model aims to reproduce the input $X$ at the output $Y$ through the latent variable $F(X)$. $L(·)$ is the loss function as shown in Eq. (3), and the parameter vector $\theta =(\theta ,{\theta }^{\prime })$ contains all the trainable variables. The AE aims to minimize the cross-entropy loss function $L(·)$ by iteratively updating the parameter $\theta$ through stochastic gradient descent (SGD), thereby enhancing the learning process on the geometric structure of the 3D constellations. This iterative process continues until the training parameter $\theta$ converges to a value that brings the variables $X_i$ and $Y_i$ as close as possible, $$L(\theta )=-\frac{1}{j}\sum _j\sum _k{X}_{jk}\log Y_{jk},$$(1)where $j$ is the size of the training batch. We calculate the average value of the loss function for all samples and update the parameter $\theta$ to minimize the loss during fine-tuned training using SGD. This enhances the learning process of the autoencoder features, and the $\theta$ is updated as follows: $${\theta }^{(\text{step}+1)}={\theta }^{(\text{step})}-\eta {\nabla }_{\theta }{L}^{\prime }[{\theta }^{(\text{step})}].$$(2)

Since the constellation points must recur frequently within the training batches, specifying the training batch size as a multiple of M is more judicious. In this paper, M represents the modulation order, which is set to 16. Figure 2 illustrates the degree of loss convergence across different training batch sizes. Larger training batches result in slower initial convergence but yield better final outcomes, while smaller batches converge faster initially but lead to poorer final results. To strike an optimal balance among convergence speed, computational cost, and effectiveness, a smaller training batch can be employed initially, gradually increasing after the initial convergence. As the lines in Fig. 2 indicate, as the training batch size grows, the parameter estimates of the model statistics become more precise. By setting the training batch across 6 ranges (8M, 16M, 32M, 64M, 128M, and 256M), it can be observed that the reconstruction error loss (REL) is lowest when the batch size is 128M. However, when the batch size increases to 256M, the REL begins to rise again. This indicates that the model’s convergence performance is optimal at a batch size of 128M. The output of the constellation point coordinates and the constellation diagram are shown in Figs. 3(a) and 3(b).

The chaotic sequences, generated by the 3D chaotic oscillator model, are represented by $$\{\begin{array}{l}{x}^{\prime }=z\\ y^{\prime }=z-y\\ z^{\prime }=\eta y+z+0.3x^2-1.5xy+0.6xz\end{array}.$$(3)

Due to the system’s extreme sensitivity to initial conditions, when the parameters of the 3D chaotic oscillator model are set to $\eta =-1$ and $(x,y,z)=[-40,-2.85,-3.22]$, it generates significantly different complex chaotic trajectories. In the event of illegal reception, if the correct key is not available, the message cannot be decrypted correctly.

The above 3D chaotic oscillator can generate three independent chaotic sequences $(x,y,z)$. Taking this chaotic sequence $x$, the 3D-CAP constellation chaotic selection mapping sequence $B_1$ is set as $B_1=\mathrm{mod}(x\times {10}^{20},5)$ so that $B_1[i]$ denotes the element in $B_1$ and different $B_1[i]$ values correspond to distinct mapping rules.

After chaotic selection mapping, Fibonacci masking of the three messages in Message1 (M1) uses Fibonacci-like polynomials. First, the polynomial $f_n(x)$ investigated by the Belgian mathematician Catalan is defined by the following recursive relation^[11]: $$f_n(x)=x{f}_{n-1}(x)+f_{n-2}(x).$$(4)

For $n\ge 3$ and $x=1$, the resulting Fibonacci polynomial is $f_0(x)=0$, $f_1(x)=1$, and $f_2(x)=x$. Under the condition $n\ge 3$, we transform the aforementioned recursive relation and Fibonacci polynomial, resulting in Eq. (5): $$\{\begin{array}{l}{f}_n(x_n)=x_{n-1}{f}_{n-1}(x_{n-1})+x_n{f}_{n-2}(x_{n-2})\\ f_0(x_0)=x_0^2+1\\ f_1(x_1)=x_1^3+2x_0\\ f_2(x_2)=x_1^4+x_0(2x_1+x_2)+x_2\end{array}.$$(5)

As depicted in Fig. 4, in step 1, the chaotic sequence matrix has a dimension of $3\times N$. $N$ represents the number of columns in the chaotic sequence matrix. Initially, we use the column selection function from Eq. (7) to obtain a new $3\times i$ matrix (where $i<N$). In step 2, taking Column $i$ as an example, the three initial chaotic factors a, b, and c within Column $i$ are calculated by Mod(, 1000) to obtain $x_0$, $x_1$, and $x_2$, respectively, after which $f_0$, $f_1$, and $f_2$ are derived from Eq. (6). In step 3, $f_0$, $f_1$, and $f_2$ are vertically arranged to form a new F column i. These new columns, collectively denoted as F, constitute the matrix F. The matrix M1 shares the same $3\times i$ dimensional structure as matrix F. The principle of reverse cross multiplication is akin to step 4. We define the sum of the column index values in matrix F and matrix M1 as $i+1$ and then perform operations on the corresponding elements within these two columns. Taking F Column $i$ and M1 Column 1 as examples, $f_0$ multiplies $m_0$ to obtain $n_0$, $f_1$ multiplies $m_1$ to obtain $n_1$, and $f_2$ operates with $m_2$ to obtain $n_2$, collectively forming a new column, designated as matrix Message2 (M2) Column i. Using the same method, we calculate the remaining columns, then combine all the computed columns to obtain the encrypted matrix M2, thus achieving a constellation masking resembling Fibonacci sequences. The computation method for $i$ is depicted in Eq. (6): $$i=\text{randi}(\text{Num}),\text{Num}\in (1,N).$$(6)

The variable Num represents the number of terms in a chaotic sequence, with its maximum value reaching $N$. Constellation rotation and subcarrier masking are then performed on the constellation-transformed message M2. The rules for generating the carrier masking factor $A$ and constellation rotation factor $B$ are as Eq. (7): $$\{\begin{array}{l}A={\{\text{Tra}\{\mathrm{mod}(y,1)\times [1/\text{sort}(y)]\}\}}^T\\ \text{Rotation}\_X=\mathrm{mod}(x,1)\times 360\\ \text{Rotation}\_Y=\mathrm{mod}(y,1)\times 360\\ \text{Rotation}\_Z=\mathrm{mod}(z,1)\times 360\\ B=\text{Vertact}(\text{Rotation}\_X,\text{Rotation}\_Y,\text{Rotation}\_Z)\end{array}.$$(7)

$\mathrm{Tra}(.)$ is the transpose function in a matrix, mod is the remainder operation function, sort is the ascending sorting function, and $T$ is the matrix transform function. The function $T$ is to reassign the target matrix to 0 or 1 by discriminating the difference between the elements of the target matrix and 1, and finally transform it to form a primitive matrix A (each row has only one 1, and the rest of the elements are 0). A is then used to displace the subcarrier perturbations with M2. Using three independent chaotic sequences ($x,y,z$), each sequence element is processed according to the above equation to generate the constellation rotation factors $\mathrm{Rotation}\_X$, $\mathrm{Rotation}\_Y$, and $\mathrm{Rotation}\_Z$, which are spliced to form a constellation rotation matrix.

This scheme employs a chaotic model with control parameters ($x$, $y$, $z$, $\eta$) and initial values of ($-40$, $-2.85$, $-3.22$, $-1$). As shown in Fig. 5, when the variation in parameter $x$ exceeds 10⁻¹⁶, the bit error rate (BER) significantly increases to 0.3095, making it impossible to correctly obtain the constellation diagram. Consequently, the illegal receiver (ONU) cannot correctly intercept the data. The conservative key space size is ${({10}^{15})}^5\times {({10}^{14})}^2={10}^{103}$, which is sufficiently large to withstand data interception attempts by illegal receivers.

We compared a variational autoencoder (VAE) with 2144 parameters to an AE with 1299 parameters. The AE is more memory-efficient and faster, while the VAE increases memory usage and latency. We also evaluated a 3D chaotic oscillator, Chen’s attractor, and the logistic map. The 3D chaotic oscillator offers better security with low complexity and high efficiency.

The IM/DD system depicted in Fig. 6 is used for validation purposes. At the transmitter, after data processing via digital signal processing (DSP), the digital signal is converted into an analog radio frequency (RF) signal by an arbitrary waveform generator (AWG, TekAWG70002A) at a sampling rate of 10 GSa/s. The RF signal is then amplified by an electrical amplifier (EA) and modulated by a Mach–Zehnder modulator (MZM) using a laser source with a linewidth of less than 100 kHz to generate a 1550 nm optical carrier. The modulated optical signal is subsequently amplified by an erbium-doped fiber amplifier (EDFA). At the receiver, a variable optical attenuator (VOA) is used to adjust the received optical power. The adjusted optical signal is detected and converted into an electrical signal by a photodetector (PD) and then sampled by a mixed-signal oscilloscope (MSO) at a rate of 50 GSa/s. After undergoing DSP, the bit sequence is retrieved at the receiver. By comparing this sequence with the original sequence, the BER performance of the constellation after GS can be deduced.

At the forward error correction (FEC) threshold of $3.8\times {10}^{-3}$, the received optical power of core 7 and BTB is about $-16.8\mathrm{dBm}$, as can be seen in Fig. 7(a). The gain of the best fiber core compared to the worst fiber core is less than 0.6 dB, indicating that the seven-core optical fiber transmission system used in the experiment has a good uniformity within the range of 2 km. In addition, when the received optical power is more significant than $-16.5\mathrm{dBm}$, the BER performance of the seven cores is within the FEC threshold, the BER of the seven cores gradually decreases with the increase of the optical power value, and the proposed chaotic selection 3D-CAP transmission scheme has good transmission performance. It can be seen that the BER performance of core 7 is the closest to that of BTB, so core 7 is chosen as the experimental channel.

The encrypted, unencrypted, and illegal GS constellations are denoted as Enc-GS-16 CAP, W/O-Enc-GS-16 CAP, and illegal-GS-16 CAP, respectively. The original constellation is denoted as 16 CAP. In order to validate the effect of the proposed encryption on the BER performance of the transmission system, Enc-GS-16 CAP, W/O-Enc-GS-16 CAP, and illegal-GS-16 CAP constellation diagrams after transmission over core 7, the performance curves of each BER are shown in Fig. 7(b). As illustrated in Fig. 7, if the encryption algorithm is intercepted illegally, the received BER is nearly 0.5. The obtained constellation diagram is entirely unresolved, and it is almost impossible for the eavesdropper to crack the message and recover the correct constellation diagram without the right key. In legitimate receiving ONUs, the performance of Enc-GS-16 CAP and W/O-Enc-GS-16 CAP is closely matched. This confirms that the proposed encryption scheme does not impact the performance of the system.

In addition, for CAP transmitted over core 7, we compared the BER performance of traditional 16 CAP with that of the proposed geometric shaped 16 CAP, disregarding the effect of the encryption method. Figure 8 presents the BER performance comparison between traditional 16 CAP and geometric shaped 16 CAP transmitted over core 7. It can be seen that, at a BER of $3.8\times {10}^{-3}$, the received optical power for traditional 16 CAP is $-15.95\mathrm{dBm}$, whereas for the proposed 16 CAP, it is $-16.97\mathrm{dBm}$. Therefore, the geometrically shaped 16 CAP demonstrates an improvement of approximately 1 dB compared to the traditional 16 CAP.

In this paper, a high-security 3D constellation GS scheme utilizing AE is introduced. By optimizing the geometric shape of high-dimensional constellations, AE enhances the transmission performance of the 3D GS communication system. The approach integrates chaotic oscillators with $k$-order Fibonacci polynomials to apply chaotic selective mapping and multi-level masking to the 3D constellation data. This method features a key space size of ${10}^{103}$, which significantly boosts the communication system’s security. Experiments conducted over a 2 km seven-core fiber and a BTB system demonstrate that the receiver sensitivity of the geometrically shaped constellation signal can be improved by up to 1 dB compared to the original constellation. Compared to a VAE with a similar structure, AE offers reduced complexity, with a 40% decrease in parameters. Additionally, the chaotic system used has lower computational complexity. The results confirm that this scheme effectively enhances both security and BER performance. However, to further advance the system, future work should focus on exploring dynamic masking parameters and pruning techniques to improve dynamic adaptability and further reduce AE complexity.