As applications such as 5G, cloud computing, cloud storage, the Internet of Things (IoT), and smart driving continue to emerge, the data capacity required for communication is surging exponentially, and the growth of end users is increasing the demand for bandwidth. Advanced coding and modulation techniques are beginning to attract widespread attention. Carrier-less amplitude and phase (CAP) modulation is a multidimensional and multi-order modulation technique^[1,2], which was first proposed by Bell Labs in the 1970s. CAP modulation has no multiplying part with the carrier but directly uses mutually orthogonal shaping filters to form the bandpass pulse signal, so this technique is called “carrier-less”^[3,4]. Compared to orthogonal frequency division multiplexing (OFDM), CAP modulation no longer requires the use of discrete Fourier transforms, which greatly reduces the computational complexity and the structure of the system^[5,6].

Meanwhile, passive optical networks (PONs) have proven to be a future-proof network architecture due to their ability to provide high bandwidth, low cost, and high data transfer rate in long-haul networks. However, since PON systems use broadcast technology for the downlink data stream, an illegal optical network unit (ONU) can easily disguise itself to hijack the downlink data signal sent by the optical line terminal (OLT). Therefore, it is extremely important to encrypt the information at the physical layer before downlink transmission in PON systems^[7,8]. Traditional upper-layer encryption has limitations such as the ease with which data headers can be exposed during data transmission, which poses significant security threats. Physical layer encryption plays a crucial role in ensuring transmission security by encrypting data signals in a high-speed and low-delay manner while minimizing the impact on the network system. As a supplement and improvement of the upper-layer encryption mechanism, the physical layer security encryption technology has broad research prospects and application potential. Among various current physical layer encryption schemes, chaos-based communication encryption technology has a broad prospect in the field of optical communication system security due to its high initial sensitivity, noise-like, and large key space^[9]. Therefore, the application of chaotic systems to physical layer communication encryption is a highly secure and promising encryption method. In addition, probabilistic shaping (PS) is a novel digital signal processing technique^[10,11] that reduces the nonlinear effects on the signal by decreasing the probability of the occurrence of signals at constellation points with high amplitude values and increasing the probability of the occurrence of constellation points with low amplitude values^[12,13]. By shaping the constellation map with a non-uniform distribution, the transmit power of the whole system can be reduced, which achieves the expansion of the transmission capacity of the channel and effectively reduces the system bit error ratio (BER)^[14,15]. Therefore, probabilistic shaping can achieve an effective increase in signal-to-noise power, which, combined with high-dimensional constellation CAP modulation, makes the location of constellation points more flexible and further improves the transmission performance of the system.

With the rapid development of optical communication technology, the communication capacity of conventional single-mode fibers is approaching the nonlinear Shannon capacity limit, in which space division multiplex (SDM) technology has become our focus of attention^[16]. Multi-core fiber (MCF) is a promising SDM fiber, which can increase the transmission capacity of fiber by increasing the number of fiber cores^[17,18]. Reference [19] conducted experimental verification of 25.45 Gb/s three-dimensional (3D) CAP signal transmission on a 7-core fiber optic communication system. Similarly, in Ref. [20], the authors used 37 km of 7-core fiber to transmit 12 channels of 40 Gb/s polarization division multiplexing quadrature phase-shift keying (PDM-QPSK) signals, using ultra-dense wavelength division multiplexing passive optical networks (WDM-PONs). In Ref. [21], 189.6 Gb/s carrier-less amplitude/phase modulation with probabilistic shaping based on the Lorenz model and superposition method (LS-PS-CAP) data signal transmission over a 2.5 km MCF was demonstrated in the experiments.

In this research, we propose a carrier-less amplitude phase modulation passive optical network (CAP-PON) scheme for optical access networks that combines dynamic probability shaping with Rubik’s cube encryption. Using a five-dimensional entangled chaos system, we enhance the efficiency and anti-noise performance of constellation points. Rubik’s cube encryption ensures high physical layer security. Experiments show that this dynamic probability shaping 3D carrier-free amplitude and phase-modulated passive optical network (DPS-3D-CAP-PON) system, transmitted over a 2 km 7-core optical fiber, significantly improves both transmission and security performance.

The system algorithm block diagram of the whole scheme is shown in Fig. 1. At the transmitter side, the raw data are converted into three signals by serial-parallel transformation (S/P). These three signals are mapped to the 3D constellation map and then subjected to a 3D dynamic probabilistic shaping (DPS) process. Then the data enter the magic cube encryption module to realize the physical layer data encryption. Finally, these three signals are processed by CAP modulation, and they are combined into one symbol for transmission. On the receiving end, the received data are demodulated by CAP modulation and then decrypted with the same key as well as the 3D DPS de-mapping process. Finally, the original data are obtained by parallel-serial transformation (P/S).

Among them, for the DPS mapping module and the Rubik’s cube encryption module, we use a new five-dimensional entangled chaotic system to generate the mask vector and the encryption sequence. The mask vector is used in the DPS module for the chaotic configuration of the constellation points with dynamic probability. The entangled chaotic dynamics equations are as follows: $$\{\begin{array}{l}\dot{x}=-ax-by+e\cos w\\ \dot{y}=-ay+bx+e\sin z\\ \dot{z}=-cz+e\cos u\\ \dot{w}=-dw+e\sin y\\ \dot{u}=-hu+e\sin x\end{array},$$(1)where $x$, $y$, $z$, $w$, and $u$ are state variables and $a$, $b$, $c$, $d$, $e$, and $h$ are system parameters. When $a=2$, $b=3$, $c=0.5$, $d=3$, $e=10$, and $h=3$, there is chaotic behavior in this entangled chaotic system, and the chaotic attractor is shown in Fig. 2.

In this module, we propose a DPS scheme. In this paper, we use a 3D constellation structure for data mapping, which consists of two positive cubes, where 16 constellation points are distributed at the vertices of these two cubes. This is shown in Fig. 3. The minimum Euclidean distance between adjacent constellation points is 2. Under the premise of 3D constellation structure mapping, the probability distribution of constellation points is changed to effectively improve the utilization rate of constellation points, thereby effectively reducing the average power and reducing the system BER under the same transmission power.

In the CAP-PON solution, dynamic probability shaping optimizes the point distribution of the 3D constellation through a five-dimensional entangled chaotic system. The chaotic system generates dynamic probabilities for each symbol and constellation point, improving the efficiency of constellation point utilization and enhancing noise resistance. The specific operation is to extract two sequences $X$ and $Y$ from the chaotic model, amplify and round them to generate mask vectors $X1$ and $Y1$, and then apply these vectors to the outer and inner points of the constellation to adjust the dynamic probability, $$\{\begin{array}{l}X1=\mathrm{floor}[\mathrm{mod}(X\times {10}^3,2)]\\ Y1=\mathrm{floor}[\mathrm{mod}(Y\times {10}^3,4)]\end{array}.$$(2)

In the first step, $X1$ is applied to the 8 constellation points at the vertices of the outer cube, and when $X1$ is 0, the data of the original constellation points are kept unchanged; when $X1$ is 1, the data of the outer constellation points are transferred to the adjacent constellation points in the inner layer for modulation to achieve DPS.

In the second step, $Y1$ is applied to all constellation points, and when $Y1$ is 0, the data of the original constellation point are kept unchanged; when $Y1$ is not 0, the data of the constellation point are transferred to the neighboring constellation points in the same layer for modulation according to the rules shown in the figure, in order to achieve the dislocation effect.

The Rubik’s cube was first invented as a mechanical puzzle toy by Professor Erno Rubik of the Hungarian Academy of Architecture in Budapest in 1974. The Rubik’s cube is usually referred to as a third-order Rubik’s cube, which is a $3\times 3\times 3$ cube structure. The number of steps refers to the number of blocks shared by two adjacent rotating surfaces in the main part of the cube. For example, each side of the third-order Rubik’s cube has three small pieces, and the third-order Rubik’s cube has a total of $4.3\times {10}^{19}$ variations through rotation.

Positive $n$ ($n\ge 2$) refers to the number of variations of the Rubik’s cube formula in Eqs. (3) and (4). Below is the odd order: $$N=\frac{8!\times 3^7\times 12!\times 2^{10}\times {(24!)}^{\frac{(n-3)(n+1)}{4}}}{{24}^{\frac{3[{(n-2)}^2-1]}{2}}},n=2k+1,k\ge 1,k\in Z.$$(3)

Below is the even order: $$N=\frac{7!\times 3^6\times {(24!)}^{\frac{n(n-2)}{4}}}{{24}^{\frac{3{(n-2)}^2}{2}}},n=2k,k\ge 1,k\in Z.$$(4)

Therefore, we propose a Rubik’s cube encryption scheme to address the high complexity of Rubik’s cubes. First, we calculate the order of the constructed Rubik’s cube. The total amount of data to be transferred is cubed and rounded to obtain the order $n$ of the Rubik’s cube. The transmitted binary numbers are then stacked into a 3D cubic array model of $n\times n\times n$. Second, we extract the $Z$, $W$, and $U$ sequences from the above chaotic model and multiply each sequence by ${10}^3$ to increase the randomness of the key. Then, we perform coset and rounding on the sequences to obtain the encrypted sequences $Z1$, $W1$, and $U1$, as in Eq. (5), $$\{\begin{array}{l}Z1=\mathrm{floor}[\mathrm{mod}(Z\times {10}^3,3)]+1\\ W1=\mathrm{floor}[\mathrm{mod}(W\times {10}^3,n)]+1\\ U1=\mathrm{floor}[\mathrm{mod}(U\times {10}^3,4)]+1\end{array}.$$(5)

The specific process of the magic cube encryption scheme is divided into three steps. First, the plane of rotation is selected. Second, the number of layers of rotation is determined. Finally, the angle of rotation is determined.

Step 1: As shown in Fig. 4, the three sides of the cube are Plane XOY, Plane XOZ, and Plane YOZ. When $Z1$ is 1, choose Plane XOY to rotate; when $Z1$ is 2, choose Plane XOZ to rotate; when $Z1$ is 3, choose Plane YOZ to rotate.

Step 2: When W1 is $1-n$, that is, determine the $1\mathrm{st}\text{–}n\mathrm{th}$ layers in the plane direction of the first step.

Step 3: When $U1$ is 1, turn the layer clockwise by 90°; when $U1$ is 2, turn the layer clockwise by 180°; when $U1$ is 3, turn the layer clockwise by 270°; when $U1$ is 4, turn the layer clockwise by 360°.

Here, we take the fourth-order Rubik’s cube as an example to demonstrate the encryption scheme under different situations. The details are shown in Fig. 5. A special point to note is that each rotation is operated based on the previous rotation.

The above principles allow the implementation of a 3D constellation DPS and Rubik’s cube encryption scheme. The Rubik’s cube encryption increases the difficulty of cracking through multi-layer rotation in the 3D cube, and the DPS optimizes the signal transmission efficiency and reduces the high-power signals. At the same time, the chaos value introduced by the five-dimensional entangled chaos model provides random input for encryption, DPS hides the encryption mode, and the Rubik’s cube encryption forms a highly complex encryption layer, together building a highly secure communication system.

To verify the performance of the DPS and Rubik’s cube encryption scheme system proposed in this paper, an experimental system was constructed as shown in Fig. 6. The weakly coupled 7-core fiber was used in the experiments, in which the cores of the 7-core fibers are arranged in a hexagonal structure with the highest space utilization. The transmission loss of MCF is less than 0.3 dB/km, while the average insertion loss of the fiber is about 1.5 dB, and the crosstalk between adjacent fiber cores is less than $-50\mathrm{dB}$. The 7-core fiber optic link setup for the experiment is in unidirectional transmission mode, i.e., data are transmitted in one direction only and there is no bidirectional transmission. So the data flow between the fiber cores is in one direction only, there is no attenuation between the cores, which exponentially increases the overall bandwidth potential of the system, and the proposed scheme demonstrates effective crosstalk management.

Specifically, first, at the OLT, the transmitter performs serial/parallel transformation of the original data to obtain the three-way input signal, maps the three-way input signal to the 3D constellation map for mapping encryption processing, and performs DPS processing to obtain the three-way shaping signal. Then the Rubik’s cube encryption is performed on the three-way shaped signals to obtain the three-way encrypted signals. The three encrypted signals are then CAP modulated and combined into one data transmission signal for receiving said data transmission signal at the receiver end and performing inverse transformation to obtain said raw data. The modulated data are encrypted as described above by an offline digital signal processing DSP, and then the encrypted data are imported into an arbitrary waveform generator (AWG, TekAWG70002A) with a sampling rate of 10 GSa/s. An electrical amplifier (EA) amplifies the electrical signal and sends it to a Mach–Zehnder modulator (MZM) for loading onto the optical carrier. The wavelength of the light source used in the experiment is 1550 nm, and the signal is amplified by an erbium-doped fiber amplifier (EDFA) and passed through a 1:8 power splitter (PS), which divides the signal into 7 equal parts and fans into the corresponding cores of the 7-core fiber.

After 2 km of transmission, the transmitted 7-core signal is demultiplexed spatially into a single-mode fiber through a fan-out device. To compensate for the transmission power loss, the EDFA is used again for amplification. The optical signal is regulated for optical power by a variable optical attenuator (VOA) before entering the receiving end. A photodiode (PD) is used to receive the optical signal and perform the photoelectric conversion. Finally, the converted electrical signal is acquired by a mixed-signal oscilloscope (MSO, TekMS073304DX) with a sampling rate of 50 GSa/s. Analog-to-digital conversion (ADC) is also implemented. Finally, the receiver receives said data transmission signal and performs inverse transformation to obtain said raw data, demodulates the data transmission signal with CAP to obtain the three-way encrypted signal, and decrypts the three-way encrypted signal with Rubik’s cube to obtain the three-way shaped data. Then inverse probabilistic shaping and inverse mapping are performed on the three-way shaping data to obtain the three-way input signal, and finally, the three-way input signal is serial/parallel transformed to obtain the original data.

As shown in Fig. 7, the BER curves of the scheme signal after 2 km transmission in the 7-core fiber are shown schematically. It can be found that the BER curves of each fiber core are almost overlap, which indicates that the transmission effect of 7-core fiber has strong stability. As shown in Fig. 7, at the legal ONU, the system BER is $1\times {10}^{-3}$, and the BER starts to appear from the received optical power of $-16\mathrm{dBm}$, indicating that the signal can effectively improve the transmission capability of the system after DPS. With the increasing optical power, the BER of each fiber core has a significant decrease. The optical power difference between the best core and the worst core is 0.63 dB. This also confirms that the 7-core fiber used in this experiment has good uniformity and stability. In addition, with the introduction of Rubik’s cube encryption, the original message cannot be decrypted correctly for the illegal receiver due to the lack of the correct key. With the increase of optical power, the BER of the illegal ONU is kept around 0.49, which shows that the encryption scheme proposed in this paper can effectively improve the security performance of the system.

In addition, we also conducted a comparison group experiment, in which one of the channels was selected to transmit the DPS-3D-CAP signal after DPS and Rubik’s cube encryption and the 3D-CAP signal after Rubik’s cube encryption only. The curve change in Fig. 8 shows that the transmission effect of the DPS-3D-CAP signal after DPS and Rubik’s cube encryption is significantly better than that of the comparison group, and the BER performance of the system is effectively improved due to the introduction of DPS.

The DPS-3D-CAP signal after DPS and Rubik’s cube encryption has an obvious BER at $-18\mathrm{dBm}$ optical power, and the 3D-CAP signal only after Rubik’s cube encryption has an obvious BER at $-17\mathrm{dBm}$ optical power. The transmission efficiency and performance of this scheme are better than those of traditional 3D-CAP transmission at the same power. Meanwhile, when the optical power is $-18.98\mathrm{dBm}$, the system BER of the encrypted DPS-3D-CAP signal is $1\times {10}^{-3}$. The system BER of the conventional 3D-CAP signal encrypted by Magic Square only is $1\times {10}^{-3}$ when the optical power is 17.51 dBm signal, and the receiver sensitivity increases by 1.47 dB.

As shown in Fig. 9, in order to verify the sensitivity of the key, the new five-dimensional entangled chaotic model is disordered for each parameter separately, and the BER after perturbation with different precisions is recorded. A sensitivity test experiment allows rigorous calculation of the key space or initial parameters by adding $1\times {10}^{-\mathit{N}}$ to a single initial value and then testing whether the experiment successfully demodulates the correct data. The key space is determined by adjusting the size of N until the node has a higher BER and is 1 order of magnitude away from complete demodulation. The core 3 with the optical power of $-17\mathrm{dBm}$ at the receiver end in Fig. 7 is taken as an example. The horizontal coordinate of the graph is the accuracy of the change of the initial value of the chaotic sequence; for example, $-19$ means that the parameters of the chaotic model are disturbed by $1\times {10}^{-19}$, and the vertical coordinate corresponds to the BER. When the precision is $-18$, the BER is low and can be decrypted normally. When the precision is $-16$, the BER rises sharply and exceeds the forward error correction (FEC) threshold and cannot be decrypted. This shows that our scheme has high sensitivity to the key. Even if the parameters and initial values of the chaotic model are changed very slightly at the illegal receiver side, it is still very difficult to crack. Conservatively, our key space can reach ${10}^{173}$. Therefore, the key space is sufficient to resist brute force cracking.

In this paper, a CAP-PON scheme is proposed based on DPS and Rubik’s cube encryption in optical access networks, using a new five-dimensional entangled chaos model, which uses the chaos model to generate dynamic probabilities for each symbol and a chaotic configuration of dynamic probabilities for each constellation point, thus improving the utilization of constellation points and enhancing the anti-noise performance of the system, using this technique to perform dynamic probability. The technique is used to perform dynamic probabilistic processing of 3D constellation points. In addition, a Rubik’s cube encryption scheme is used to encrypt the signal using the high complexity property of the Rubik’s cube. Due to the introduction of the chaos model, high-security encryption of constellation points is performed from the physical layer, which significantly improves the security of the whole system. In this scheme, the simultaneous dislocation of multidimensional information by the new five-dimensional entangled chaos system can provide a key space of ${10}^{173}$, which can significantly increase the security of the system and effectively resist attacks from illegal receivers. To verify the transmission and security performance of this scheme, a 3D-CAP signal after DPS and Rubik’s cube encryption is experimentally demonstrated in this scheme for a system transmission over 2 km of 7-core fiber. The experiments show that the sensitivity of the signal after DPS and Rubik’s cube encryption increases by 1.47 dB compared with that of the conventional 3D-CAP signal receiver.