Abstract

The direct optical transportation of images through multimode fibres (MMFs) is highly sought after in compact photonic systems for MMF-based optical information processing. However, MMFs are highly scattering media, thus degrading information transmitted through them. Existing approaches utilize artificial neural networks or spatial light modulators to reconstruct images scrambled after propagation through the fibre. Despite these advances, achieving direct optical image transportation through MMFs using integrated optical elements with micrometre-scale footprints remains challenging. Here we develop a miniaturized diffractive neural network (DN₂s) integrated on the distal facet of a MMF for the direct all-optical image transportation through the fibre. The DN₂s has a footprint of 150 μm by 150 μm and is fabricated on the facet of a 0.35-m-long MMF using three-dimensional two-photon nanolithography. The fibre-integrated DN₂s enables single-shot optical transportation of images with flat phases in real time for a constant configuration of the MMF. The system achieves a minimum image reconstruction feature size of approximately 4.90 μm over a field of view 65 μm by 65 μm when imaging handwritten digits. Transfer learning is also demonstrated by the direct optical transportation of HeLa cell images projected by spatial light modulators, which were not part of the training dataset. The concept and implementation pave the way to the integration of miniaturized DN₂s with MMFs for compact photonic systems with unprecedented functionalities.

Main

Optical fibres are fundamental components in modern science and technology owing to their inherent advantages, including strong light confinement capabilities and efficient light transmission, and have thus been used in numerous applications, including fibre communications^1,2, optical trapping³, nonlinear optical effects analysis⁴, optical endoscopy⁵ and quantum information processing⁶. Compared with single-mode fibres (SMFs)^7,8, multimode fibres (MMFs) can support a much larger number of guided modes (~10³ to ~10⁴), offering the attractive advantage of high-capacity information and image transportation within the diameter of a hair. Such technology is of great importance for a plethora of compact photonic systems, from minimally invasive endoscopy^9,10, to chip-scale mode-multiplexing and demultiplexing for short-distance interconnections^11,12, on-chip quantum photonics^13,14 and hybrid photonic integration¹⁵.

However, MMFs are regarded as highly scattering media^16,17, where the MMF-supported guided modes travel at different velocities as a consequence of the modal dispersion phenomenon¹⁸, leading to spatial wavefront distortions, that is, speckle patterns at the fibre distal facet¹⁹. Thus, MMFs are naturally unsuitable for spatial optical signal transportation owing to modal dispersion, resulting in a fundamental bottleneck for direct spatial signal transmission through MMFs.

Advances in MMF-based technologies involved wavefront-reshaping methods based on spatial light modulators (SLMs) or artificial neural networks (ANNs) to achieve spatial signal transportation or processing through MMFs. The wavefront-shaping methods based on SLMs have demonstrated the transportation of optical information through MMFs, including via digital holography²⁰, the transmission matrix (TM) measurements^21,22, the compressive sampling²³ and the fibre memory effect²⁴. Important progress in optically untangling light propagation through MMFs was theoretically and experimentally demonstrated on the basis of the optical inverse concept^25,26, where a multiplane light converter was utilized to achieve the single-shot propagation of scenes through an MMF. This scheme can be understood as a physical matrix preconditioner or a linear diffractive optical neural network (ONN) designed by the optical inverse method. Recently, the mathematical reconstruction methods based on ANNs were introduced to reconstruct the distorted image by MMFs²⁷, including multilayer perceptron²⁸, deep learning network²⁹ and U-net^30,31. However, the program-based ANNs cannot achieve physical image transportation in real space, and bulk optics-based SLMs pose a fundamental footprint limit towards micrometre-scale spatial optical signal transmission for MMF-based technologies on the tip of a fibre in compact photonic systems^32,33,34.

Regarded as an ONN³⁵, the free-space diffractive neural networks (DN₂s)³⁶ have been proposed as more efficient ANN approaches based on deep learning to directly process the optical matrix multiplication at the speed of light, and realizing the high number of connectivity in ANNs, such as optical image classification³⁷, decryption and phase detection³⁸. The DN₂s offer unprecedented parallel processing capability for multidimensional optical inputs, where synthetic complex optical fields can be manipulated³⁹. Furthermore, the free-space ONNs have also been demonstrated to reconstruct optical fields for scattering medium⁴⁰, such as frosted glass⁴¹, and opaque occlusions⁴² using SLMs or three-dimensionally printed phase plates.

Thereby, introducing a fibre-integrated miniaturized DN₂s to achieve the direct optical image transportation through MMFs would alleviate the necessity to use bench-top SLMs and is of great importance for current MMF-based technologies, including optical communications, optical computing and quantum science. However, traditional fabrication methods, such as ion-beam lithography⁴³, and chemical-etching techniques⁴⁴, are not suitable for three-dimensional (3D) MMF-integrated fabrication, as they fail to create 3D arbitrary structures for efficient light field manipulation at the fibre facet.

Here, we report the direct all-optical image transportation through a MMF using a miniaturized DN₂s integrated at the distal facet. We used the 3D galvo-scanning two-photon nanolithographic (GS-TPN) fabrication approach to integrate a miniaturized DN₂s with a footprint of 150 μm by 150 μm at the distal facet of a commercial 0.35-m-long MMF. Due to the physics-driven nature of the DN₂s and the TM obtained according to the accurate vectorial theory of light propagation in the MMFs⁴⁵, the fibre-integrated miniaturized DN₂s based on the multilayer diffractive elements demonstrates the direct optical image transportation at the distal facet of an MMF.

Optical images at the fibre input can be reconstructed at the output of the DN₂s for constant configuration of the MMF. The well-trained DN₂s at the distal facet of the MMF achieved an image reconstruction feature size of approximately 4.90 μm with an image size of 65 μm by 65 μm, and an average optical intensity contrast of around 4%, corresponding to around 35% diffraction efficiency per diffractive layer. The fibre-integrated miniaturized DN₂s also demonstrates a transfer learning characteristic, as the DN₂s is equivalent to the inverse TM through an optical inference process based on the position-related TMs used in the training process. Transfer learning schemes were applied to the fibre-integrated miniaturized DN₂s for the direct optical transportation of SLM-projected images of HeLa cells. Alternatively to the multiplane light converter designed by the optical inverse method²⁶, the fibre-integrated miniaturized DN₂s was developed on the basis of the optical forward propagation and optimization using the connectivity in ANNs. This concept alleviates the technological necessity of utilizing SLMs for future MMF-based technologies in compact photonic systems, including miniature rigid endoscope⁴⁶, short-range optical communications¹⁷ and quantum photonics⁶.

Fibre-integrated DN₂s for MMF image transportation

The conceptual design of the fibre-integrated miniaturized DN₂s for MMF image transportation is illustrated in Fig. 1. A commercial MMF with a core diameter of 105 μm with a numerical aperture (NA) of 0.22 is utilized to transmit the optical images directly as a key component for MMF image transportation. A coherent, monochromatic continuous-wave laser at a wavelength of 532 nm was chosen to generate the optical images incident into the proximal facet of the MMF. This MMF supports around N = 4,650 spatial modes per polarization channel (given by N ≈ (πdNA/2λ)², where d is the diameter of the fibre and λ is the wavelength of the incident light). Due to the varied velocities of a large number of guided modes in the MMF, the propagating optical images became speckle patterns at the MMF distal facet. The well-trained DN₂s integrated at the distal facet of the MMF function as a physical wavefront correction device, where the amplitude and phase information propagating out of the MMF can be manipulated through the multiple layers of the DN₂s, and desired optical images are reconstructed. Consequently, the miniaturized DN₂s at the MMF distal facet achieve direct optical image transportation without the need to use SLMs.

Fig. 1: Schematic illustration of the fibre-integrated miniaturized DN₂s for all-optical image transportation through a MMF.

Integration of the vectorial model of a MMF and DN₂s

To accurately achieve the amplitude and phase correction at the distal facet of the MMF using miniaturized DN₂s, the accurate vectorial model of an MMF should be utilized in the training process of the DN₂s. However, unlike the traditionally widely used SMFs, where the transmitted modes are well defined as Gaussian distributions, the propagating modes in MMFs were believed to be chaotic, the mathematical modelling of which has been achieved recently⁴⁶. These variations in the amplitudes and phases induced by the MMF mean the data cannot be directly applied to train DN₂s, which require accurate datasets including the inputs and output speckles (amplitude and phase). Instead of adopting recently developed methods such as the digital holography method to measure the TM, we introduced the deterministic vectorial light propagation model within MMFs to train the DN₂s (see Supplementary Note 1 for modelling details). We obtain the vectorial modelling results of the TMs for a 35-cm-long, 105-μm-core-diameter step-index MMF (Fig. 2a). Each row in the TM represents the amplitudes and phases of all output focal points for a single-input focal point mode sent into the fibre. Due to space constraints, the basis of modes for the TMs shown is reduced to a quarter of the full dimension, while the complete TMs are shown in Supplementary Fig. 1.

Fig. 2: Development process of the fibre-integrated miniaturized DN₂s based on the complex TM of the MMF.

Numerical performance of the fibre-integrated DN₂s

After the acquisition of the amplitude and phase information of the MMF-induced speckle patterns, the overall training datasets were used to train the multilayer DN₂s for fibre end speckle reconstruction. During the training process, the multilayer DN₂s diffracted and modulated many inputs numerically on the basis of the Rayleigh–Sommerfeld theory³⁷. The schematic training process of the DN₂s is shown in Fig. 2b, and further details about the training process are discussed in Supplementary Notes 3 and 4. Specifically, the first layer (the input layer) receives the amplitude and phase distribution from the fibre distal facet, and through the optical inference process performed in each of the diffractive layers, the wavefront of the fibre output can be modulated. Due to the physical wavefront manipulation fact, the output intensity was directly optimized to desired distributions in the training process; thus, the intensity distribution over (x, y) in the output plane of the DN₂s was captured using a charge-coupled device (CCD) camera in the experimental setup. In consequence, a direct optical imaging process was realized using the fibre-integrated miniaturized DN₂s. In the training of DN₂s, a stochastic optimization method, namely adaptive moment estimation (Adam), was utilized to optimize the training process. The negative Pearson correlation coefficient (NPCC) and structural similarity index measure (SSIM) were both used to evaluate the MMF speckle pattern reconstruction performance of the DN₂s. To efficiently train the DN₂s, a recurrent training method was used. For every 10 training epochs of the total 200 training epochs, a group of test data was sent into the DN₂s model, and the performance was evaluated by calculating the provisional SSIM and NPCC. If the SSIM is approaching 1, and NPCC approaching −1, the optimization of the DN₂s is believed to be evolving in the correct direction, and the optimization continues. Otherwise, the algorithm reoptimizes the neural networks for higher SSIM and lower NPCC.

The training and testing results of the DN₂s are shown in Fig. 3a. The evolution of the NPCC and the SSIM with an increasing number of training epochs suggested a well-defined recurrent training process, with the lowest NPCC value reaching −0.725 and the highest SSIM reaching 0.68. These results show that the DN₂s were a successful model for retrieving the speckled optical field produced by an MMF. The numerical test results for speckle pattern reconstruction are shown in Fig. 3b; an average SSIM of 0.68 was obtained for the test dataset (handwritten digits from 0 to 9).

Fig. 3: Performance of the fibre-integrated miniaturized DN₂s.

Fabrication of the fibre-integrated miniaturized DN₂s

To integrate the DN₂s on the distal facet of the MMF and achieve efficient wavefront manipulation for the output beam, we designed and printed a hollow tower structure on the distal facet of the MMF to expand and fully capture its output in free space and to hold the fibre-integrated miniaturized DN₂s. A built-in-house 3D GS-TPN fabrication system was used to achieve this goal, owing to its advantages in fabricating arbitrary 3D structures with nanometre feature sizes, especially free-standing 3D structures on small surfaces (see Supplementary Note 10 for a detailed discussion). The phase values of each diffractive neuron in the DN₂s were calculated to generate relative height distributions and then printed with IP-dip (commercial photoresist from Nanoscribe GmbH) in the shape of a cuboid. The 3D GS-TPN fabrication results are reported in Fig. 4a–c. The final DN₂s fabricated on the distal facet of the MMF contained three diffraction layers with distinct neuron patterns; each of the diffractive neurons had a width of approximately 875 nm, the neuron density in the three-layer DN₂s was 3.92 × 10⁸ neurons cm⁻², and the phase could be accurately modulated between 0 and 2π.

Fig. 4: Fabrication of the fibre-integrated miniaturized DN₂s.

Experimental performance of the fibre-integrated DN₂s

We experimentally tested the performance of the fibre-integrated miniaturized DN₂s in reconstructing scrambled optical images transmitted through MMFs. In the experiments, a coherent laser source and a SLM were used to project optical images on the proximal facet of the MMF, and the output was detected with a CCD camera at the distal facet. The capturing of the reconstructed optical images was obtained by moving the imaging objective at different axial positions away from the fibre-integrated miniaturized DN₂s. The corresponding results are shown in Fig. 5a. The experimental layout and detailed description of the characterization procedure are shown in Supplementary Fig. 23.

Fig. 5: Experimental characterization of the fibre-integrated miniaturized DN₂s in the straight state.

Capability of transfer learning

Traditional ANN-based machine learning algorithms suffer from a common problem called transfer learning. In other words, ANNs are not capable of performing image reconstruction when the input datasets belong to a different category compared with the training dataset²⁹. It is discovered that the proposed fibre-integrated miniaturized DN₂s in our work can reconstruct SLM-projected HeLa cell images from the MMF speckles (Fig. 5d), with an average SSIM value of approximately 0.53, even though the HeLa cell images were not included in the training of the DN₂s. The cellular optical image reconstruction results demonstrate that the miniaturized DN₂s at the distal facet of the MMF can be potentially utilized in biological endoscopic imaging scenarios. The transfer learning characteristic is a direct result of the physics-driven nature of the DN₂s, where the DN₂s represent position-related input–output correlations in MMF-based systems, with inputs at certain positions generating different output patterns. Consequently, the DN₂s enable an outstanding transfer learning capability.

Fibre bending effect on image reconstruction

Another challenge in using MMF for image transportation is the deformation effect of the fibre, where the disorder induced by the fibre deformation/bending greatly impairs the image reconstruction properties (the schematic diagram is shown in Fig. 6a). In the DN₂s, the effect of fibre deformation on the performance of the fibre-integrated miniaturized DN₂s was considered by incorporating the deformed-fibre dataset with the straight-fibre dataset DN₂s training process. In this process, the TM for a 35-cm-long MMF bent at an angle of 90° was first obtained (Supplementary Fig. 5), and 6,000 training data pairs and 500 test data pairs for the deformed fibre state were generated and introduced into the training process.

Fig. 6: Experimental characterization of the fibre-integrated miniaturized DN₂s in a bent state (bending angle of 90°).

Methods

3D GS-TPN

The fibre-integrated miniaturized DN₂s integrated at the distal facet of the MMF was achieved using a 3D GS-TPN fabrication system built in-house that incorporates a femtosecond laser source (Coherent Axon 1064), second-harmonic generation module (Coherent SGH), piezo nanotranslation stage (PI-545), one pair of two-dimensional galvo mirrors (Thorlabs GVS002), scan lens (Thorlabs SL50-CLS2) and tube lens (Thorlabs TTL200MP) utilized to reduce the focus aberration during the galvo-scanning fabrication process. The mechanical and dynamic control of the system is controlled by a home-developed program, by which the 3D CAD models can be translated into the fabrication documents required by the program. The galvo mirrors rotate at a fast speed without the movement of the nanopiezo translation stage, thus enabling ultrafast movement of the fabrication voxel.

During the fabrication of the miniaturized DN₂s integrated at the distal facet of the MMF, a 35-cm-long MMF (Thorlabs, FG105LCA, NA 0.22) with a core diameter of 105 µm was first cleaved to obtain a flat end face. Then, the MMF was installed on a specially designed microstage that could be used to adjust the orientation of the MMF. A hollow tower structure with a height and width of 75 µm and 150 µm, respectively, was designed to cover the total output facet of the MMF to ensure efficient wavefront manipulation on the MMF. The height of the hollow tower structure was determined according to the direct output facet of the MMF. To prevent the fibre beam from impacting the sidewall of the tower, the first layer in the DN₂s was designed to have a diameter much larger than the mode field diameter of the fibre output. A video of the integration process is demonstrated in Supplementary Video 4, in which the video clips showing the multilayer fabrication process are provided and each diffraction layer was carefully imaged along the z direction. Due to the high-precision alignment and fabrication capability in the 3D GS-TPN system, the multilayer DN₂s can be successfully integrated on the fibre tip. A close-up scanning electron microscopy (SEM) image verifies the successful fabrication of the designed fibre-integrated miniaturized DN₂s (Fig. 4). Further details regarding the design and fabrication processes can be found in Supplementary Figs. 20–22.

Characterization setup

In the characterization experiments, a coherent continuous wave laser with a wavelength of 532 nm (JCOPTIX-532), a SLM (LBTEK-2K-8), a polarized beam splitter (JCOPTIX-PB532), to generate the optical images incident on the fibre-proximal facet, and a 4-f imaging system were aligned with two objectives with a magnification of 20× and a NA of 0.4 (Olympus 1-U2B225) at the proximal and distal facet of the MMF. This setup can generate optical holographic images and images incident on the proximal facet of the MMF. The alignment of the MMF is controlled by a pair of 3D fibre alignment stages (Newport 466A). Details of the layout of the setup are shown in Supplementary Fig. 23.

Training and test data

The MNIST handwritten datasets are obtained from the MNIST database with original pixel numbers 28 × 28. The HeLa Cell image datasets are obtained from the Murphy lab (http://murphylab.web.cmu.edu). All the datasets are resized to have pixel numbers of 120 × 120 in this work. In addition, the dataset with complicated natural scenes (the Imagenet database, https://www.image-net.org/) was also utilized to numerically test the performance of the fibre-integrated miniaturized DN₂s in reconstructing the scrambled natural scenes through MMF, and the detailed results and discussion are presented in Supplementary Note 7.

Experimental obtained data enhancement

The experimentally obtained images are normalized by the intensity distribution per image, where the contrast of patterns is mathematically enhanced by a MATLAB code. A schematic diagram of the data enhancement process is shown in Supplementary Fig. 24.