Abstract

Multiplexed fluorescence imaging provides valuable biological insights from the cellular to the tissue level but remains limited in live-mammal studies by the lack of a fluorescent palette capable of overcoming photon scattering and autofluorescence noise for real-time, multiplexed in vivo imaging. Here we present a fluorophore palette engineered from erbium(III)-phthalocyanine complexes, termed the lanthanide rainbow (Lanbow), which offers tunable near-infrared absorption and a unified 1,530 nm emission with brightness surpassing that of existing molecular dyes. Lanbow uses excitation-encoded and efficient single-band detection in the 1,500–1,900 nm shortwave infrared subregion, where tissue scattering and autofluorescence are minimized, enabling up to nine-colour imaging in deep tissues. We also demonstrate fluorescence-guided surgery featuring multiparametric anatomical identification and functional assessment, with deep-learning networks automating real-time analysis for intraoperative guidance. This study establishes a transformative platform for real-time, highly multiplexed imaging in live mammals.

Main

Over the past decades, the expanding spectral palette of fluorescent probes has transformed optical multiplexing through information-rich spectral readouts^1,2,3,4,5. This advancement underpins technologies such as spectral flow cytometry^6,7 and multispectral imaging^8,9, producing high-content datasets that drive artificial intelligence (AI)-powered bioimage analysis^10,11 and accelerate discovery from the single-cell¹² to tissue scale¹³. Extending these capabilities to live mammalian systems offers exciting opportunities for translational imaging. There, capturing high-content real-time functional information in native environments could reveal critical insights into physiology and disease. However, the inherent opacity of mammalian tissues remains a major obstacle for optical interrogation. In the emission range of most conventional fluorescent probes (400–1,000 nm)^14,15,16,17, photon scattering and tissue autofluorescence reduce signal penetration and contrast¹⁸, with spectral fidelity degraded in a wavelength-dependent manner. Currently, there is no available fluorescent probe palette that can overcome these challenges and support high-fidelity spectral techniques for real-time multiplexed imaging in vivo.

As part of the shortwave infrared or second near-infrared (SWIR/NIR-II, 1,000–2,000 nm) region, the 1,500–1,900 nm wavelength range has recently emerged as an optimal region for high-resolution deep-tissue imaging^19,20,21. This window offers reduced photon scattering and minimal autofluorescence, while moderate water absorption suppresses multiply scattered diffusive noise²², enhancing imaging contrast (Fig. 1a). However, leveraging these advantages for multiplexed imaging remains a formidable challenge in both fluorophore design and spectral imaging technologies. Fluorescent probes emitting in this range are scarce, and those available often suffer from small Stokes shifts, broad spectral bandwidths and substantially reduced brightness owing to increased electron–vibrational coupling at longer wavelengths^23,24. These limitations are further exacerbated by the photon inefficiency of conventional emission-based spectral imaging, which relies on multiple narrow band-pass filters with each capturing only a fraction of emitted photons^25,26, making real-time in vivo multiplexing impractical.

Fig. 1: Development and expansion of Lanbow fluorophore palette for live mammalian multispectral imaging.

Lanbow with tunable absorption and single 1,530 nm emission

The Er³? ion emits a narrowband signal around 1,530 nm owing to the inner-shielded 4f–4f electron transition ⁴I_15/2–⁴I_13/2 (ref. ²⁷), making it highly suitable for high-contrast biological imaging. Although this transition is formally forbidden, it can be enabled through energy transfer from light-harvesting dye ligands²⁸, a process known as sensitization, which enhances population levels for radiative emission (Fig. 1b). We first identified the sensitization effect of a phthalocyanine dye in a sandwiched structure, where an Er³? ion is positioned between the dye and a tripodal ligand. This complex (EP673) exhibited bright 1,530 nm emission under 670 nm excitation at the phthalocyanine absorption band (Fig. 1c,d). In this structure, phthalocyanine metalization enhances symmetry, leading to narrowband absorption owing to degenerate electronic transitions²⁹. This insight led us to hypothesize that, by preserving the sandwich design and fine-tuning phthalocyanine absorption through substituent engineering, we could create distinct multispectral identities, while energy-level decoupling between the absorption and emission units would enable a consistent emission output. However, maintaining efficient and uniform sensitization across the palette remains a key challenge.

To test this, we synthesized hybrid fluorophores with three types of substituent (alkyl, ether and amine) at the α, β or both positions of the four peripheral aromatic rings (Fig. 1e and Supplementary Note 1). These fluorophores, designated EP673 to EP772, feature Er³? ions sandwiched between a tripodal ligand and substituted phthalocyanines (Supplementary Fig. 1 and Supplementary Table 1) and exhibit peak absorption wavelengths from 673 nm to 772 nm (Fig. 1f), with narrow bandwidths (full-width at half-maximum (FWHM) of 20–48 nm) and high molar absorptivities (? >10⁵ M⁻¹ cm⁻¹) (Supplementary Table 2). In addition, a unified emission at 1,530 nm was observed, with excitation spectra closely matching their absorption profiles (Supplementary Fig. 2).

Through excited-state dynamics studies of ether-substituted fluorophores, we identified a triplet energy transfer mechanism in which phthalocyanine ligands absorb light, undergo intersystem crossing (ISC) to triplet states at approximately 9,300–8,700 cm⁻¹ and transfer energy to Er³? with efficiencies of up to 81% (Supplementary Note 1). While not all fluorophores could be technically characterized, the consistently high emission brightness across the palette (>12 M⁻¹ cm⁻¹; Supplementary Table 2) suggests the presence of a universally efficient sensitization mechanism. These findings deepen our understanding of lanthanide sensitization and demonstrate ligand fine-tuning as a generalizable design principle for constructing bright, spectrally programmable Lanbow fluorophores.

Compared with previously reported molecular dyes in the same emission range (Fig. 1g and Supplementary Table 3), our fluorophores exhibited approximately 4-fold higher average brightness, 7-fold narrower average FWHM in the readout spectra and 5-fold longer average Stokes shifts, exceeding 750 nm. Taken together, the Lanbow fluorophore palette overcomes the common photophysical limitations that typically constrain conventional fluorophores, paving the way for scalable excitation multispectral imaging in the shortwave infrared window.

Multispectral imaging with Lanbow and its performance in mimicking tissue

To evaluate the imaging performance of Lanbow, we developed a multispectral imaging method using an indium gallium arsenide (InGaAs) camera to capture single-band fluorescence signals in the 1,500–1,700 nm range. Excitation was provided by an optical parametric oscillator (OPO) laser with continuously tunable wavelengths, scanning in 5 nm increments from 670 nm to 790 nm to generate image datasets with 25 spectral channels (Fig. 2a). Benchmarking against conventional multi-emission-based SWIR spectral imaging remains challenging owing to the limited availability of variable band-pass filters and the constraints of existing methods, which have primarily demonstrated two-colour resolution²⁵. Notably, phthalocyanine ligands exhibited tail emissions in the 850–1,000 nm range under identical excitation conditions. We collected these emissions using the same excitation-scanning set-up, ensuring an unbiased comparison of emission wavelength effects.

Fig. 2: Performance of Lanbow coupled with multispectral imaging in scattering and autofluorescence tissues.

Multispectral imaging performance with Lanbow in autofluorescence tissue

We next evaluated whether Lanbow-enabled multispectral imaging could overcome tissue autofluorescence. EP737 and EP772 were formulated with Pluronic F127 micelles (Supplementary Fig. 3) and administered intravenously and intragastrically to mice for whole-body imaging in the 1,500–1,700 nm and 850–1,000 nm ranges (Fig. 2h). Mapping the 1,500–1,700 nm dataset onto the phasor plane revealed a linear combination of two endmembers (pure spectral signatures of pixels belonging to a single fluorophore), EP737 and EP772 (Fig. 2i, top). By contrast, the 850–1,000 nm dataset exhibited a triangular distribution, incorporating an additional broad autofluorescence signature (Fig. 2j, top). The spectral signatures of EP737 and EP772 in this range were broadened by scattering, while the autofluorescence signal showed no clear tissue correlation (Supplementary Fig. 4), likely originating from multiple heterogeneous sources³³. As a result, unmixing the 850–1,000 nm dataset only partially removed autofluorescence, obscuring vascular structures (Fig. 2j, bottom). By contrast, unmixing the 1,500–1,700 nm dataset successfully resolved vascular and intestinal features (Fig. 2i, bottom), achieving threefold higher vascular contrast (Fig. 2k) and a 1.4-fold lower average relative residual (ARR; Fig. 2l), which quantifies the difference between original and unmixed data. These findings demonstrate that Lanbow-enabled multispectral imaging effectively eliminates tissue autofluorescence interference, simplifying spectral analysis.

Dynamic multispectral imaging with Lanbow in mouse model

We investigated whether Lanbow-enabled multispectral imaging could provide high-contrast, information-rich visualization in live mammals. Our primary focus was on fluorescence-guided surgery, which has been clinically proven to enhance surgical outcomes^34,35,36. However, current fluorescence-guided surgery techniques typically rely on single-colour visualization, limiting their ability to capture the complex dynamics of surgical environments. Real-time, high-contrast multi-parameter imaging is crucial for accurately assessing anatomical and functional changes, particularly in advanced robot-assisted procedures^37,38 that demand higher precision and intraoperative adaptability.

In colorectal cancer surgery, accurate lesion localization, margin assessment and lymphadenectomy guidance are critical for optimizing resection and minimizing recurrence³⁹. In addition, dynamic assessments of vascular anatomy, perfusion and gut function provide critical intraoperative insights in the selection of optimal bowel resection sites for reducing complications such as anastomotic leakage and reoperation^40,41. We then applied our method in a colorectal cancer mouse model to demonstrate the ability for intraoperative imaging of multiple targets, including blood vessels, lymph nodes, tumours and the intestinal tracts.

We investigated the fluorophore biodistribution in mice for accurate anatomical labelling (Extended Data Fig. 3 and Supplementary Fig. 5). We identified F127 and bovine serum albumin (BSA) as effective formulations for erbium(III)-phthalocyanine (EP) fluorophores, yielding nanoparticles with sizes ranging from 10 nm to 30 nm and exhibiting long-term spectral stability for up to 7 days in phosphate-buffered saline (PBS) and fetal bovine serum, as well as across a pH range of 3–10 (Supplementary Fig. 3). F127-formulated fluorophores exhibited prolonged vascular circulation, with tumour-to-tissue ratio peaking at 72 h post-injection, likely owing to the abnormal vasculature. EP772, which binds to BSA, rapidly cleared via the reticuloendothelial system and accumulated in mesenteric lymph nodes, generating sustained signals. To simultaneously label mesenteric lymph nodes and tumours at the time of surgery, we intravenously injected EP772/BSA and EP725/F127 at 73 h and 72 h pre-imaging, respectively (Fig. 3a). For colonic and intestinal labelling, EP737/F127 and EP679/F127 were administered via intragastric injection 2.5 h and 0.5 h before surgery, respectively, allowing natural transit through different intestinal segments. Immediately before imaging, EP699/F127 was injected intravenously to label blood vessels. Mice were then imaged using a custom-built wide-field dynamic imaging system, using sequential laser excitation in five channels (671 nm, 690 nm, 730 nm, 760 nm and 785 nm) and frame-synchronized exposure in an InGaAs camera (Fig. 3b). Dye concentrations and illumination intensities showed minimal effects on excitation spectra consistency (Supplementary Note 3). Photostability assessments of all five fluorophores under these conditions demonstrated consistent excitation profiles over time, supporting stable spectral unmixing during extended dynamic imaging (Supplementary Fig. 6).

Fig. 3: Five-colour live-mammal imaging with Lanbow.

AI-powered automated workflow facilitates real-time live-mammal multispectral imaging

Our method was successfully replicated in 21 additional individual mice (data publicly available on Zenodo and Huggingface). The collected high-quality datasets allowed us to leverage AI to streamline the specialized analysis workflow. To illustrate this concept, we developed and trained EndmemberNet, a two-stage neural network designed for real-time endmember extraction from multispectral data streams (Fig. 4a and Supplementary Note 4). The core principle of EndmemberNet is to detect tissue regions where the spectral signatures are dominated by a single fluorophore; thus we term ‘pure regions’. Examples include mesenteric arteries exclusively labelled with EP699 and the primary tumour labelled with EP725 during imaging. EndmemberNet processes a time-point multispectral dataset as input, beginning with a weighted fusion step that combines images captured at different wavelengths to enhance wavelength-specific information (Supplementary Fig. 7). In the first stage, the endmember detection neural network (Supplementary Fig. 8) predicts detection boxes, localizing suspected pure regions within each image. By independently analysing augmented images, the network maintains wavelength awareness, effectively capturing spectral variations across different wavelengths. Detection results are then aggregated based on confidence scores to determine the most probable pure spectral signature for each fluorophore (Supplementary Fig. 9). In the second stage, the endmember segmentation neural network (Supplementary Fig. 10) refines the segmentation of pure regions. These regions are averaged to derive accurate endmember spectral signatures, which are subsequently used for unmixing (Supplementary Fig. 11). In our study, the available high-quality datasets, with manually annotated ‘pure regions’, were sufficient for training without the need for additional large-scale datasets.

Fig. 4: Lanbow-derived EndmemberNet realizes real-time live-mammal multispectral imaging.

Discussion

We have developed Lanbow, a SWIR fluorophore palette engineered from erbium(III)-phthalocyanine complexes for live mammalian fluorescent multiplexing. Unlike existing fluorescent dyes and proteins with multiple emissions, Lanbow fluorophores feature tunable absorption and a single 1,530 nm emission, introducing a new spectral colour paradigm. They also offer higher brightness, larger Stokes shifts and narrower spectral bandwidths than current SWIR fluorescent dyes in similar emission ranges. To leverage these advantages, we established an efficient excitation-scanning multispectral imaging method with single-band emission collection within the 1,500–1,900 nm ‘tissue-transparent’ window. This approach overcomes the photon efficiency limitations and wavelength-dependent performance variability of multi-emission-based multispectral methods, ensuring consistent high-contrast imaging across all channels and significantly enhancing multiplexing capacity in live mammalian imaging. In our demonstration, we achieved nine-colour multiplexing with minimal crosstalk in deep tissues using a minimal set of excitation channels matching the number of fluorophores. This advancement enhances fluorescence-guided surgery by enabling real-time visualization and quantification of dynamic intraoperative multiparametric information. Furthermore, the robust data quality facilitated the development of an AI-powered automated online processing pipeline, eliminating reliance on labour-intensive workflows and expert knowledge, paving the way for broader clinical accessibility (Supplementary Table 5).

Described in this work, the Lanbow concept leverages a single Er³⁺ ion to produce a theoretically unlimited palette of colours by molecularly engineering the excitation profiles of sensitizing ligands. Looking ahead, this strategy holds great promise for extension to other lanthanide ions such as Yb³⁺, Nd³⁺, Ho³⁺ and Tm³⁺, which offer orthogonal, narrowband SWIR emission channels. In principle, this scalability can be achieved through ligand design tailored to the specific energy levels of each ion. However, several challenges remain: (i) the triplet energy of the ligand must be carefully matched to the excited states of the target lanthanide to ensure efficient energy transfer, and (ii) certain lanthanide emissions, such as those from Tm³⁺, are susceptible to strong vibrational coupling, which can quench radiative transitions and reduce brightness. Overcoming these challenges could expand the current erbium-based system into a broader ‘lanthanide rainbow’, offering unprecedented spectral programmability beyond what is achievable with existing fluorophore platforms such as organic dyes, fluorescent proteins, quantum dots and lanthanide nanocrystals. Such a system would support high-throughput multiplexed imaging by combining excitation multispectral method with orthogonal emission channels. Moreover, the molecular engineering of ligands is not limited to spectral tuning. It could also enable functional sensing capabilities through responsive changes in excitation profiles. Collectively, these advances would enable highly versatile imaging and sensing applications across scales from cellular and tissue imaging to whole-organ and live-animal studies.

Methods

Materials and synthesis

For all materials and synthesis details, structural and photophysical characteristics, see Supplementary Figs. 14–41, Supplementary Methods and Supplementary Tables 1 and 2.

General set-up for multispectral imaging

A general wide-field imaging configuration was used. Excitation light was delivered through a metal-cladded multimode fibre, collimated and expanded before being directed onto the sample plane via a dichroic mirror. The reflected light was perpendicularly projected onto the sample surface, ensuring uniform illumination. Emitted fluorescence was collected through the dichroic mirror, filtered and focused onto an electrically cooled InGaAs camera (NIRvana 640, 640 × 512 pixels; Princeton Instruments) using a 50 mm focal length lens (Navitar, SWIR-50). Er³? emission was collected using a 1,500 nm long-pass (LP) filter (Thorlabs, FELH1500), while phthalocyanine ligand emission was collected using an 850 nm LP filter (Thorlabs, FESH850) and a 1,000 nm short-pass (SP) filter (Thorlabs, FELH1000).

High-resolution spectral sampling used a femtosecond OPO laser (Coherence, Chameleon Discovery NX) as the tunable excitation source, providing continuous wavelength tuning with 1 nm spectral resolution. The laser output was coupled into a 450 μm core metal-cladded multimode fibre (Changchun New Industries Optoelectronics Tech.) for delivery. The collimated, expanded and homogenized excitation light was incrementally scanned across a wavelength range of 670–790 nm in 5 nm steps, ensuring precise excitation of the sample.

Dynamic compressive sampling used a continuous-wave laser system for synchronized, sequential illumination. Excitation light from five discrete semiconductor lasers (671 nm, 690 nm, 730 nm, 760 nm and 785 nm; Changchun New Industries Optoelectronics Tech.) was combined using a five-in-one metal-cladded multimode fibre (200 μm input, 400 μm output) and homogenized with a liquid-core fibre. The laser power densities were calibrated to 100 mW cm⁻² for 671 nm, 690 nm, 730 nm and 760 nm, and 125 mW cm⁻² for 785 nm. A custom-designed add-in module in LightField imaging software (Princeton Instruments, 4.17.7.2311 version) enabled precise control of laser exposure times and inter-laser intervals. Synchronized operation of the laser drivers and the InGaAs camera was achieved using transistor–transistor logic (TTL) signals generated by a data acquisition device (National Instruments). The emitted light was filtered through a 1,500 nm LP filter and focused onto the InGaAs camera. The analogue-to-digital conversion rate of camera was set to 12.5 kHz, with the gain adjusted to the high-sensitivity mode. Each laser pulse lasted 50 ms, during which the camera exposure was synchronized with the illumination. A 10 ms delay between laser triggers ensured complete readout of each image frame, maintaining high fidelity in the dynamic spectral data acquisition process. See Supplementary Note 6 and Supplementary Table 4 for a detailed tutorial for in vivo real-time multispectral imaging set-up.

Tissue phantom study

Intralipid solution (1% v/v) was chosen as a tissue phantom owing to its similar absorption and scattering properties compared with biological tissues. Nine glass capillary tubes each filled with THF solutions of EP fluorophores (EP673, EP679, EP699, EP720, EP725, EP737, EP765 and EP772) and EB766 (in this study, our previously reported EB766 (ref. ⁴⁷), which exhibits a similar monochromatic Er³? emission, was used to replace EP680 owing to its overlapping spectra with EP679) were parallelly placed under a cylindrical culture dish and covered with different volumes of 1% intralipid for imaging. The series numbers 1–9 represent the fluorophores EP673, EP679, EP699, EP720, EP725, EP737, EP765, EB766 and EP772. Multispectral datasets (x, y, λ) were acquired using the general wide-field imaging configuration equipped with an OPO laser for high-resolution sampling. The generated multispectral datasets underwent phasor analysis, where excitation spectra were extracted and subsequently corrected by wavelength-dependent energy distribution of the excitation light (Supplementary Fig. 13).

Preparation of F127 micelle formulation of EP fluorophores

EP fluorophore (1 μmol) in 1 ml of chloroform was mixed with 6 ml of F127 (25 mg ml⁻¹) chloroform solution. The solvent was removed by rotary evaporation. Under ultrasonic oscillation, 5 ml of PBS (pH 7.4) was added, followed by ultrafiltration (30 K Amicon Ultra, 4,500 rpm for 10 min). The final formulation, containing 500 µM fluorophores in 2 ml PBS, was stored at 4 °C.

Preparation of EP fluorophore/BSA complex

EP772 in DMSO (500 µM, 400 μl) was added dropwise to 10 ml of PBS containing 13.4 mg BSA, and sonicated for 10 min. The solution was concentrated by ultrafiltration (4,500 rpm, 10 min). The final solution of EP772/BSA complex (400 μl) was stored at 4 °C.

Animal handles

All animal procedures were performed in accordance with the guidelines of the Institutional Animal Care and Use Committee of Fudan University, in agreement with the institutional guidelines for animal handling. All of the animal experiments were authorized by the Shanghai Science and Technology Committee. Female BALB/c mice (6–8 weeks old, average weight of 20 g) were purchased from Shanghai JSJ Laboratory Animal, randomly allocated and housed in specific pathogen-free and standard environmental conditions (22–25 °C, 45–55% humidity, 12 h/12 h dark/light cycle) with free access to water and standard laboratory food and randomly selected from cages for all imaging experiments. Before imaging, all the mice were anaesthetized by an intraperitoneal injection with Avertin (2%, v/v, 100 μl/10 g body weight). The mouse was kept anaesthetized using a nose cone delivering 2 l min⁻¹ air mixed with 4% isoflurane throughout imaging.

Dual-colour excitation multispectral imaging in mouse

Before imaging, 150 μl (500 μM) EP772/F127 micelles were administered intragastrically 2 h prior, while 200 μl (500 μM) EP737/F127 micelles were injected intravenously via the tail vein. The mouse was then killed post-injection to avoid artefacts during spectral scanning. Multispectral datasets (x, y, λ) were acquired using the general wide-field imaging configuration equipped with an OPO laser for high-resolution sampling.

Orthotopic colorectal cancer mouse model

The murine colorectal carcinoma cell line CT26 (catalogue number TCM37) was cultured in a sterile environment for the orthotopic colorectal cancer mouse model. Female BALB/c mice were anaesthetized with isoflurane and positioned in dorsal recumbency on a flat surface maintained at 37 °C throughout the procedure. The abdominal skin was aseptically prepared using iodophor disinfectant. A midline abdominal incision (1–2 cm) was made, starting approximately 2 cm below the xiphoid process and extending through the skin and peritoneum. The caecum was carefully exposed and flattened, and 25 μl of a 1 × 10⁶ CT26 cell suspension in PBS was slowly injected into the caecum at a shallow depth. Successful injections were confirmed by visible swelling on the caecum surface. After injection, the caecum was returned to the peritoneal cavity, and the peritoneum and skin were sutured. Following surgery, the mice were allowed to recover in a clean cage placed on a warming pad. Once fully alert and active, the animals were returned to their original cages. Imaging experiments were typically performed 10 days after surgical implantation.

Imaging study of fluorophore biodistribution in vivo

To evaluate the biodistribution of F127-formulated fluorophores following systemic administration, one mouse bearing an orthotopic colorectal tumour was intravenously injected with EP772/F127 (500 µM, 200 µl). Imaging was performed to monitor vascular circulation. At 12 h, 48 h and 72 h post-injection, the mouse was anaesthetized for laparotomy imaging to record fluorescence signal variations in the tumour. For the biodistribution of BSA-formulated fluorophores following systemic administration, two female BALB/c mice were used to minimize the potential harm associated with repeated laparotomies. Both mice were intravenously injected with EP772/BSA (500 µM, 200 µl) simultaneously. Imaging was conducted at specific time points: one mouse was imaged at 1 h and 12 h post-injection, while the other was imaged at 6 h, 48 h and 72 h. Laparotomy was performed to expose the mesenteric lymph nodes before imaging. To assess the gastrointestinal metabolism of F127-formulated fluorophores, a mouse was intragastrically administered EP772/F127 (500 µM, 60 µl). At time points of 0.5 h, 1 h, 1.5 h, 2 h and 2.5 h post-administration, the mouse was anaesthetized, and non-invasive imaging was performed after hair removal. All images were acquired using the general wide-field imaging configuration with a 1,500 LP filter and 760 nm excitation.

Dynamic excitation multispectral imaging in vivo

Seventy-three hours before imaging, EP772/BSA (500 µM, 200 µl) complexes were intravenously injected into the tail vein of mice bearing orthotopic colorectal tumours to label the mesenteric lymph nodes. One hour later, EP725/F127 (500 µM, 200 µl) micelles were intravenously administered through the tail vein to label the orthotopic colorectal tumour. At 2.5 h and 0.5 h before imaging, EP737/F127 (500 µM, 60 µl) and EP679/F127 (500 µM, 60 µl) micelles were administered intragastrically, respectively. EP699/F127 micelles were injected intravenously through the tail vein immediately before imaging. The mice were anaesthetized with abdominal fur removed, and underwent laparotomy to expose the intestinal tract for imaging. Time-lapse multispectral datasets (x, y, λ, t) were collected using the general wide-field imaging configuration, equipped with continuous-wave lasers for dynamic compressive sampling. For comparison, mice labelled with single kind of fluorophore were imaged using the same experimental setting.

Phasor analysis

Spectra/phasor transformation was performed using HySP-0.9.18 software developed by F. Cutrale. For each pixel in a data cube, the Fourier coefficients of its normalized spectra define the coordinates of its phasor point (z(n)):

where λ_s and λ_f are starting and ending wavelengths, respectively; I is the intensity; ω = 2π/τs with τ_s = number of spectral channels; and n is the harmonic (n = 1 or n = 2).

Linear unmixing

Linear unmixing was used to decompose multispectral data into individual fluorophore contributions. The observed spectrum at each pixel in a multispectral data cube and the reference spectra of the fluorophores, extracted from the same data cube, were modelled as a system of linear equations:

Here Y represents the observed pixel spectrum, A is the matrix of fluorophore spectra, and m is the noise present in the observed spectrum. The vector X corresponds to the relative abundances of the fluorophores at each pixel. Given Y and A, the solution for X is obtained using matrix inversion or regularization techniques to account for potential overfitting and noise amplification:

This method assumes that the spectra in A are linearly independent and accurately represent the fluorophores present in the sample. To validate the unmixing results, a visual example is provided in Supplementary Fig. 11.

Quantitative metrics

In the capillary experiment, the contrast of each unmixed channel was calculated and normalized using the following formula:

where σ is the standard deviation in signal intensities of all pixels in the image and μ is the mean signal intensity of all pixels in the image.

Bleed-through quantifies the extent to which signal from non-target channels leaks into the target channel. It was calculated using the following formula:

where n is the number of non-target channels, I(non-target) is the intensity of the non-target channel observed in the target channel and I(target) is the intensity of the target channel.

SSIM and RMSE were used to evaluate the fidelity of multispectral data cube and excitation spectra under the interference of phantom tissue, respectively. These metrics compare data cubes/spectra acquired at varying depths of phantom tissue with a reference data cube/spectrum obtained in the absence of phantom tissue.

SSIM was calculated using the following formula:

where μ_x, μ_y, σ_x and σ_y are the mean and standard deviation of images x and y. σ_xy is the cross covariance between images x and y. , , and L is the dynamic range of images x and y (L = 1 here). All images are normalized before calculation. The range of the SSIM is [0, 1], and a high SSIM denotes better structural similarity, that is, high fidelity.

RMSE was calculated using the following formula:

where k is the number of excitation wavelengths, y_i is the intensity of the ith excitation wavelength in the absence of phantom tissue and  is the intensity of the ith excitation wavelength acquired at varying depths of phantom tissue. All spectra are normalized before calculation.

EndmemberNet training

To train EndmemberNet, we collected 32 multispectral datasets from different mice. These datasets were divided into training, validation and test sets, comprising 19, 3 and 10 groups, respectively. The final model was selected based on its best performance on the validation set. The training process utilized a two-stage neural network framework:

1. a.

   Detection network: We adopted YOLOv5 as the backbone for the detection network. YOLOv5 consists of a CSPDarknet53-based backbone for feature extraction, a feature pyramid network and path aggregation network for multi-scale feature fusion, and a detection head for predicting bounding boxes, confidence scores and class probabilities.

2. b.

   Segmentation network: For the segmentation stage, we used DeepLabv3+ with a ResNet-50 backbone. The network incorporates Atrous Spatial Pyramid Pooling to capture multi-scale information and combines low-level and high-level features to produce accurate segmentation masks.

To enhance data diversity and ensure wavelength awareness, we applied a weighted synthesis method for data augmentation. This method merges five spectral images captured at different excitation wavelengths into a single weighted image, with random weights assigned to each wavelength. Model performance was evaluated using the Intersection over Union (IoU) metric for both detection and segmentation tasks. On the test set, the detection and segmentation IoU scores were 0.46 and 0.42, respectively. Despite the challenges posed by unclear boundaries and distributed spectral signature regions, an IoU ≥0.4 demonstrated effective coverage of target areas, ensuring high recall and minimizing the risk of missing spectral signatures. The runtime for processing a single sample averaged 6.7 ms for detection and 7.6 ms for segmentation. Additional details, including mathematical formulations, network architecture diagrams and training hyperparameters, are provided in Supplementary Figs. 7–11 and Supplementary Note 5.

Real-time online processing

To achieve real-time processing during imaging acquisition, we developed a programme interface to integrate EndmemberNet into our image acquisition software. The software, implemented in Python, supports unmixing using either a GPU or CPU. Leveraging GPU acceleration (for example, GeForce RTX 3060), the software provides real-time feedback for unmixing, enabling researchers to obtain results during experiments. The software manages the following workflow: image acquisition (controls the LightField programme to capture excitation-synchronized frames by sequentially triggering five individual lasers); data packaging (packages frames into 3D (x, y, λ) batches and saves them in a user-defined directory); data processing (inputs the data batch into EndmemberNet to identify spectral signatures of endmembers and performs linear unmixing); visualization and storage (displays the unmixing results in real time and simultaneously saves them for further analysis).

Reporting summary

Further information on research design is available in the Nature Portfolio Reporting Summary linked to this article.

Data availability

All training and testing data involved in the experiments can be downloaded from the Zenodo links at https://doi.org/10.5281/zenodo.13622692 or from the Huggingface links at https://huggingface.co/datasets/Orange066/Unmixing_TrainValTestData. The experimental data supporting the findings of this study are available within the article and Supplementary Information. Source data are provided with this paper.

Code availability

The PyTorch code of our EndmemberNet, together with trained models, as well as some example images for inference are publicly available at https://github.com/Orange066/EndmemberNet; you can also download it from the Zenodo links at https://doi.org/10.5281/zenodo.13622929 or from the Huggingface links at https://huggingface.co/Orange066/Unmixing_Model. Furthermore, we also provide an offline demo for EndmemberNet at http://fdudml.cn:6789. This newly built interactive software platform facilitates users to freely and easily use our trained model. We shared all models on Zenodo at https://doi.org/10.5281/zenodo.13622929 and Huggingface at https://huggingface.co/Orange066/Unmixing_Model. Finally, we have shared our software on Zenodo at https://doi.org/10.5281/zenodo.13622929 and Huggingface at https://huggingface.co/Orange066/Unmixing_Model. The software provides tools for real-time fluorescence image acquisition and unmixing, offering an efficient solution for researchers in related fields. In addition, it includes examples of unmixing as demonstrated in this paper. We used the Pycharm software for code development.