Fig. 1. Applications of cross-modality transformations across biological scales. At the largest scales, virtual staining is used to enhance imaging contrast. At intermediate scales, virtual staining is used in conjunction with noise reduction techniques. At the smallest scales, superresolution is used to study systems far beyond the optical diffraction limit. Image created with the assistance of BioRender.

Fig. 2. Contrast between physical and virtual approaches to obtain a stained image. In the physical approach, the sample undergoes a series of complex procedures, including preparation, staining, and imaging. Tissue preparation may involve fixing, embedding, and sectioning, among other steps. Similarly, histological staining of an unstained sample requires permeabilization, chemical dye application, washing, counterstaining, and protocol optimization before imaging. In contrast, virtual staining offers a simplified alternative to these protocols, eliminating the need for physical processing³⁷ or staining of the sample.³⁸ In the virtual approach, an unaltered or unstained sample is processed through a virtual staining network to generate a stained image, with results equivalent to physical staining. Physically stained images serve as training data, or input, for the model, especially when transforming between different stains is the objective. Created with the assistance of BioRender. (Tissue image adapted from Berkshire Community College Bioscience Image Library.)

Fig. 3. Representative applications of cross-modality transformations for tissue imaging using DL. (a) Virtual staining of an unlabeled sample image to obtain the equivalent H&E stained image. Adapted from Rana et al.³⁹ (b) Stain-to-stain translation where the input and output are images from two different staining procedures, in this case H&E to IHC staining for cytokeratin (CK). Adapted from Hong et al.⁴⁸ (c) Multi-stain model that is able to transform unlabeled tissue images into different staining options simultaneously: H&E, orcein, and PSR. Adapted from Li et al.⁴⁰ (d) Cross-modality transform to apply a segmentation method, or potentially a stain, in a previously incompatible modality. In this case, an AI segmentation for MRI images is transcribed to CT images. Adapted from Dou et al.⁴⁹ (e) Biopsy-free cross modality transformation, where not only the staining procedure but also the sample preparation is avoided. Using CRM as a noninvasive technique for in vivo measurements, the resulting images incorporate features comparable to H&E, despite being incompatible with traditional staining techniques in such conditions. Adapted from Li et al.³⁷

Fig. 4. Virtual cell staining using DL. (a) Helgadottir et al. introduced a cGAN to virtually stain lipid droplets, cytoplasm, and nuclei using bright-field images of human stem-cell-derived fat cells (adipocytes). The U-Net-based generator processes bright-field image stacks captured at various $z$ positions to generate virtually stained fluorescence images. A CNN-based discriminator is trained to differentiate between the virtually generated stains and real fluorescently stained samples, conditioned on the input bright-field image. (b) The virtual staining of lipid droplets (green channel) and cytoplasm (red channel) exhibits a high degree of fidelity, as evidenced in the fine details of the lipid droplet internal structure and the enhanced contrast among distinct cytoplasmic components (highlighted by the arrows). Panels (a) and (b) adapted from Helgadottir et al.⁵⁴ (c) Unsupervised cross-modality image transformation using UTOM. Two GANs, G and F, are trained concurrently to learn bidirectional mappings between image modalities. The model incorporates a cycle-consistency loss ($L_{\text{cycle}}$) to ensure the invertibility of transformations, while a saliency constraint ($L_{\mathrm{sc}}$) preserves key image features and content in the generated outputs. (d) UTOM achieves performance comparable to a supervised CNN trained on paired samples, without requiring paired training data. Panels (c) and (d) adapted from Li et al.⁵⁵ (e) Co-registered label-free reflectance and fluorescence images acquired using a multimodal LED array reflectance microscope. (f) Multiplexed prediction displaying DNA (blue), endosomes (red), the Golgi apparatus (yellow), and actin (green). Zoomed-in views, with white circles, highlight representative cell morphology across different phases of the cell cycle. Adapted from Cheng et al.¹⁰⁶

Fig. 5. Superresolution physical principles. (a) Illustration of the PSF resulting from imaging object of diameter $D_1$ below the diffraction limit of an optical system, leading to an image of diameter $D_2$. (b) Low-resolution image of simulated emitters alongside ground-truth emitter positions. Image reproduced with permission from Ref. 56.

Fig. 6. Superresolution applied architecture. The superresolution network enhances image resolution by training on pairs of simulated low-resolution (LR) and high-resolution ground-truth images or on wide-field (WF) and STORM images from a STORM microscope. First, the LR/WF image undergoes preprocessing through a subpixel edge detector to generate an edge map, both of which serve as inputs to the network. Training is guided by a multi-component loss function that incorporates the combination of multiscale structure similarity index measure and mean absolute error loss (MS-SSIM L1) to capture pixel-level accuracy between the superresolution (SR) and ground-truth/STORM images through multiscale similarity and mean absolute error, perceptual loss to assess feature map differences via the visual geometry group network, adversarial loss using a U-Net discriminator to differentiate ground-truth/STORM images from SR images, and frequency loss to compare differences in the frequency spectrum between SR and ground-truth/STORM images within a specific frequency range using the fast Fourier transform function. This comprehensive loss function helps the superresolution network model achieve precise and perceptually accurate superresolution imaging. Image adapted from Chen et al.¹⁴⁷

Fig. 7. Potential application perspectives of AI on biological samples imaging. Current developments found in the literature are contained in green boxes, while speculative prospects for the future are contained in yellow boxes. Starting from the top left, AI is extensively used in diagnostics such as virtual staining and other cross-modality transforms (image in the green panel adapted from Li et al.¹⁸⁷). (a) In the future, this could lead to in vivo analysis of virtual biopsies instead of performing tissue extraction and preparation. (b) Cross-modality transform could progressively transition the limits of the different scales outlined in this review. (c) As a consequence of broader data availability and in vivo imaging, AI models could also transform and predict sample evolution over time. (d) Further, cross-modality transformations could also become more accessible and be routinely integrated in the sampling process, obtaining simultaneously different modalities with one single measurement. In the bottom left panel, these ideas could be integrated in treatment prediction and design for biological samples, leading to personally tailored therapy. The potential of superresolution to better characterize molecules structures is nowadays used for protein-folding determination (image in the green panel adapted from Kumar et al.¹⁸⁸), but implementing other information input from different modalities could enhance the reach of our knowledge (centered, bottom panel). The application of AI extracting cellular information from 3D structures hosted in increasingly complex in vitro systems that better replicate the dynamic conditions of in vivo systems (bottom right panel), is highlighted. AI is already implemented in assisted-surgery settings and equipment (right green panel adapted from Zhang et al.¹⁸⁹), which could be greatly improved by including real-time contrast enhancement and segmentation from cross-modality transformations. Images in the yellow boxes were created with the assistance of Designer, using DALL·E 3 technology, and BioRender. They have demonstrative purposes and do not hold real scientific meaning beyond the visualization of the ideas expressed.

Table 1. Overview of the key parameters for common approaches of DL in microscopy across scales.