【AIGC One Sentence Reading】:Our study introduces gCFPM, a fast, high-quality colorization method for Fourier ptychographic microscopy, enhancing pathological imaging efficiency.
【AIGC Short Abstract】:This study introduces gCFPM, a method using a dual-color-space constrained model for fast, high-quality full-color imaging in Fourier ptychographic microscopy. It achieves superior colorization and sharpness with extended depth of field, outperforming existing methods and enhancing high-throughput digital pathology.
Note: This section is automatically generated by AI . The website and platform operators shall not be liable for any commercial or legal consequences arising from your use of AI generated content on this website. Please be aware of this.
Abstract
Full-color imaging is essential in digital pathology for accurate tissue analysis. Utilizing advanced optical modulation and phase retrieval algorithms, Fourier ptychographic microscopy (FPM) offers a powerful solution for high-throughput digital pathology, combining high resolution, large field of view, and extended depth of field (DOF). However, the full-color capabilities of FPM are hindered by coherent color artifacts and reduced computational efficiency, which significantly limits its practical applications. Color-transfer-based FPM (CFPM) has emerged as a potential solution, theoretically reducing both acquisition and reconstruction threefold time. Yet, existing methods fall short of achieving the desired reconstruction speed and colorization quality. In this study, we report a generalized dual-color-space constrained model for FPM colorization. This model provides a mathematical framework for model-based FPM colorization, enabling a closed-form solution without the need for redundant iterative calculations. Our approach, termed generalized CFPM (gCFPM), achieves colorization within seconds for megapixel-scale images, delivering superior colorization quality in terms of both colorfulness and sharpness, along with an extended DOF. Both simulations and experiments demonstrate that gCFPM surpasses state-of-the-art methods across all evaluated criteria. Our work offers a robust and comprehensive workflow for high-throughput full-color pathological imaging using FPM platforms, laying a solid foundation for future advancements in methodology and engineering.
Using an optical microscope to analyze pathology slides is a benchmark routine in clinical disease diagnosis and nonclinical medical examinations. With the rapid development of computational methods, digital imaging solutions, such as whole slide imaging (WSI), have become increasingly mainstream in pathology.1–3 However, conventional WSI systems employing objectives with a high numerical aperture (NA) suffer from several issues, such as a limited field of view (FOV) and depth of field (DOF).4,5 Solutions involving mechanical lateral scanning and -axis focusing require highly precise hardware and can potentially distort pathology images. Therefore, alternative solutions for achieving high-resolution (HR) and high-throughput WSI are greatly desired.
Thanks to advancements in optical modulation and postdigital processing techniques, optical computational imaging offers solutions for biomedical applications with simpler hardware and more robust imaging results. Adapted from microwave synthetic aperture techniques, Fourier ptychographic microscopy (FPM) is a promising candidate for high-throughput pathology imaging.6–8 In a typical implementation, FPM utilizes a programmable LED array for illumination modulation and a phase-retrieval-based aperture synthesis algorithm for superresolution. This optimization framework allows FPM to correct optical aberrations without external setups.9,10 Consequently, FPM can achieve large FOV, HR, and aberration-free imaging using a low NA objective. FPM has been proven feasible in various fields, such as label-free quantitative pathology analysis,11,12 high-throughput cytometric analysis,13,14 and three-dimensional tissue imaging.15,16
One of the main obstacles to the large-scale application of FPM is the forward model mismatch caused by inaccurate system parameter estimation and the vignetting effect,17,18 which prevent the FPM reconstruction algorithm from outputting full-field, high-quality images. To address this issue, Zhang et al.19 proposed a feature domain backpropagation method (FD-FPM) to bypass the model mismatch, demonstrating nonblocking, full-field FPM reconstruction. Another significant challenge is the high computational demand for FPM reconstruction. In regular pathology, at least three-channel imaging (RGB) is needed for tissue differentiation, and more than three wavelength channels are required when combining spectral imaging techniques. This requirement substantially increases the time needed for FPM data acquisition and reconstruction, compromising FPM’s high-throughput advantage. A straightforward solution involves wavelength multiplexing methods, such as RGB simultaneous illumination with a chromatic camera20,21 or RGB multiplexed illumination with a monochrome camera.22,23 However, the former suffers from degraded raw data due to the Bayer filter, and the latter requires a mixed-state decoupling algorithm, which does not reduce reconstruction time. Overall, wavelength multiplexing-based methods cannot adequately address the problem of high computational demand.
Sign up for Advanced Photonics Nexus TOC. Get the latest issue of Advanced Photonics Nexus delivered right to you!Sign up now
Instead of reconstructing HR images for each channel and then generating an RGB HR image, color-transfer-based FPM (CFPM) assumes that the low-resolution (LR) color image contains sufficient color information.24–26 By transferring this color information to the HR single-channel image, CFPM can generate high-quality HR color images, thereby easing the computational burden. We classify existing CFPM methods as nonmodel-based and model-based, depending on whether they consider the statistical or spatial properties of the color space (such as RGB space) in the LR color image and HR image. As an example of a nonmodel-based method, wavelet-based FPM image colorization (wavCFPM) fuses the wavelet coefficients of all three channels of the LR RGB image with the green-channel HR reconstructed image.24 However, distortion can occur when the red or blue channel LR image is fused into the mismatched green channel HR image. Another nonmodel-based method conducts similar fusion in the image’s frequency domain (SFCFPM).27 In addition, one can utilize a neural network for FPM colorization,28 but this requires a large-sized labeled training set and suffers from a lack of generality and physical interpretability.29–31 In contrast, model-based CFPM colorizes the HR image according to statistical or spatial properties in the color domain. For example, Gao et al.25 proposed colorizing the HR gray image by matching its RGB histogram to that of the LR RGB image. Although this method involves color statistical properties, it still suffers from decreased color contrast due to the lack of spatial color information. Chen et al.26 proposed a spatially filtered method (CFFPM) to iteratively predict the HR color image using color-space alternating projection (AP). Inspired by the Gerchberg–Saxton (GS) algorithm in phase retrieval,32 color-space AP enables the predicted image to obtain the identical color distribution of the LR color image in Lab color space while maintaining its green channel consistency with the HR image. However, CFFPM did not reveal its underlying mathematical principles, which likely share roots with GS in optimization theory, thereby hindering its further improvement and perfection. For example, one could expect that model-based CFPM could be solved in a noniterative way with high precision.
In this work, we report a fast and high-quality colorization method for full-color pathological imaging using FPM through a generalized dual-color-space constrained model. This model addresses the colorization problem from an optimization standpoint. Rather than iterative calculation, we demonstrate that the model has a closed-form solution when linear color transforms are employed, enabling efficient colorization. Additionally, we introduce several preprocessing strategies to further improve the performance of colorization. We refer to the complete framework as generalized CFPM (gCFPM). Both simulations and experiments demonstrate that the proposed method outperforms state-of-the-art techniques in various ways, including time consumption and colorization quality. Moreover, gCFPM offers an extended DOF compared to the traditional methods. We believe that our work will significantly advance the application of FPM in high-throughput pathological imaging.
2 Results
2.1 Principle of gCFPM
In the regular workflow of FPM, hundreds of raw LR images are used to reconstruct an HR complex transmission function of the object using iterative algorithms. To expedite the retrieval of an HR color image, CFPM utilizes an LR color image to colorize the retrieved HR gray image. Without loss of generality, we denoted this HR gray image as (, or ). Specifically, model-based CFPM aims to predict HR color image matches the color distribution of while maintaining consistent spatial resolution information with , as illustrated in Fig. 1(a). To extract the color distribution of an image defined in regular RGB space, it must be transformed into another color space where spatial resolution information and color distribution are decoupled. We denoted the extracted LR color distribution as , where is the arbitrary color transformation. With two known observations in different color spaces— in the RGB space and in the transformed color space—these serve as constraints for predicting . This leads to the formulation of a generalized dual-color-space constrained model as follows:
Figure 1.Principle of gCFPM. (a) Schematic of generalized dual-color-space-constrained model for color transfer; (b) principle of RGB-to-IHS transform; and (c) flow chart of gCFPM using IHS-RGB space constraints.
It is important to note that upsampling of is excluded in Eq. (1) for simplicity but should not be omitted when implemented in practice. Color-space AP can be regarded as a solution to Eq. (1) by iteratively projecting onto two sets defined by and . By measuring difference of prediction and known observations using Euclidean distance, Eq. (1) can be transformed into a nonconstrained optimization problem, where means Euclidean norm and is the operator extracting channel from an RGB image. To solve Eq. (2), we introduce intensity–hue–saturation (IHS) color space,33,34 which has been widely used in pan-sharpening tasks within the field of remote sensing.35–38 As shown in Fig. 1(b), the IHS space is connected with RGB space by a linear coordinate transformation. This transformation rotates the RGB color cube so that the new intensity axis aligns with the gray axis, while its perpendicular plane spans the intensity-independent color space. When represented in a polar coordinate system, the hue and saturation correspond to the angular and radial coordinates, respectively. However, Eq. (2) minimizes Euclidean distance in color space, so we still present it in a Cartesian coordinate system with two orthogonal bases, and . Using this linear IHS transform, cost function in Eq. (2) becomes quadratic, allowing it to be solved in closed form.
In addition to the colorization model described above, we have proposed several strategies to enhance the performance of gCFPM, as outlined in the flow chart in Fig. 1(c). The first step is to record the LR color image. Rather than using the center LED for illumination, which leads to coherent color artifacts,9,28,39 we illuminate all the LEDs in the bright-field region. This incoherent illumination eliminates artifacts and produces color images that are more consistent with human visual perception. The second step is to decide in which channel the raw data of FPM should be recorded. We use image entropy (IE) to measure the information content40 in each channel of ; the one with the highest IE is selected for FPM data recording and HR gray image reconstruction. The third step involves applying FPM algorithms to recover , the system’s coherent transfer function , and the optical transfer function . Assuming a system with low chromatic aberration, the color map is refined by deconvolution with the recovered . This constitutes the fourth step. Finally, the HR color image is predicted using the generalized dual-color-space constrained model.
2.2 Full Color Imaging via gCFPM
Experiments were conducted using a self-built FPM platform and a personal computer (PC). The FPM platform is equipped with a high-brightness programmable RGB LED array, with center wavelengths at 631.23, 538.86, and 456.70 nm, respectively. It also features a 4× objective (Olympus OPLN4× NA 0.10) and a CCD camera sensor (ZWO ASI178MM, pixel pitch, ). The synthesized NA of the FPM platform is approximately 0.63. The PC is equipped with an Intel Core i5-12400F CPU and Nvidia GeForce RTX3060 Ti graphics card. Bright-field LEDs are covered by a diffuser to ensure uniform incoherent illumination when recording LR color images. The white balance of LR color images is corrected using a dynamic threshold algorithm.41 FPM raw data are downsampled by a factor of 2 to reduce the computational burden. Then the central region of is cropped for reconstruction using FD-FPM, resulting in an HR image with a resolution of .
Figure 2(a) shows the full-color imaging result of a cat stomach smooth muscle section, whose photograph is shown in Fig. 2(b). FPM HR image reconstruction is performed in the green channel. For comparison, the LR color image and the HR image obtained by directly fusing the three channels recovered by FPM (3-FPM) are also included. Enlarged regions of interest (ROIs) are displayed in Fig. 2(c). The results demonstrate that gCFPM achieves significant resolution improvement while preserving rich color information. Although the color distribution is derived from the LR color map, it does not compromise spatial resolution, which is comparable to that of 3-FPM. However, the HR color image obtained through 3-FPM shows inconsistent background color tones across the FOV, as evidenced by ROIs 1 and 2. Additionally, the zoomed-in area in ROI 3 reveals that the 3-FPM result is significantly affected by dispersion-like artifacts around the edges, which nearly obscure the details. In contrast, the gCFPM result maintains clarity, allowing for easy recognition of details. Figure 2(d) illustrates the total time consumption of gCFPM and 3-FPM. In our practice, single-channel data recording and image reconstruction of resolution on a GPU typically take around 2 and 7 min, respectively, which triples when the R, G, and B channels are processed separately in 3-FPM. While gCFPM requires only single-channel reconstruction and additional colorization, taking , this results in a time savings of approximately two-thirds compared to traditional methods. To further validate our method, we inspect two additional sections, as shown in Fig. 3. The reconstruction channel of FPM HR image for Figs. 3(c) and 3(d) is red and green, respectively. These two samples also demonstrate the excellent imaging performance of our method.
Figure 2.Full-color imaging via gCFPM. (a) Imaging result of the sample; (b) photograph of cat stomach smooth muscle section; (c) enlarged view of ROI; and (d) full process time comparison of 3-FPM (three-channel FPM) and gCFPM.
Figure 3.(a) Photograph of stratified epithelium section; (b) photograph of lycopodium sporophyll spike longitudinal section; (c) imaging result of (a); and (d) imaging result of (b).
This section introduces the detailed solution of Eq. (2) with IHS transform. One can easily extend the following formula to other linear color transforms. The RGB-to-IHS transform is given as
is the intensity, and and are the two orthogonal bases in the color plane. For simplicity, we denote the and related row vector in the transform matrix as and . That is, , . Then the cost function in Eq. (2) is rephrased as
To minimize Eq. (4), the corresponding derivative is set to zero. This leads to solving the following linear equation:
The coefficient matrix proves to be full rank and inversible for all three channels of RGB. Therefore, is given by directly solving Eq. (5). This linear solution to Eq. (5) indicates that each channel of can be expressed as a linear combination of , , and . For example, taking , calculation of is detailed as follows:
The above derivation is based on the linear color transform. However, nonlinear transform is more common in practical applications because it realizes better decoupling of images’ color and resolution information.42–45 To solve a dual-color-space-constrained model based on a nonlinear transform, AP on two color spaces is the most straightforward method. Another potential solution is gradient descent-based optimization,46–48 which requires the nonlinear color transform to be continuously differentiable or have an existing subgradient. Both methods are iterative and do not guarantee convergence. In contrast, the linear transform-based solution is closed form and unique.
3.2 Solving Deconvolution Model in Transformed Color Space
Besides the object’s HR image, FPM also recovers the coherent transfer function , which characterizes the optical aberration of the system, and the incoherent optical transfer function is then derived as the autocorrelation of .49,50 This enables correction and contrast enhancement of color distribution in recorded observations. Assuming the system’s chromatic aberration is minor, all three channels of are degraded by . This can be equally modeled in a linear-transformed color space, resulting in the following equation as , where represents the ideal distribution of , and denote the Fourier transform and its inversion, signifies the element-wise product, and , 2. We introduce a Tikhonov-regularized deconvolution model as51where is the regularization weight and means the difference operator. The second term in Eq. (7) restricts the edge energy of to prevent color artifacts. The Fourier domain solution to Eq. (7) is given as where is the transfer function of .
4 Discussion
4.1 Comparison with Other Methods
To validate the performance of gCFPM, we compare it with wavCFPM, SFCFPM, and CFFPM. Algorithm parameters are set according to the suggestion in the corresponding paper. All the methods use an incoherent LR image as a color donor for the sake of fairness, although a coherent image with significant color artifacts is adopted in their original versions.
First, a comparison was conducted using simulation data, with system parameters consistent with those used in the experiment. The ground truth of the color target and the result using different methods are shown in Fig. 4(a). Quantitative comparison using root-mean-squared error (RMSE) and color extension of structure similarity (CSSIM)52 is displayed in Fig. 4(b). The zoomed-in region of the gCFPM result displays rich color with minimal structural and color artifacts. Both RMSE and CSSIM evaluation demonstrate that the result of gCFPM is closest to the ground truth.
Figure 4.Comparison of different colorization methods. (a) Simulation result; (b) RMSE and CSSIM comparison in (a); and (c) comparison using experimental data. The upper row in (c) shows the result of ROI 3 in the stratified epithelium section; the lower row in (c) shows the result of ROI 1 in lycopodium sporophyll spike L.S.
Next, a comparison was made using experimental data. Results of ROI 3 of a stratified epithelium section and ROI 1 of lycopodium sporophyll spike L.S. are illustrated, as displayed in Fig. 4(c). We use an incoherent HR color image obtained by 20× objective with 0.4 NA as a reduced reference. In ROI 3 of the stratified epithelium section, where color tones are monotonous, gCFPM still delivers satisfying visual quality, whereas wavCFPM exhibits noticeable resolution loss and CFFPM shows reduced color contrast. In ROI 1 of the lycopodium sporophyll spike L.S., FSCFPM, and CFFPM display significant color loss around detailed structures, failing to distinguish adjacent pink and blue areas. Moreover, CFFPM is disrupted by mosaic-like artifacts resulting from feature-matching-based color transfer, which may fail in regions with minor structures or gradual changes. In contrast, gCFPM maintains HR without losing color details, offering quality comparable to that of the reference image obtained with the high NA objective. Further, we evaluated each method on the entire FOV image of using quantitative indices, including colorization time on CPU, color image sharpness,53 and colorfulness.54 The assessment values are provided in Table 1. For both of the inspected samples, the sharpness value of wavCFPM is noticeably lower than that of the others, while the abnormally high sharpness value of CFFPM is partly due to mosaic-like artifacts. gCFPM demonstrates competitive sharpness compared to other methods while achieving the highest colorfulness value and requiring the least time for processing. Both qualitative and quantitative comparisons demonstrate the superiority of the proposed method over others.
Time (s)
Stratified epithelium section
Lycopodium sporophyll spike L.S.
Sharpness
Colorfulness
Sharpness
Colorfulness
20 × 0.4NA
—
17.5961
0.6733
24.2565
0.6535
wavFPM
22.9502
14.1853
0.6959
20.0768
0.5937
FSCFPM
3.7196
19.9778
0.7301
29.4396
0.5546
CFFPM
1622.7449
25.8384
0.6401
30.7840
0.5964
gCFPM
3.6868
20.3490
0.8220
28.5747
0.6435
Table 1. Quantitative comparison of different methods in experiments.
Due to the limited DOF of high-NA objectives, traditional systems require a high-precision -axis controller and supporting algorithms to suppress focus drifting. In contrast, FPM-based systems typically have a large primary DOF due to the use of low-NA objectives, which is further extended by robust aberration correction algorithms. This section demonstrates large DOF full-color imaging using gCFPM. We defocused the sample from to in steps of , with the imaging results of gCFPM shown in Fig. 5(a). For comparison, the result using a 20× objective is also provided. The gCFPM method presents clear structures and abundant color information within a DOF of . The color distribution at is slightly disrupted by enhanced chromatic aberration in the large defocus region. In contrast, the 20× objective only produces a clear image in the in-focus position, reflecting its extremely small DOF. To quantitatively assess the extended DOF of gCFPM, the entire FOV was segmented into 100 blocks, and the focus quality55 and sharpness indices were applied, as shown in Figs. 5(b) and 5(c). The solid line represents the mean value across all blocks, and error bars represent the indices’ variation across these blocks. The opposite behavior of gCFPM’s error bar indicates that blocks across the whole FOV are all precisely in focus, while those with sparse or dense image context are well distinguished. The quantitative assessment is consistent with the qualitative analysis, which confirms the effectiveness of gCFPM in maintaining image quality across an extended DOF.
Figure 5.(a) Imaging results of gCFPM and 20× objective with defocus ranges from to ; (b), (c) focus quality and sharpness assessment on two full-color imaging methods.
In addition to the IHS transform, the gCFPM model can be extended to other linear color transforms. We present an additional result on the simulation image in Table 2, using YUV transform with three different standards (BT.601,56 BT.709,57 and BT.202058) and pseudo-KL transform (PKLT).35,59 The results show that the quantitative assessment of PKLT is slightly better than that of the other four transforms. However, this improvement is too minor to be noticeable by visual perception.
RMSE
1-CSSIM
IHS
0.0928
0.1355
YUV (BT.601)
0.0926
0.1354
YUV (BT.709)
0.0927
0.1354
YUV (BT.2020)
0.0927
0.1354
PKLT
0.0891
0.1325
Table 2. Assessment of gCFPM based on the extended linear transform.
Having broken the trade-off between resolution and FOV, FPM paves the way for the next generation of low-cost, high-precision digital pathology imaging. However, the high computational demands for FPM have hindered its broader application, particularly in high-throughput settings where true-color imaging is essential. CFPM offers a partial solution by transferring color from an LR image to the HR reconstructed gray-scale image. However, previous approaches have not fully harnessed the potential of color transfer, leaving room for improvement in both computational efficiency and colorization quality.
In this paper, we proposed a fast full-color imaging method for Fourier ptychography microscopy called gCFPM, based on a generalized dual-color-space-constrained model. This model addresses the color transfer problem in FPM from an optimization standpoint, enabling noniterative and precise colorization using only an LR color image and a single-channel FPM reconstruction. This significantly improves the efficiency of high-throughput FPM imaging platforms. Additionally, we introduced several preprocessing strategies to further enhance the colorization quality of the proposed method. Both simulation and experimental results demonstrate that the proposed method outperforms state-of-the-art techniques in terms of colorization quality and computation time. Furthermore, our method supports large DOF full-color imaging without requiring high-precision hardware. These findings indicate that gCFPM fully utilizes both color and resolution information from limited raw data, providing results comparable to traditional optical microscopy.
In the current framework of CFPM, details beyond the objective’s bandwidth are completely lost, and even gCFPM exhibits some inevitable color distortion in these areas. To overcome these inherent limitations, future improvements may come from enhancing the raw data. With advanced modulation and demodulation techniques, CFPM has the potential to break the trade-off between imaging efficiency and color accuracy, just as it has addressed the trade-off between resolution and FOV. Nevertheless, we are confident that the proposed method has sufficient capability for most applications and will serve as a solid foundation for future developments.
Yanqi Chen is an MS degree student in optics at Xi’an Institute of Optics and Precision Mechanics (XIOPM), Chinese Academy of Sciences (CAS), China. He received his bachelor’s degree in Optical Information Science and Technology from Nanjing Normal University, China, in 2019. His current research focuses on Fourier ptychographic microscopy and coherent diffraction imaging.
Jiurun Chen is a PhD student in Electronic Information Engineering at Tsinghua Shenzhen International Graduate School, China. He received his bachelor’s degree in Mechanical Engineering from Central South University, China, in 2020, and his master’s degree in Electronic Information Engineering from the Xi’an Institute of Optics and XIOPM CAS, China, in 2023. His current research focuses on biomedical engineering and optical imaging.
Zhiping Wang is an MS degree student in physics of life at University of Basel, Switzerland. He received his bachelor’s degree in physics from Lanzhou University, China, in 2024. His current research focuses on computational imaging and structural biology.
Yuting Gao is a PhD student in Electronic Engineering at XIOPM, CAS, China. She received her bachelor’s degree in Electronic Engineering from Xi’an University of Architecture and Technology, China, in 2019. Her current research focuses on Fourier ptychographic microscopy.
Yonghong He is a professor and PhD supervisor in biomedical engineering and optics at Tsinghua Shenzhen International Graduate School, China. He received his PhD in optics from South China Normal University in 2002 and completed a postdoc at Cranfield University, UK. His current research focuses on biomedical optics, OCT imaging, and high-throughput biochip analysis.
Yishi Shi is a professor and PhD supervisor in opto-electronic engineering at University of Chinese Academy of Sciences, China. He received his PhD in optics from University of Chinese Academy of Sciences in 2008 and completed a postdoc in opto-electronic engineering in 2010. His current research focuses on opto-electronic information, artificial intelligence, and optical imaging.
An Pan is an associate professor and a principal investigator at XIOPM, CAS, China, and the head of the Pioneering Interdiscipline Center of the State Key Laboratory of Transient Optics and Photonics. He received his BE degree in electronic science and technology from Nanjing University of Science and Technology (NJUST), China, in 2014, and he obtained his PhD in optical engineering at XIOPM, CAS, China, in 2020. He was a visiting graduate at Bar-Ilan University, Israel, in 2016 and California Institute of Technology (Caltech), USA, from 2018 to 2019, respectively. He focuses on the computational optical imaging and biophotonics area and is among the first to work on Fourier ptychography. He was selected as the 2024 Optica Ambassador and is the winner of the 2021 Forbes China 30 Under 30 List, 2021 Excellent Doctoral Dissertation of CAS, 2020 Special President Award of CAS, 2019 OSA Boris P. Stoicheff Memorial Scholarship, the 1st Place Poster Award of the 69th Lindau Nobel Laureate Meetings in Germany (Lindau Scholar), and 2017 SPIE Optics and Photonics Education Scholarship. He has published 40 peer-reviewed journal papers and is a referee for more than 40 peer-reviewed journals. He is an early career member of Optica and SPIE.
[34] W. Carper et al. The use of intensity-hue-saturation transformations for merging spot panchromatic and multispectral image data. Photogramm. Eng. Remote Sens., 56, 459-467(1990).