Optical microscopic imaging technologies, attracting growing interest in recent years, have extended human capabilities to explore unknown fields, especially biology and biomedical science^[1–4]. A pinhole conjugate spatially to the focus on the focal plane blocks out-of-focus signals and improves spatial resolution and contrast in conventional confocal microscopes. Nowadays, confocal microscopy has become an important tool for studying biological macromolecular spatial and temporal interactions. However, it requires high enough illumination intensity and forms an energy concentration pattern on the focused area, contributing to damage in biological samples and nonlinear effects of fluorescent groups.

Strong light exposure may potentially induce irreversible damage to biological samples via phototoxicity^[5–10] and photobleaching^[11,12]. Excessive light illumination could lead to subtle cellular structural alteration both in the direct irradiation area and in the scattering part of the fixed tissue. Phototoxicity could result in mitotic arrest, reactive oxygen species production, mitochondrial respiratory loss, genome damage, and other dysfunctions in live-cell microscopy. Besides, photo-bleaching of fluorophores caused by strong exposure also deeply troubles many situations. Meanwhile, low-flux imaging has had growing importance as biomedical research has focused on smaller and smaller scales, such as protein allosteric inhibition in pathology^[13,14] and metabolite-regulated functional pathways^[15,16], which may have weaker signals exceeding the limits of ordinary imaging methods and emphasize the recording of various dimensions of photons.

Advanced sensors are transforming signal detection to the low-light level^[17–26]. At the same time, various emerging imaging principles and computational algorithms enable extreme energy detection^[27–31]. Single photons, operating at the fundamental limit of classic electromagnetic waves, can broaden the realm of quantum technologies in quantum computing^[32–35], quantum simulation^[36–39], quantum communication^[40–42], and quantum imaging^[43–46] by virtue of the unique features of being robust to environmental noise, having low decoherence properties, and having low detectable energy levels. Low-flux imaging has more practical implications for microscopic biomedical imaging, but this imaging method is not specifically optimized for hardware-based applications of fluorescent confocal systems and special post-imaging computational reconstruction algorithms^[47–50].

Here, we propose a photonic timestamped confocal microscopy (PT-Confocal) scheme, in which we treat light as a stream of photons in a picture of quantum optics. We exploit the time-correlated single-photon counting module to record the photon temporal distribution information and maximum likelihood estimation, discrete wavelet transform, and deep-learning algorithms for reconstruction. The experimental results demonstrate that we can recover same-quality fluorescent pictures using the information of a minimum of 10 photons.

We design and build the experimental prototype based on the proposed PT-Confocal imaging scheme, as shown in Fig. 1(a). The whole system contains two sections: In the illumination path, a pulsed laser diode with a pulse width of Tp=8ns and a repetition period of Tr=2.4µs is used as the illumination source. A dual-axis galvanometer coupled with the scanning lens and tube lens guides the filtered excitation photons to raster scan 400×400 points on the fluorescent samples magnified 60 times, with the goal of angular uniformity along both axes. The laser-excited fluorescent photons enter the detection path. Reflected by the beam splitter and filtered by the band-pass filter, the fluorescent signal photons are focused on the pinhole by the achromatic lenses and collected to the single-photon avalanche diode (SPAD) detector. We calibrate the pinhole position and the fiber coupler to improve the maximal detection efficiency of the system up to η=10%. (see Sec. 4 for details)

All timing signals are controlled and programmed by the field-programmable gate array (FPGA) module, as shown in Fig. 1(b). The galvanometer conducts a two-dimensional (2D) raster scanning pattern, in which the Y-axis moves one step and the X-axis shifts 400 steps. The time-correlated single-photon counting module timestamps all returned fluorescent photons with the 16 ps timing resolution. The controller module generates a predefined “start” signal in every position and later steers the laser pulse. A recording module records the start signal, the timestamps of emission photons, and a defined “stop” signal. The measurement of the above pattern is repeated at every physical position for limited time to estimate the intensity of fluorophores. In order to reconstruct a timestamped image, the photons have to be attributed to their tagged pixels, which is done by storing the absolute time stream of the photons in addition to the interval time from the laser pulse. Line and frame marker signals from the confocal microscope scanner are also recorded to sort the time stream of photons into the different pixels. (see Sec. 4 for details)

We image the immuno-stained, fixed mouse kidney tissue in Figs. 2–4 and demonstrate the capacity of PT-Confocal in biomedical fluorescent imaging. We reconstruct an initial intensity for each pixel by maximum-likelihood estimation (Max-L). Assume the expected emission photon number of pixel (x,y) is P(x,y), and we can program the laser power so that the range of P(x,y) is between 0 to 1. Meanwhile, pulse interval (ΔI) between two signals follows the Poisson distribution: P(ΔI)=e−1/P(x,y)ΔI!P(x,y)ΔI,(1)where the maximum likelihood estimation of ΔI for this distribution is 1/P(x,y). We use the experimental values of ΔI to estimate P(x,y) and reconstruct the image. Poisson noise prevents these pixel-by-pixel estimates from providing a high-quality intensity image, as shown in Figs. 2(a)–2(c). Sparse photons bring a big variation pattern among neighbor pixels. We suppress the noise by exploiting spatial correlation of real-world scenes, in which neighboring pixels have strong distance and intensity correlations. Such correlations can be captured through the sparsity in the scene discrete wavelet transform (DWT) coefficients^[47]. The optimized results of a DWT-based regularization are shown in Figs. 2(d)–2(f). The regularization gives a higher structural similarity (SSIM) image than the raw data and little loss of detail. We rebuild the imaging results by decreasing the photon number from more than 1000 to a minimum of ten photons. Our proposed scheme using the average inter-photon time is enabled to optimize the image quality even using the first ten photons per pixel, which is beneficial to eliminate laser-caused damage.

Various computational algorithms could improve the sparse spatial characteristics of our outputs. The deep-learning-based denoising imaging schemes have brought about significant breakthroughs in a variety of imaging tasks^[51–54]. The training of networks can be viewed as extracting the high-dimensional features that bridge the noising and denoising image spaces. In conventional neural networks, the spatial domain channel attention mechanism uses structural differences in the spatial domain as the input, which would lose the hierarchical representations of high-frequency information. Instead, we employ the deep Fourier channel attention network to learn the characteristics of the power spectrum in the frequency domain^[54]. (see Sec. 4 for details)

For each noise level, a corresponding network model is trained. In each residual block, the spatial channel attention mechanism utilizes the average intensity of the feature maps, which means only the zero frequency is used. In contrast, the frequency channel attention mechanism enables the network to learn each feature map about all frequency components in the power spectrum, which is beneficial for reconstructing rich spatial information. The training procedure updates the weights of the network, and once it is completed, the model’s performance is evaluated using the test set. Our reconstructed results turn out to recover two-dimensional details [Figs. 2(g)–2(i)] about immuno-stained tissue, which demonstrates its capacity in a biomedical imaging application. The results indicate that deep learning enables the reconstruction quality of 10 photons to converge to that of a strongly exposed control.

To measure the image quality for the image x and y, we calculate the structural similarity (SSIM) index defined as SSIM(x,y)=(2μxμy+c1)(2σxy+c2)(μx2+μy2+c1)(σx2+σy2+c2),(2)where μx,μy is the average of x,y, σx2,σy2 is the variance of x,y, respectively, and σxy is the covariance of x and y. c1=(k1L)2, c2=(k2L)2 are two variables to stabilize the division with a weak denominator. L is the dynamic range of the pixel values. k1=0.01 and k2=0.03 by default. The comparison of the SSIM for the Max-L, DWT, and DL is shown in Fig. 2(j). Decreasing the intensity of light causes SSIM decay. In the condition of 10 photons per pixel, the DL algorithm gives a better SSIM [Fig. 2(j)] and a smoother spatial feature [Figs. 2(k)–2(m)], which implicates that the deep learning algorithm is more conducive to image optimization and reconstruction under extremely low-light conditions. With the increase of the number of photons, the results of the maximum likelihood estimation gradually increase, and the difference among Max-L, DWT, and deep learning methods is getting smaller. This is mainly due to the increase in the number of samples, which makes the variance of the estimation decrease, and the estimation of the parameters is more accurate and thus the results are better. When the number of photons reaches 30, the difference in the reconstructed image is minimal. The DWT method will lose some details which is mainly due to the threshold selection. The deep learning method focuses on improving the image resolution and enhancing the spatial correlation that will lead to a slight decrease in contrast, which is mainly related to the design of the DFCAN algorithm is to improve system resolution.

To validate the application of PT-Confocal in real-world scenarios, we then tested its imaging results in multiplexed fluorescence-labeled samples. Figure 3 illustrates that PT-Confocal has the capacity of multi-color fluorescence imaging. To demonstrate that these effects cannot be achieved simply by accumulating photon counts, we created a control with photon-counting information per pixel equal to or less than the total intensity of 10 photons. In comparison to a cumulative image with the same photon count [Fig. 3(a)], PT-Confocal [Figs. 3(b) and 3(c)] utilizes the same number of photon information to achieve a much higher quality.

Three-dimensional (3D) imaging in confocal microscopes is considered in terms of 3D transfer functions, representing the capacity to detect axial imaging properties. Figure 4(a) shows a 3D reconstruction of mouse glomerulus and renal tubules with 4.27µm thickness. By integrating the 3D panning system, it is possible to accomplish the complete capabilities of a commercial confocal system. In detail, Figs. 4(b)–4(g) exploit 2D sample images in the depth of 2.13 and 3.20µm. The deep-learning-based reconstructed images restore the 3D object even in the condition of a few photons. Furthermore, the result enables extended applications in vivo tissue imaging.

PT-Confocal microscopy is able to extract more valuable parameters than photon-counting schemes with the same sparse data. Although the current imaging speed of PT-Confocal is limited by the mechanical frequency of the oscillator and is not sufficient for live-cell imaging, this drawback can be reduced by incorporating a multi-pinhole or spin-disk pinhole array and integrating a SPAD-array camera^[55] to shorten the time required for mechanical actuation in the future. This enhancement will be meaningful in many areas where either radiation damage or properties of samples make the collection of enough photons impossible.

The decreased exposure enables us to obtain significantly more information with fewer photons than ever before, possibly solving a range of biological problems that were previously difficult to scan. PT-Confocal can be used to observe fragile samples with weak signals, such as expansion microscopy^[56–59], as an alternative of chemical or biological signal amplification methods^[60,61]. Expansion microscopy necessitates more powerful signal amplification or detection technologies. PT-Confocal has great potential in such ultralow-flux situations, whereas conventional confocal microscopy requires a sufficient number of photons to eliminate the influence of Poisson noise.

Additionally, PT-Confocal creates whole new opportunities for various optical imaging scenarios that involve sparse and weak signal detection, such as metabolite imaging^[15,16,62] and neuron imaging^[63,64]. PT-Confocal is capable of detecting extremely weak signals and hence detects tiny changes in the levels of metabolites in living systems, and thus may represent a novel optical in vivo metabolite imaging solution. Moreover, specific tissues, such as the spin cord or brain, may have a relatively sparse signal intensity. Reconstructing images through the temporal distributions of a few photons could avoid unnecessary exposure.

In summary, we present a confocal microscopic imaging system based on photonic timestamped reconstruction. While most confocal fluorescence microscopy systems require long exposure times and strong excitation, PT-Confocal can reconstruct results using as few as the timestamped information of ten photons, significantly reducing the exposure of sample by at least a factor of 100 compared to conventional methods. PT-Confocal may significantly improve the low-light performance of the immuno-fluorescent imaging and significantly reduce the minimum number of photons needed to explore life activities. The pioneering work will boost other significant applications, in particular live biomedical microscopy requiring low-flux detection, which will pave the way for more extensive and practical exploration of single-photon microscopic imaging.

The details of the experimental setup are shown in Fig. S1 in the Supplementary Materials. The pulsed laser (WSLP-488-012 m-4-PD,WSLP-405-015 m-4 and WSLP-520-030 m-4-PD, Wavespeectrum-laser, Inc.) is controlled by a voltage regulator and an output of FPGA. The voltage regulator provides a base voltage that allows the laser’s current to be 0.001 A, and the FPGA pulses for excitation. The single-mode laser goes through a single-mode fiber and a protected silver reflective collimator (RC04FC-P01, Thorlabs, Inc.), and becomes parallel light. The band-pass (Newport, Inc.) filters light enters a dual-axis scanning galvo system (GVS012, Thorlabs, Inc.). FPGA also achieves point-by-point 2D scanning by controlling the voltage of the mirror in the galvo mirror system. The galvo mirrors coordinate with a bi-telecentric optical system containing a scanning lens (CLS-SL, Thorlabs, Inc.) and a tube lens (TTL200-A, Thorlabs, Inc.) to scan linearly on the focal plane. An immuno-stained sample (F24630, ThermoFisher Scientific, Inc. and ArgoSIM, Argolight, Inc.) is placed near the focal plane of the objective. The emission light of the sample is filtered and collected by the lens and pinhole (P25K, Thorlabs, Inc.) in the reflected path of BS. SPAD (SPCM-NIR, Excelitas Technologies, Corp.) detects the signal, with a dead time of 20 ns and a measured dark count of 50 cps (count per second). The TCSPC module records the location signal from controller and the timestamps of excited photons from the optical path. BS: 50:50 beam splitter.

All recorded images are limited by the Poisson noise. The image denoising algorithm attempts to remove this noise from the image. The methodology of the discrete wavelet transform-based image de-noising is very effective to handle the recorded images corrupted by Poisson noise, which generally has the following process. The discrete 2D wavelet transform can handle the noisy images in the orthogonal domain. Then, thresholding the noisy detail coefficients. Finally, performing an inverse discrete wavelet transform to obtain the de-noised image in the spatial domain. The reasonable threshold plays an important role in the denoising process. A small threshold value will retain the noisy coefficients whereas a large threshold value leads to the loss of coefficients that carry image signal details. The regularization parameter we used is penalty coefficient. We tried it and chose a value of 0.08. The wavelet we use is Daubechies, type: orthogonal, and the 4th order wavelet decomposition is used.

The deep Fourier attention network (DFCAN) model is defined and implemented in a previous work^[54]. The previous work demonstrated the ability of DFCAN in restoring the structured illumination microscopy (SIM) image from a low signal to noise ratio frequency domain image captured by SIM. The main block in DFCAN is the Fourier channel attention (FCA) module. In this module, we introduced a fast Fourier transform layer to apply different layers in the spectrum space. In our experiment, the fluctuation introduced by Poisson fitting will mainly affect the high-frequency components. The DFCAN model we adopted is adapted to the purpose of denoising while maintaining high data fidelity. The neural networks is implemented in Python 3.7 and TensorFlow 2.0 framework and trained with NVIDIA GTX2060Q 6 G.