• Infrared and Laser Engineering
  • Vol. 51, Issue 11, 20220536 (2022)
Zihan Xiong1,2, Liangfeng Song1,2, Xin Liu1, Chao Zuo1, and Peng Gao1
Author Affiliations
  • 1School of Physics, Xidian University, Xi’an 710071, China
  • 2Hangzhou Institute of Technology, Xidian University, Hangzhou 311200, China
  • show less
    DOI: 10.3788/IRLA20220536 Cite this Article
    Zihan Xiong, Liangfeng Song, Xin Liu, Chao Zuo, Peng Gao. Performance enhancement of fluorescence microscopy by using deep learning (invited)[J]. Infrared and Laser Engineering, 2022, 51(11): 20220536 Copy Citation Text show less
    References

    [1] N Hamilton. Quantification and its applications in fluorescent microscopy imaging. Traffic, 10, 951-961(2009).

    [2] R Shi, C Jin, H Xie, et al. Multi-plane, wide-field fluorescent microscopy for biodynamic imaging in vivo. Biomed Opt Express, 10, 6625-6635(2019).

    [3] J W Goodman, R Lawrence. Digital image formation from electronically detected holograms. Applied Physics Letters, 11, 77-79(1967).

    [4] Y Fan, J Li, L Lu, et al. Smart computational light microscopes (SCLMs) of smart computational imaging laboratory (SCILab). PhotoniX, 2, 1-64(2021).

    [5] P Gao, C Yuan. Resolution enhancement of digital holographic microscopy via synthetic aperture: a review. Light: Advanced Manufacturing, 3, 105-120(2022).

    [6] Peng Gao, Kai Wen, Xueying Sun, et al. Review of resolution enhancement technologies in quantitative phase microscopy. Infrared and Laser Engineering, 48, 0603007(2019).

    [7] J W Lichtman, J A Conchello. Fluorescence microscopy. Nature Methods, 2, 910-919(2005).

    [8] J A Conchello, J W Lichtman. Optical sectioning microscopy. Nature Methods, 2, 920-931(2005).

    [9] K E Murfin, J Chaston, H Goodrich-Blair. Visualizing bacteria in nematodes using fluorescent microscopy. Journal of Visualized Experiments, 68, e4298(2012).

    [10] M Mickoleit, B Schmid, M Weber, et al. High-resolution reconstruction of the beating zebrafish heart. Nature Methods, 11, 919-922(2014).

    [11] B N Giepmans, S R Adams, M H Ellisman, et al. The fluorescent toolbox for assessing protein location and function. Science, 312, 217-224(2006).

    [12] A E Palmer, R Y Tsien. Measuring calcium signaling using genetically targetable fluorescent indicators. Nature Protocols, 1, 1057-1065(2006).

    [13] J Boulanger, C Kervrann, P Bouthemy, et al. Patch-based nonlocal functional for denoising fluorescence microscopy image sequences. IEEE Transactions on Medical Imaging, 29, 442-454(2009).

    [14] E Betzig, G H Patterson, R Sougrat, et al. Imaging intracellular fluorescent proteins at nanometer resolution. Science, 313, 1642-1645(2006).

    [15] S W Hell, J Wichmann. Breaking the diffraction resolution limit by stimulated emission: stimulated-emission-depletion fluo-rescence microscopy. Optics Letters, 19, 780-782(1994).

    [16] M G Gustafsson. Surpassing the lateral resolution limit by a factor of two using structured illumination microscopy. Journal of Microscopy, 198, 82-87(2000).

    [17] P Gao, B Prunsche, L Zhou, et al. Background suppression in fluorescence nanoscopy with stimulated emission double depletion. Nature Photonics, 11, 163-169(2017).

    [18] T A Klar, S Jakobs, M Dyba, et al. Fluorescence microscopy with diffraction resolution barrier broken by stimulated emission. Proc Natl Acad Sci USA, 97, 8206-8210(2000).

    [19] H Shroff, C G Galbraith, J A Galbraith, et al. Live-cell photoactivated localization microscopy of nanoscale adhesion dynamics. Nat Methods, 5, 417-423(2008).

    [20] J Mertz. Optical sectioning microscopy with planar or structured illumination. Nature Methods, 8, 811-819(2011).

    [21] J Icha, M Weber, J C Waters, et al. Phototoxicity in live fluorescence microscopy, and how to avoid it. Bioessays, 39, 1700003(2017).

    [22] D A Helmerich, G Beliu, S S Matikonda, et al. Photoblueing of organic dyes can cause artifacts in super-resolution microscopy. Nature Methods, 18, 253-257(2021).

    [23] Wang SC. Artificial Neural wk [M]Interdisciplinary Computing in Java Programming. Boston, MA: Springer, 2003: 81100.

    [24] H C Shin, H R Roth, M Gao, et al. Deep convolutional neural networks for computer-aided detection: CNN architectures, dataset characteristics and transfer learning. IEEE Trans Med Imaging, 35, 1285-1298(2016).

    [25] H Wang, Y Rivenson, Y Jin, et al. Deep learning enables cross-modality super-resolution in fluorescence microscopy. Nature Methods, 16, 103-110(2019).

    [26] H Zhang, C Fang, X Xie, et al. High-throughput, high-resolution deep learning microscopy based on registration-free generative adversarial network. Biomed Opt Express, 10, 1044-1063(2019).

    [27] H Zhou, R Cai, T Quan, et al. 3D high resolution generative deep-learning network for fluorescence microscopy imaging. Optics Letters, 45, 1695-1698(2020).

    [28] M Z Li, H M Shan, S Pryshchep, et al. Deep adversarial network for super stimulated emission depletion imaging. Journal of Nanophotonics, 14, 016009(2020).

    [29] C N Christensen, E N Ward, P Lio, et al. ML-SIM: Universal reconstruction of structured illumination microscopy images using transfer learning. Biomedical Optics Express, 12, 2720-2733(2021).

    [30] Z H Shah, M Müller, T C Wang, et al. Deep-learning based denoising and reconstruction of super-resolution structured illumination microscopy images. Photonics Research, 9, B168-B181(2021).

    [31] L Jin, B Liu, F Zhao, et al. Deep learning enables structured illumination microscopy with low light levels and enhanced speed. Nature Communications, 11, 1934(2020).

    [32] C Ling, C L Zhang, M Q Wang, et al. Fast structured illumination microscopy via deep learning. Photonics Research, 8, 1350-1359(2020).

    [33] Boyd N, Jonas E, Babcock H, et al. DeepLoco: Fast 3D localization microscopy using neural wks [ZOL]. bixiv, (20180226)[20220801]. https:doi.g10.1101267096.

    [34] E Nehme, L E Weiss, T Michaeli, et al. Deep-STORM: super-resolution single-molecule microscopy by deep learning. Optica, 5, 458-464(2018).

    [35] A Speiser, L R Müller, P Hoess, et al. Deep learning enables fast and dense single-molecule localization with high accuracy. Nature Methods, 18, 1082-1090(2021).

    [36] M Weigert, U Schmidt, T Boothe, et al. Content-aware image restoration: pushing the limits of fluorescence microscopy. Nature Methods, 15, 1090-1097(2018).

    [37] Z Wang, L Zhu, H Zhang, et al. Real-time volumetric reconstruction of biological dynamics with light-field microscopy and deep learning. Nature Methods, 18, 551-556(2021).

    [38] X Zhang, Y Chen, K Ning, et al. Deep learning optical-sectioning method. Optics Express, 26, 30762-30772(2018).

    [39] C Bai, C Liu, X H Yu, et al. Imaging enhancement of light-sheet fluorescence microscopy via deep learning. IEEE Photonics Technology Letters, 31, 1803-1806(2019).

    [40] L Huang, H Chen, Y Luo, et al. Recurrent neural network-based volumetric fluorescence microscopy. Light Sci Appl, 10, 62(2021).

    [41] Y C Wu, Y Rivenson, H D Wang, et al. Three-dimensional virtual refocusing of fluorescence microscopy images using deep learning. Nature Methods, 16, 1323-1331(2019).

    [42] K Ning, X Zhang, X Gao, et al. Deep-learning-based whole-brain imaging at single-neuron resolution. Biomedical Optics Express, 11, 3567-3584(2020).

    [43] C Bai, X Yu, T Peng, et al. 3D imaging restoration of spinning-disk confocal microscopy via deep learning. IEEE Photonics Technology Letters, 32, 1131-1134(2020).

    [44] H Zhang, Y Zhao, C Fang, et al. Exceeding the limits of 3D fluorescence microscopy using a dual-stage-processing network. Optica, 7, 1627-1640(2020).

    [45] L Hu, S Hu, W Gong, et al. Image enhancement for fluorescence microscopy based on deep learning with prior knowledge of aberration. Optics Letters, 46, 2055-2058(2021).

    [46] L Xiao, C Fang, L Zhu, et al. Deep learning-enabled efficient image restoration for 3D microscopy of turbid biological specimens. Optics Express, 28, 30234-30247(2020).

    [47] Y Lecun, Y Bengio, G Hinton. Deep learning. Nature, 521, 436-444(2015).

    [48] W S Mcculloch, W Pitts. A logical calculus of the ideas immanent in nervous activity. The Bulletin of Mathematical Biophysics, 5, 115-133(1943).

    [49] Lecun Y. A theetical framewk f backpropagation[C]Proceedings of the 1988 Connectionist Models Summer School, 1988: 2128.

    [50] G E Hinton, R R Salakhutdinov. Reducing the dimensionality of data with neural networks. Science, 313, 504-507(2006).

    [51] Krizhevsky A, Sutskever I, Hinton G. Image classification with deep convolutional neural wks[C]NIPS''12: Proceedings of the 25th International Conference on Neural Infmation Processing Systems, 2012, 1: 1097–1105.

    [52] G Ongie, A Jalal, C A Metzler, et al. Deep learning techniques for inverse problems in imaging. IEEE Journal on Selected Areas in Information Theory, 1, 39-56(2020).

    [53] N Ghosh, K Bhattacharya. Cube beam-splitter interferometer for phase shifting interferometry. Journal of Optics, 38, 191-198(2009).

    [54] M T Mccann, K H Jin, M Unser. Convolutional neural networks for inverse problems in imaging: A review. IEEE Signal Processing Magazine, 34, 85-95(2017).

    [55] O''shea K, Nash R. An introduction to convolutional neural wks [EBOL]. (20151126)[20220801]. https:arxiv.gabs1511.08458.

    [56] Pang S, Du A, gun M A, et al. Beyond CNNs: exploiting further inherent symmetries in medical images f segmentation [EBOL]. (20200508)[20220801]. https:arxiv.gabs2005.03924.

    [57] Ronneberger O, Fischer P, Brox T. U: Convolutional wks f biomedical image segmentation[C]International Conference on Medical Image Computing Computerassisted Intervention, 2015: 234241.

    [58] Feizabadi M M, Shujjat A M, Shahid S, et al. Interactive latent interpolation on MNIST dataset [EBOL]. (20201015)[20220801]. https:arxiv.gabs2010.07581.

    [59] Linlin Zhu, Lu Han, Hong Du, et al. Multi-active contour cell segmentation method based on U-Net network. Infrared and Laser Engineering, 49, 20200121(2020).

    [60] L R Medsker, L Jain. Recurrent neural networks. Design and Applications, 5, 64-67(2001).

    [61] S Hochreiter, J Schmidhuber. Long short-term memory. Neural Computation, 9, 1735-1780(1997).

    [62] Vinyals O, Toshev A, Bengio S, et al. Show tell: A neural image caption generat[C]Proceedings of the IEEE conference on computer vision pattern recognition, 2015: 31563164.

    [63] Sutskever I, Vinyals O, Le Q V. Sequence to sequence learning with neural wks [EBOL]. (20140910)[20220801]. https:arxiv.gabs1409.3215.

    [64] Graves A. Generating sequences with recurrent neural wks [EBOL]. (20130804)[20220801]. https:arxiv.gabs1308.0850v5.

    [65] M Sajjad, S Kwon. Clustering-based speech emotion recognition by incorporating learned features and deep BiLSTM. IEEE Access, 8, 79861-79875(2020).

    [66] Goodfellow I, PougetAbadie J, Mirza M, et al. Generative adversarial s [EBOL]. (20140610)[20220801]. https:arxiv.gabs1406.2661.

    [67] Isola P, Zhu JY, Zhou T, et al. Imagetoimage translation with conditional adversarial wks[C]Proceedings of the IEEE Conference On Computer Vision Pattern Recognition, 2017: 11251134.

    [68] Brock A, Donahue J, Simonyan K. Large scale GAN training f high fidelity natural image synthesis [EBOL]. (20180928)[20220801]. https:arxiv.gabs1809.11096v2.

    [69] Cao J, Hou L, Yang MH, et al. Remix: Towards imagetoimage translation with limited data[C]Proceedings of the IEEECVF Conference on Computer Vision Pattern Recognition, 2021: 1501815027.

    [70] Wang X, Yu K, Wu S, et al. Esrgan: Enhanced superresolution generative adversarial wks[C]Proceedings of the European Conference On Computer Vision (ECCV) Wkshops, 2018.

    [71] E Abbe. Beiträge zur theorie des mikroskops und der mikroskopischen wahrnehmung. Archiv für Mikroskopische Anatomie, 9, 413-418(1873).

    [72] Pawley J. Hbook of Biological Confocal Microscopy [M]. New Yk: Springer Science & Business Media, 2006.

    [73] Wei Ji, Tao Xu, Bei Liu. Super-resolution fluorescent micro-scopy: A brief introduction to the Nobel Prize in Chemistry 2014. Chinese Journal of Nature, 36, 404-408(2014).

    [74] M J Rust, M Bates, X J N M Zhuang. Sub-diffraction-limit imaging by stochastic optical reconstruction microscopy (STORM). Nature Methods, 3, 793-796(2006).

    [75] R Heintzmann, T Huser. Super-resolution structured illumi-nation microscopy. Chemical Reviews, 117, 13890-13908(2017).

    [76] J Tam, D Merino. Stochastic optical reconstruction microscopy (STORM) in comparison with stimulated emission depletion (STED) and other imaging methods. J Neurochem, 135, 643-658(2015).

    [77] B Huang, M Bates, X Zhuang. Super-resolution fluorescence microscopy. Annu Rev Biochem, 78, 993-1016(2009).

    [78] L Schermelleh, R Heintzmann, H Leonhardt. A guide to super-resolution fluorescence microscopy. J Cell Biol, 190, 165-175(2010).

    [79] J P Nguyen, F B Shipley, A N Linder, et al. Whole-brain calcium imaging with cellular resolution in freely behaving Caenorhabditis elegans. Proc Natl Acad Sci USA, 113, E1074-1081(2016).

    [80] M F Juette, T J Gould, M D Lessard, et al. Three-dimensional sub-100 nm resolution fluorescence microscopy of thick samples. Nature Methods, 5, 527-529(2008).

    [81] P Prabhat, S Ram, E S Ward, et al. Simultaneous imaging of different focal planes in fluorescence microscopy for the study of cellular dynamics in three dimensions. IEEE Transactions on NanoBioscience, 3, 237-242(2004).

    [82] C Johnson, J Exell, J Kuo, et al. Continuous focal translation enhances rate of point-scan volumetric microscopy. Optics Express, 27, 36241-36258(2019).

    [83] H Li, C Guo, D Kim-Holzapfel, et al. Fast, volumetric live-cell imaging using high-resolution light-field microscopy. Biomedical Optics Express, 10, 29-49(2019).

    [84] M Pascucci, S Ganesan, A Tripathi, et al. Compressive three-dimensional super-resolution microscopy with speckle-saturated fluorescence excitation. Nature Communications, 10, 1327(2019).

    [85] H Gong, D Xu, J Yuan, et al. High-throughput dual-colour precision imaging for brain-wide connectome with cytoarchitectonic landmarks at the cellular level. Nature Communications, 7, 1-12(2016).

    [86] P M Carlton, J Boulanger, C Kervrann, et al. Fast live simultaneous multiwavelength four-dimensional optical microscopy. Proceedings of the National Academy of Sciences, 107, 16016-16022(2010).

    [87] F Luisier, T Blu, M Unser. Image denoising in mixed Poisson-Gaussian noise. IEEE Trans Image Process, 20, 696-708(2011).

    [88] E Soubies, F Soulez, M T Mccann, et al. Pocket guide to solve inverse problems with GlobalBioIm. Inverse Problems, 35, 104006(2019).

    [89] M Arigovindan, J C Fung, D Elnatan, et al. High-resolution restoration of 3D structures from widefield images with extreme low signal-to-noise-ratio. Proceedings of the National Academy of Sciences, 110, 17344-17349(2013).

    [90] S Setzer, G Steidl, T Teuber. Deblurring Poissonian images by split Bregman techniques. Journal of Visual Commu-nication and Image Representation, 21, 193-199(2010).

    [91] Zhang Y, Zhu Y, Nichols E, et al. A poissongaussian denoising dataset with real fluescence microscopy images[C]Proceedings of the IEEECVF Conference on Computer Vision Pattern Recognition, 2019: 1171011718.

    [92] G M Hagen, J Bendesky, R Machado, et al. Fluorescence microscopy datasets for training deep neural networks. GigaScience, 10, giab032(2021).

    [93] C Belthangady, L A Royer. Applications, promises, and pitfalls of deep learning for fluorescence image reconstruction. Nat Methods, 16, 1215-1225(2019).

    [94] E M Christiansen, S J Yang, D M Ando, et al. In silico labeling: predicting fluorescent labels in unlabeled images. Cell, 173, 792-803e719(2018).

    [95] C Ounkomol, S Seshamani, M M Maleckar, et al. Label-free prediction of three-dimensional fluorescence images from transmitted-light microscopy. Nature Methods, 15, 917-920(2018).

    [96] I M Khater, S T Aroca-Ouellette, F Meng, et al. Caveolae and scaffold detection from single molecule localization microscopy data using deep learning. PLoS One, 14, e0211659(2019).

    [97] X Li, G Zhang, H Qiao, et al. Unsupervised content-preserving transformation for optical microscopy. Light Sci Appl, 10, 44(2021).

    [98] Chen X, Kel M E, He S, et al. Artificial confocal microscopy f deep labelfree imaging [EBOL]. (20211028)[20220801]. https:arxiv.gabs2110.14823.

    [99] Robitaille L É, Dur A, Gardner MA, et al. Learning to become an expert: Deep wks applied to superresolution microscopy[C]ThirtySecond AAAI Conference on Artificial Intelligence, 2018.

    Zihan Xiong, Liangfeng Song, Xin Liu, Chao Zuo, Peng Gao. Performance enhancement of fluorescence microscopy by using deep learning (invited)[J]. Infrared and Laser Engineering, 2022, 51(11): 20220536
    Download Citation