• Advanced Photonics Nexus
  • Vol. 3, Issue 6, 066010 (2024)
Hang Su1,2,†, Yanping He1,2, Baoli Li1,2, Haitao Luan1,2..., Min Gu1,2 and Xinyuan Fang1,2,*|Show fewer author(s)
Author Affiliations
  • 1University of Shanghai for Science and Technology, School of Artificial Intelligence Science and Technology, Shanghai, China
  • 2University of Shanghai for Science and Technology, Institute of Photonic Chips, Shanghai, China
  • show less
    DOI: 10.1117/1.APN.3.6.066010 Cite this Article Set citation alerts
    Hang Su, Yanping He, Baoli Li, Haitao Luan, Min Gu, Xinyuan Fang, "Teacher-student learning of generative adversarial network-guided diffractive neural networks for visual tracking and imaging," Adv. Photon. Nexus 3, 066010 (2024) Copy Citation Text show less
    References

    [1] X. Wang. Intelligent multi-camera video surveillance: a review. Pattern Recognit. Lett., 34, 3-19(2013).

    [2] J. Janai et al. Computer vision for autonomous vehicles: problems, datasets and state of the art. Found. Trends Comput. Graph. Vis., 12, 1-308(2020).

    [3] Y. Shu et al. Interactive design of intelligent machine vision based on human–computer interaction mode. Microprocess Microsyst., 75, 103059(2020).

    [4] J. Li et al. Moving target detection and tracking algorithm based on context information. IEEE Access, 7, 70966-70974(2019).

    [5] X. Chen et al. Transformer tracking, 8126-8135(2021).

    [6] Z. Xu et al. A multichannel optical computing architecture for advanced machine vision. Light Sci. Appl., 11, 255(2022).

    [7] X. Feng et al. Computer vision algorithms and hardware implementations: a survey. Integration, 69, 309-320(2019).

    [8] G. Wetzstein et al. Inference in artificial intelligence with deep optics and photonics. Nature, 588, 39-47(2020).

    [9] W. Shi et al. LOEN: lensless opto-electronic neural network empowered machine vision. Light Sci. Appl., 11, 121(2022).

    [10] Y. Du et al. Object-adaptive LSTM network for real-time visual tracking with adversarial data augmentation. Neurocomputing, 384, 67-83(2020).

    [11] H. Rebecq et al. High speed and high dynamic range video with an event camera. IEEE Trans. Pattern Anal. Mach. Intell., 43, 1964-1980(2019).

    [12] N. Messikommer et al. Event-based asynchronous sparse convolutional networks, 415-431(2020).

    [13] G. Gallego et al. Event-based vision: a survey. IEEE Trans. Pattern Anal. Mach. Intell., 44, 154-180(2020).

    [14] L. Wang et al. Event-based high dynamic range image and very high frame rate video generation using conditional generative adversarial networks, 10081-10090(2019).

    [15] G. Chen et al. Event-based neuromorphic vision for autonomous driving: a paradigm shift for bio-inspired visual sensing and perception. IEEE Signal Process. Mag., 37, 34-49(2020).

    [16] R. W. Baldwin et al. Time-ordered recent event (TORE) volumes for event cameras. IEEE Trans. Pattern Anal. Mach. Intell., 45, 2519-2532(2022).

    [17] M. Gehrig, D. Scaramuzza. Recurrent vision transformers for object detection with event cameras, 13884-13893(2023).

    [18] X. Wang et al. Visevent: reliable object tracking via collaboration of frame and event flows. IEEE Trans. Cybern.(2023).

    [19] H. Luan et al. 768-ary Laguerre-Gaussian-mode shift keying free-space optical communication based on convolutional neural networks. Opt. Express, 29, 19807-19818(2021).

    [20] Y. Han et al. Robust visual tracking based on adversarial unlabeled instance generation with label smoothing loss regularization. Pattern Recognit., 97, 107027(2020).

    [21] H. Yu et al. Conditional GAN based individual and global motion fusion for multiple object tracking in UAV videos. Pattern Recognit. Lett., 131, 219-226(2020).

    [22] J. T. Springenberg. Unsupervised and semi-supervised learning with categorical generative adversarial networks. arXiv:1511.06390(2015).

    [23] J. Gui et al. A review on generative adversarial networks: algorithms, theory, and applications. IEEE Trans. Knowl. Data Eng., 35, 3313-3332(2021).

    [24] T. Karras et al. A style-based generator architecture for generative adversarial networks, 4401-4410(2019).

    [25] T. Karras et al. Alias-free generative adversarial networks. Adv. Neural Inform. Process. Syst., 34, 852-863(2021).

    [26] C. Liu et al. Intelligent coding metasurface holograms by physics-assisted unsupervised generative adversarial network. Photonics Res., 9, B159-B167(2021).

    [27] W. Xu et al. DRB-GAN: a dynamic resblock generative adversarial network for artistic style transfer, 6383-6392(2021).

    [28] Y. H. Kim et al. GRA-GAN: generative adversarial network for image style transfer of gender, race, and age. Expert Syst. Appl., 198, 116792(2022).

    [29] F. Fahimi et al. Generative adversarial networks-based data augmentation for brain–computer interface. IEEE Trans. Neural Netw. Learn. Syst., 32, 4039-4051(2020).

    [30] B. Bosquet et al. A full data augmentation pipeline for small object detection based on generative adversarial networks. Pattern Recogn., 133, 108998(2023).

    [31] B. Li et al. Ultralow-power spiking neural networks for 1024-ary orbital angular momentum shift keying free-space optical communication. J. Opt., 25, 074001(2023).

    [32] W. Peebles et al. GAN-supervised dense visual alignment, 13470-13481(2022).

    [33] X. Lin et al. All-optical machine learning using diffractive deep neural networks. Science, 361, 1004-1008(2018).

    [34] X. Yang et al. Complex-valued universal linear transformations and image encryption using spatially incoherent diffractive networks. Adv. Photonics Nexus, 3, 016010(2024).

    [35] K. Zhang et al. Advanced all-optical classification using orbital-angular-momentum-encoded diffractive networks. Adv. Photonics Nexus, 2, 066006(2023).

    [36] Y. Huang et al. Sophisticated deep learning with on-chip optical diffractive tensor processing. Photonics Res., 11, 1125-1138(2023).

    [37] Ç. Işıl et al. All-optical image denoising using a diffractive visual processor. Light Sci. Appl., 13, 43(2024).

    [38] X. Luo et al. Metasurface-enabled on-chip multiplexed diffractive neural networks in the visible. Light Sci. Appl., 11, 158(2022).

    [39] T. Zhou et al. In situ optical backpropagation training of diffractive optical neural networks. Photonics Res., 8, 940-953(2020).

    [40] T. Yan et al. All-optical graph representation learning using integrated diffractive photonic computing units. Sci. Adv., 8, eabn7630(2022).

    [41] Z. Li et al. Event-based diffractive neural network chip for dynamic action recognition. Opt. Laser Technol., 169, 110136(2024).

    [42] J. Li et al. Class-specific differential detection in diffractive optical neural networks improves inference accuracy. Adv. Photonics, 1, 046001(2019).

    [43] M. S. Sakib Rahman, A. Ozcan. Integration of programmable diffraction with digital neural networks. ACS Photonics, 11, 2906-2922(2024).

    [44] Y. Li et al. Analysis of diffractive neural networks for seeing through random diffusers. IEEE J. Sel. Top. Quantum Electron., 29, 1-17(2022).

    [45] B. Bai et al. To image, or not to image: class-specific diffractive cameras with all-optical erasure of undesired objects. eLight, 2, 1-20(2022).

    [46] X. Chang et al. Complex-domain-enhancing neural network for large-scale coherent imaging. Adv. Photonics Nexus, 2, 046006(2023).

    [47] M. Gu et al. Optically digitalized holography: a perspective for all-optical machine learning. Engineering, 5, 363-365(2019).

    [48] X. Fang et al. Orbital angular momentum-mediated machine learning for high-accuracy mode-feature encoding. Light Sci. Appl., 13, 49(2024).

    [49] M. S. S. Rahman et al. Learning diffractive optical communication around arbitrary opaque occlusions. Nat. Commun., 14, 6830(2023).

    [50] Y. Li et al. Optical information transfer through random unknown diffusers using electronic encoding and diffractive decoding. Adv. Photonics, 5, 046009(2023).

    [51] C. Qian et al. Performing optical logic operations by a diffractive neural network. Light Sci. Appl., 9, 59(2020).

    [52] H. Zhu et al. Space-efficient optical computing with an integrated chip diffractive neural network. Nat. Commun., 13, 1044(2022).

    [53] T. Fu et al. Photonic machine learning with on-chip diffractive optics. Nat. Commun., 14, 70(2023).

    [54] T. Zhou et al. Large-scale neuromorphic optoelectronic computing with a reconfigurable diffractive processing unit. Nat. Photonics, 15, 367-373(2021).

    [55] S. Zheng et al. Incoherent imaging through highly nonstatic and optically thick turbid media based on neural network. Photonics Res., 9, B220-B228(2021).

    [56] X. Fang et al. High-dimensional orbital angular momentum multiplexing nonlinear holography. Adv. Photonics, 3, 015001(2021).

    [57] J. Chen et al. Dynamic graph CNN for event-camera based gesture recognition, 1-5(2020).

    [58] Y. Deng et al. A voxel graph CNN for object classification with event cameras, 1172-1181(2022).

    [59] P. Wzorek, T. Kryjak. Traffic sign detection with event cameras and DCNN, 86-91(2022).

    [60] F. Becattini et al. Understanding human reactions looking at facial microexpressions with an event camera. IEEE Trans. Ind. Inf.., 18, 9112-9121(2022).

    [61] G. Jing et al. Neural network-based surrogate model for inverse design of metasurfaces. Photonics Res., 10, 1462-1471(2022).

    [62] I. Goodfellow et al. Generative adversarial nets. Adv. Neural Inform. Process. Sys., 27(2014).

    [63] E. Brophy et al. Generative adversarial networks in time series: a systematic literature review. ACM Comput. Surv., 55, 1-31(2023).

    [64] D. Torbunov et al. UVCGAN: UNet vision transformer cycle-consistent GAN for unpaired image-to-image translation, 702-712(2023).

    [65] J.-Y. Zhu et al. Unpaired image-to-image translation using cycle-consistent adversarial networks, 2223-2232(2017).

    [66] Y. Yuan et al. Unsupervised image super-resolution using cycle-in-cycle generative adversarial networks, 701-710(2018).

    [67] M. Zhao et al. A 3D nanoscale optical disk memory with petabit capacity. Nature, 626, 772-778(2024).

    [68] B. Bai et al. Pyramid diffractive optical networks for unidirectional image magnification and demagnification. Light Sci. Appl., 13, 178(2024).

    [69] K. Li et al. Multi-dimensional multiplexing optical secret sharing framework with cascaded liquid crystal holograms. Opto-Electron. Adv., 7, 230121(2024).

    [70] W. Meng et al. 100 Hertz frame-rate switching three-dimensional orbital angular momentum multiplexing holography via cross convolution. Opto-Electron. Sci., 1, 220004-220004(2022).

    [71] J. Li et al. Unidirectional imaging using deep learning–designed materials. Sci. Adv., 9, eadg1505(2023).

    Hang Su, Yanping He, Baoli Li, Haitao Luan, Min Gu, Xinyuan Fang, "Teacher-student learning of generative adversarial network-guided diffractive neural networks for visual tracking and imaging," Adv. Photon. Nexus 3, 066010 (2024)
    Download Citation