• Optics and Precision Engineering
  • Vol. 32, Issue 2, 237 (2024)
Shibei LIU and Ying CHEN*
Author Affiliations
  • Key Laboratory of Advanced Process Control for Light Industry (Ministry of Education), Jiangnan University, Wuxi214122, China
  • show less
    DOI: 10.37188/OPE.20243202.0237 Cite this Article
    Shibei LIU, Ying CHEN. Audio object detection network with multimodal cross level feature knowledge transfer[J]. Optics and Precision Engineering, 2024, 32(2): 237 Copy Citation Text show less

    Abstract

    As one of the inherent properties of objects, sound can provide valuable information for target detection. At present, the method of target positioning only by monitoring environmental sound is less robust. To solve this problem, a multi-modal self-supervised target detection network under cross-level feature knowledge transfer was proposed. First of all, in view of the teachers network and students at the same characteristics of network learning ability of the limited problem, design based on the integration of teachers across level knowledge transfer loss, through the way of attention fusion deep and shallow characteristics of students, more efficient learning to the corresponding teacher middle layer characteristics, to extract more knowledge, combined with KL divergence, realize the alignment of teachers and students network alignment. In addition, in order to solve the problem of missing localization information, localization distillation loss was added, and more localization information was obtained by fitting the distribution of the teacher. With the network trained in the multimodal audiovisual detection MAVD dataset, the mAP values improve by 6.71%, 14.36% and 10.32% from the baseline network at IOU values of 0.5,0.75 and average, respectively. The experimental results demonstrate the superiority of this detection network.
    Shibei LIU, Ying CHEN. Audio object detection network with multimodal cross level feature knowledge transfer[J]. Optics and Precision Engineering, 2024, 32(2): 237
    Download Citation