An Attention Enhanced Cross-Modal Image–Sound Mutual Generation Model for Birds. (1st February 2021)

• Record Type: Journal Article
• Authors: Hao, Wangli; Han, Meng; Li, Shancang; Li, Fuzhong
• Is Part Of: Computer journal. Volume 65:Number 2(2022)
• Page Start: 410
• Page End: 422
• Publication Date: 2021-02-01
• Subjects: attention enhanced -- cross-modal audiovisual -- bird generation -- generative adversarial network; Computers -- Periodicals; 005.1
• DOI: 10.1093/comjnl/bxaa188
• Languages: English
• ISSNs: 0010-4620

Abstract

Abstract: Cross-modal bird image–audio mutual generation has appealing potential benefits for bird classification. To achieve promising cross-modal bird visual–audio mutual generation, we propose an attention enhanced cross-modal cycle adversarial generation network. Specifically, the attention module endows our model with long-term intra-modality dependency and inter-modality dependency capabilities, which can provide more information during the generation process and further improve the generation performance. Moreover, because there was no dataset concerning bird visual–audio mutual generation, the authors established a novel bird cross-modal generation dataset, called Bird_Crossmodal_Generation (BCG). Based on BCG, our model obtains promising performance and achieves significant improvement under both inception score and Frechet inception distance criteria. The experimental results validate the feasibility of the proposed task and the superiority of our model. Additionally, this investigation provides a basis for more researchers to develop cross-modality methods for bird visual–audio generation.