A deep grouping fusion neural network for multimedia content understanding. Issue 9 (12th April 2022)

• Record Type: Journal Article
• Authors: Song, Lingyun; Yu, Mengzhen; Shang, Xuequn; Lu, Yu; Liu, Jun; Zhang, Ying; Li, Zhanhuai
• Is Part Of: IET image processing. Volume 16:Issue 9(2022)
• Page Start: 2398
• Page End: 2411
• Publication Date: 2022-04-12
• Subjects: Image processing -- Periodicals; 621.36705
• DOI: 10.1049/ipr2.12496
• Languages: English
• ISSNs: 1751-9659

Abstract

Abstract: How Deep Neural Networks (DNNs) best cope with the understanding of multimedia contents still remains an open problem, mainly due to two factors. First, conventional DNNs cannot effectively learn the representations of the images with sparse visual information. For example, the images describing knowledge concepts in textbooks. Second, existing DNNs cannot effectively capture the fine‐grained interactions between the images and text descriptions. To address these issues, we propose a deep Cross‐Media Grouping Fusion Network (CMGFN), which mainly has two distinctive properties: 1) CMGFN can effectively learn visual features from the images with sparse visual information. This is achieved by first progressively adjusting the attention of convolution filters to valuable visual regions, and then enhancing the use of key visual information in feature construction. 2) By a cross‐media grouping co‐attention mechanism, CMGFN can effectively use the interactions between visual features of different semantics and textual descriptions, to learn cross‐media features representing different fine‐grained semantics in different groups. Empirical studies demonstrate that CMGFN not only achieves state‐of‐the‐art performance on the multimedia documents containing sparse visual information, but also shows superior general applicability on other multimedia data, e.g., the multimedia fake news.