Modality-specific and shared generative adversarial network for cross-modal retrieval. (August 2020)

• Record Type: Journal Article
• Authors: Wu, Fei; Jing, Xiao-Yuan; Wu, Zhiyong; Ji, Yimu; Dong, Xiwei; Luo, Xiaokai; Huang, Qinghua; Wang, Ruchuan
• Is Part Of: Pattern recognition. Volume 104(2020:Aug.)
• Publication Date: 2020-08
• Subjects: Cross-modal retrieval -- Generative adversarial networks (GAN) -- Modality-specific feature learning -- Modality-shared feature learning; Pattern perception -- Periodicals; Perception des structures -- Périodiques; Patroonherkenning; 006.4
• DOI: 10.1016/j.patcog.2020.107335
• Languages: English
• ISSNs: 0031-3203

Abstract

Highlights: We propose a Modality-Specific and Shared Generative Adversarial Network approach. The modality-specific and modality-shared features are jointly explored and leveraged. The inter-modal invariance and the inter- and intra-modal discrimination is well modeled. Superiority of our approach is demonstrated on multiple benchmark multi-modal datasets. Abstract: Cross-modal retrieval aims to realize accurate and flexible retrieval across different modalities of data, e.g., image and text, which has achieved significant progress in recent years, especially since generative adversarial networks (GAN) were used. However, there still exists much room for improvement. How to jointly extract and utilize both the modality-specific (complementarity) and modality-shared (correlation) features effectively has not been well studied. In this paper, we propose an approach named Modality-Specific and Shared Generative Adversarial Network (MS 2 GAN) for cross-modal retrieval. The network architecture consists of two sub-networks that aim to learn modality-specific features for each modality, followed by a common sub-network that aims to learn the modality-shared features for each modality. Network training is guided by the adversarial scheme between the generative and discriminative models. The generative model learns to predict the semantic labels of features, model the inter- and intra-modal similarity with label information, and ensure the difference between the modality-specific and modality-shared features, while the discriminative model learns to classify the modality of features. The learned modality-specific and shared feature representations are jointly used for retrieval. Experiments on three widely used benchmark multi-modal datasets demonstrate that MS 2 GAN can outperform state-of-the-art related works.