Graph-based multimodal fusion with metric learning for multimodal classification. (November 2019)

• Record Type: Journal Article
• Authors: Angelou, Michalis; Solachidis, Vassilis; Vretos, Nicholas; Daras, Petros
• Is Part Of: Pattern recognition. Volume 95(2019:Nov.)
• Page Start: 296
• Page End: 307
• Publication Date: 2019-11
• Subjects: Multimodal fusion -- Multimodal metric learning -- Multimodal classification -- Distance graphs; Pattern perception -- Periodicals; Perception des structures -- Périodiques; Patroonherkenning; 006.4
• DOI: 10.1016/j.patcog.2019.06.013
• Languages: English
• ISSNs: 0031-3203

Abstract

Highlights: Multimodal classification results surpasses the state-of-the-art methods. Works with missing modalities. Works with any kind and any number of modalities. Abstract: In this paper, a graph-based, supervised classification method for multimodal data is introduced. It can be applied on data of any type consisting of any number of modalities and can also be used for the classification of datasets with missing modalities. The proposed method maps the features extracted from every modality to a space where the intrinsic structure of the multimodal data is kept. In order to map the extracted features of the different modalities into the same space and, at the same time, maintain the feature distances between similar and dissimilar modality data instances, a metric learning method is used. The proposed method has been evaluated on NUS-Wide, NTU-RGBD and AV-Letters multimodal datasets and has shown competitive results with the state-of-the-art methods in the field, while is able to cope with datasets with missing modalities.