Topic driven multimodal similarity learning with multi-view voted convolutional features. (March 2018)

• Record Type: Journal Article
• Authors: Gao, Xinjian; Mu, Tingting; Goulermas, John Y.; Wang, Meng
• Is Part Of: Pattern recognition. Volume 75(2018:Mar.)
• Page Start: 223
• Page End: 234
• Publication Date: 2018-03
• Subjects: Convolutional auto-encoder -- Representation learning -- Multi-view learning -- Multimodal similarity learning; Pattern perception -- Periodicals; Perception des structures -- Périodiques; Patroonherkenning; 006.4
• DOI: 10.1016/j.patcog.2017.02.035
• Languages: English
• ISSNs: 0031-3203

Abstract

Highlights: A novel similarity learning model with layered architecture. The representation layer preserves a multi-view voted local neighbour structure. The multimodal layer computes distributional similarity over sparse relation types. The hidden relation neurons are initialized by cluster centres to encode topics. Comparison with seven competing methods shows effectiveness of the proposed model. Abstract: Similarity (and distance metric) learning plays a very important role in many artificial intelligence tasks aiming at quantifying the relevance between objects. We address the challenge of learning complex relation patterns from data objects exhibiting heterogeneous properties, and develop an effective multi-view multimodal similarity learning model with much improved learning performance and model interpretability. The proposed method firstly computes multi-view convolutional features to achieve improved object representation, then analyses the similarities between objects by operating over multiple hidden relation types (modalities), and finally fine-tunes the entire model variables via back-propagating a ranking loss to the convolutional layers. We develop a topic-driven initialization scheme, so that each learned relation type can be interpreted as a representative of semantic topics of the objects. To improve model interpretability and generalization, sparsity is imposed over these hidden relations. The proposed method is evaluated by solving the image retrieval task using challenging image datasets, and is compared with seven state-of-the-art algorithms in the field. Experimental results demonstrate significant performance improvement of the proposed method over the competing ones.