Semisupervised charting for spectral multimodal manifold learning and alignment. (March 2021)

• Record Type: Journal Article
• Authors: Pournemat, Ali; Adibi, Peyman; Chanussot, Jocelyn
• Is Part Of: Pattern recognition. Volume 111(2021)
• Publication Date: 2021-03
• Subjects: Semi-supervised learning -- Multimodal data -- Functional map -- Manifold learning -- Data fusion -- Hyperspectral images; Pattern perception -- Periodicals; Perception des structures -- Périodiques; Patroonherkenning; 006.4
• DOI: 10.1016/j.patcog.2020.107645
• Languages: English
• ISSNs: 0031-3203

Abstract

Highlights: A novel multimodal data fusion semisupervised manifold learning technic is proposed. Functional mapping to extend limited supervised information on modalities manifolds. Simultaneously learns manifold in each modality, and aligns them. Joint diagonalization of within- and between-modality graph Laplacian. Decoupling/modality-specific parts are deemphasized during learning common manifold. Abstract: For one given scene, multimodal data are acquired from multiple sensors. They share some similarities across the sensor types (redundant part of the information, also called coupling part) and they also provide modality-specific information (dissimilarities across the sensors, also called decoupling part). Additional critical knowledge about the scene can hence be extracted, which is not extractable from each modality alone. For the processing of multimodal data, we propose in this paper a model to simultaneously learn the underlying low-dimensional manifold in each modality, and locally align these manifolds across different modalities. For each pair of modalities we first build a common manifold that represents the corresponding (redundant) part of information, ignoring non-corresponding (modality specific) parts. We propose a semi-supervised learning model, using a limited amount of prior knowledge about the coupling and decoupling components of the different modalities. We propose a localized version of Laplacian eigenmaps technique specifically designed to handle multimodal manifold learning, in which the ideas of local patching of the manifolds, also known as manifold charting, is combined with the joint spectral analysis of the graph Laplacians of the different modalities. The limited given supervised information is then extending on the manifold of each modality. The idea of functional mapping is finally used to align the different manifolds across modalities. The evaluation of the proposed model using synthetic and real-world multimodal problems shows promising results, compared to several related techniques.