Concept formation through multimodal integration using multimodal BERT and VQ-VAE. (16th February 2023)

• Record Type: Journal Article
• Authors: Miyazawa, Kazuki; Nagai, Takayuki
• Is Part Of: Advanced robotics. Volume 37:Number 4(2023)
• Page Start: 281
• Page End: 296
• Publication Date: 2023-02-16
• Subjects: Multimodal learning -- robot learning -- transformers -- concept formation -- natural language processing; Robotics -- Periodicals; Robotics -- Japan -- Periodicals; Robotics; Japan; Periodicals; 629.89205
• DOI: 10.1080/01691864.2022.2141583
• Languages: English
• ISSNs: 0169-1864

Abstract

Abstract : Humans form concepts from multimodal information obtained from multiple sensory organs and understand the environment and language by predicting unobserved information through these concepts. Robots can also make better predictions by forming concepts from multimodal information in a bottom-up manner. In contrast, BERT, a language model, acquires semantic representations of sentences applicable to various language comprehension tasks by learning relationships between words through mask prediction learning. We believe that BERT can be extended to multimodality and learn multimodal representations for world understanding by learning relationships among data across modalities. Therefore, we propose a multimodal learning model using BERT and VQ-VAEs with multimodal information as input and evaluate the model in terms of model predictability and concept formation. To verify the usefulness of the proposed model, experiments were conducted using a multimodal dataset of objects acquired by a robot. The experimental results showed that the proposed model has higher category classification accuracy and word prediction performance than multimodal latent Dirichlet allocation (MLDA), which is used as a concept formation model. An analysis of the representation of the model showed that clusters of object categories were formed within the model. GRAPHICAL ABSTRACT: UF0001