Brain-inspired Multimodal Learning Based on Neural Networks. Issue 1 (September 2018)

• Record Type: Journal Article
• Authors: Liu, Chang; Sun, Fuchun; Zhang, Bo
• Is Part Of: Brain science advances. Volume 4:Issue 1(2018)
• Page Start: 61
• Page End: 72
• Publication Date: 2018-09
• Subjects: multimodal learning -- brain-inspired learning -- deep learning -- neural networks; Neurosciences -- Periodicals; Medical innovations -- Periodicals; Medical innovations; Neurosciences; Periodicals; 616.8005
• DOI: 10.26599/BSA.2018.9050004
• Languages: English
• ISSNs: 2096-5958

Abstract

Modern computational models have leveraged biological advances in human brain research. This study addresses the problem of multimodal learning with the help of brain-inspired models. Specifically, a unified multimodal learning architecture is proposed based on deep neural networks, which are inspired by the biology of the visual cortex of the human brain. This unified framework is validated by two practical multimodal learning tasks: image captioning, involving visual and natural language signals, and visual-haptic fusion, involving haptic and visual signals. Extensive experiments are conducted under the framework, and competitive results are achieved.