'When' and 'what' did you see? A novel fMRI-based visual decoding framework. (7th October 2020)

• Record Type: Journal Article
• Authors: Wang, Chong; Yan, Hongmei; Huang, Wei; Li, Jiyi; Yang, Jiale; Li, Rong; Zhang, Leiyao; Li, Liang; Zhang, Jiang; Zuo, Zhentao; Chen, Huafu
• Is Part Of: Journal of neural engineering. Volume 17:Number 5(2020:Oct.)
• Publication Date: 2020-10-07
• Subjects: fMRI -- visual perception decoding -- recurrent neural networks (RNNs); Neurosciences -- Periodicals; Biomedical engineering -- Periodicals; 612.8
• DOI: 10.1088/1741-2552/abb691
• Languages: English
• ISSNs: 1741-2560

Abstract

Abstract: Objective. Visual perception decoding plays an important role in understanding our visual systems. Recent functional magnetic resonance imaging (fMRI) studies have made great advances in predicting the visual content of the single stimulus from the evoked response. In this work, we proposed a novel framework to extend previous works by simultaneously decoding the temporal and category information of visual stimuli from fMRI activities. Approach. 3 T fMRI data of five volunteers were acquired while they were viewing five categories of natural images with random presentation intervals. For each subject, we trained two classification-based decoding modules that were used to identify the occurrence time and semantic categories of the visual stimuli. In each module, we adopted recurrent neural network (RNN), which has proven to be highly effective for learning nonlinear representations from sequential data, for the analysis of the temporal dynamics of fMRI activity patterns. Finally, we integrated the two modules into a complete framework. Main results. The proposed framework shows promising decoding performance. The average decoding accuracy across five subjects was over 19 times the chance level. Moreover, we compared the decoding performance of the early visual cortex (eVC) and the high-level visual cortex (hVC). The comparison results indicated that both eVC and hVC participated in processing visual stimuli, but the semantic information of the visual stimuli was mainly represented in hVC. Significance. The proposed framework advances the decoding of visual experiences and facilitates a better understanding of our visual functions.