Reconstructing natural images from human fMRI by alternating encoding and decoding with shared autoencoder regularization. (March 2022)

• Record Type: Journal Article
• Authors: Qiao, Kai; Chen, Jian; Wang, Linyuan; Zhang, Chi; Tong, Li; Yan, Bin
• Is Part Of: Biomedical signal processing and control. Volume 73(2022)
• Publication Date: 2022-03
• Subjects: fMRI -- Visual reconstruction -- Visual encoding -- Shared autoencoder -- Alternating optimization; Signal processing -- Periodicals; Biomedical engineering -- Periodicals; Signal Processing, Computer-Assisted -- Periodicals; Image Processing, Computer-Assisted -- Periodicals; Biomedical Engineering -- Periodicals; 610.28
• DOI: 10.1016/j.bspc.2021.103397
• Languages: English
• ISSNs: 1746-8094

Abstract

Highlights: We proposed the mutual promotion of visual encoding and reconstruction models. We designed alternating optimization bassed on seim-supervised learning. We devised inter-sample differentiated representations to augment small dataset. Our proposed model achieved state-of-the-art reconstruction performance. Abstract: Reconstructing the viewed natural images from the corresponding functional magnetic resonance imaging (fMRI) of human visual cortices is extremely difficult. Utilizing deep learning techniques, the quality of reconstructed images can be significantly improved. This however is subject to availability of sufficient number of pair samples, which is not currently the case. In this study, we propose to perform alternating encoding and decoding to realize the mutual promotion of both based on shared semi-supervised learning and accomplish better reconstruction of natural images from the corresponding fMRI voxels. In our proposed method, the encoder and decoder are used to respectively map the images to the fMRI voxels (visual encoding), and the fMRI voxels to the images (visual reconstruction). We argue that combining the encoder and decoder in different sequences can form two converse and shared autoencoders to regularize the supervised learning of both through the unsupervised learning of fMRI voxels and images. More importantly, we alternatingly train the encoder and decoder with shared autoencoder regularization. Here, the premise is that a better encoder can produce a better decoder, and vice versa. The pixel-level identification of the proposed method achieves up to 89.5%, which indicates at least 2% superiority and is the new state-of-the-art compared to the previous works in terms of the image reconstruction performance.