Decoding natural scenes based on sounds of objects within scenes using multivariate pattern analysis. (November 2019)

• Record Type: Journal Article
• Authors: Wang, Xiaojing; Gu, Jin; Xu, Junhai; Li, Xianglin; Geng, Junzu; Wang, Bin; Liu, Baolin
• Is Part Of: Neuroscience research. Volume 148(2019)
• Page Start: 9
• Page End: 18
• Publication Date: 2019-11
• Subjects: Cross modality -- Scene decoding -- Functional connectivity -- Multivariate pattern analysis -- fMRI; Neurosciences -- Research -- Periodicals; Neurosciences -- Research -- Japan -- Periodicals; Neurology -- Periodicals; Neurosciences -- Periodicals; Neurosciences -- Recherche -- Périodiques; Neurosciences -- Recherche -- Japon -- Périodiques; Neurosciences -- Research; Japan; Periodicals; 612.8
• DOI: 10.1016/j.neures.2018.11.009
• Languages: English
• ISSNs: 0168-0102

Abstract

Highlights: Scenes could be decoded based on the sound patterns of objects within the scenes. One semantic relationship between sounds and scenes was represented in the occipitotemporal cortex. Lateral occipital area was not sensitive to the openness of the scenes. A distinct network was involved in processing scenes and sounds. Abstract: Scene recognition plays an important role in spatial navigation and scene classification. It remains unknown whether the occipitotemporal cortex could represent the semantic association between the scenes and sounds of objects within the scenes. In this study, we used the functional magnetic resonance imaging (fMRI) technique and multivariate pattern analysis to assess whether diff ; ;erent scenes could be discriminated based on the patterns evoked by sounds of objects within the scenes. We found that patterns evoked by scenes could be predicted with patterns evoked by sounds of objects within the scenes in the posterior fusiform area (pF), lateral occipital area (LO) and superior temporal sulcus (STS). The further functional connectivity analysis suggested significant correlations between pF, LO and parahippocampal place area (PPA) except that between STS and other three regions under the scene and sound conditions. A distinct network in processing scenes and sounds was discovered using a seed-to-voxel analysis with STS as the seed. This study may provide a cross-modal channel of scene decoding through the sounds of objects within the scenes in the occipitotemporal cortex, which could complement the single-modal channel of scene decoding based on the global scene properties or objects within the scenes.