Multimodal neural networks better explain multivoxel patterns in the hippocampus. (October 2022)

• Record Type: Journal Article
• Authors: Choksi, Bhavin; Mozafari, Milad; VanRullen, Rufin; Reddy, Leila
• Is Part Of: Neural networks. Volume 154(2022)
• Page Start: 538
• Page End: 542
• Publication Date: 2022-10
• Subjects: Multimodal networks -- Concept cells -- Hippocampus -- fMRI -- Deep learning -- Neuroscience; Neural computers -- Periodicals; Neural networks (Computer science) -- Periodicals; Neural networks (Neurobiology) -- Periodicals; Nervous System -- Periodicals; Ordinateurs neuronaux -- Périodiques; Réseaux neuronaux (Informatique) -- Périodiques; Réseaux neuronaux (Neurobiologie) -- Périodiques; Neural computers; Neural networks (Computer science); Neural networks (Neurobiology); Periodicals; 006.32
• DOI: 10.1016/j.neunet.2022.07.033
• Languages: English
• ISSNs: 0893-6080

Abstract

Abstract: The human hippocampus possesses "concept cells", neurons that fire when presented with stimuli belonging to a specific concept, regardless of the modality. Recently, similar concept cells were discovered in a multimodal network called CLIP (Radford et al., 2021). Here, we ask whether CLIP can explain the fMRI activity of the human hippocampus better than a purely visual (or linguistic) model. We extend our analysis to a range of publicly available uni- and multi-modal models. We demonstrate that "multimodality" stands out as a key component when assessing the ability of a network to explain the multivoxel activity in the hippocampus.