Vowel Sound Synthesis from Electroencephalography during Listening and Recalling. (7th January 2021)

• Record Type: Journal Article
• Authors: Akashi, Wataru; Kambara, Hiroyuki; Ogata, Yousuke; Koike, Yasuharu; Minati, Ludovico; Yoshimura, Natsue
• Is Part Of: Advanced intelligent systems. Volume 3:Number 2(2021)
• Page Start: n/a
• Page End: n/a
• Publication Date: 2021-01-07
• Subjects: brain activity signals -- cortical current source estimations -- deep-learning -- electroencephalography -- speech syntheses; Artificial intelligence -- Periodicals; Robotics -- Periodicals; Control theory -- Periodicals; 006.3
• DOI: 10.1002/aisy.202000164
• Languages: English
• ISSNs: 2640-4567

Abstract

Abstract : Recent advances in brain imaging technology have furthered our knowledge of the neural basis of auditory and speech processing, often via contributions from invasive brain signal recording and stimulation studies conducted intraoperatively. Herein, an approach for synthesizing vowel sounds straightforwardly from scalp‐recorded electroencephalography (EEG), a noninvasive neurophysiological recording method is demonstrated. Given cortical current signals derived from the EEG acquired while human participants listen to and recall (i.e., imagined) two vowels, /a/ and /i/, sound parameters are estimated by a convolutional neural network (CNN). The speech synthesized from the estimated parameters is sufficiently natural to achieve recognition rates >85% during a subsequent sound discrimination task. Notably, the CNN identifies the involvement of the brain areas mediating the "what" auditory stream, namely the superior, middle temporal, and Heschl's gyri, demonstrating the efficacy of the computational method in extracting auditory‐related information from neuroelectrical activity. Differences in cortical sound representation between listening versus recalling are further revealed, such that the fusiform, calcarine, and anterior cingulate gyri contributes during listening, whereas the inferior occipital gyrus is engaged during recollection. The proposed approach can expand the scope of EEG in decoding auditory perception that requires high spatial and temporal resolution. Abstract : Vowel sounds during listening and recalling are synthesized from brain activity signals (electroencephalography [EEG]) with remarkably high sound recognition rates >85%. The synthesizer constructed by cortical current source estimation and deep‐learning uses signals from physiologically plausible brain‐areas consisting of a ventral auditory pathway, further revealing area differences between the listening and recalling processes. This approach considerably expands applications of EEG.