A deep learning approach for decoding visually imagined digits and letters using time–frequency–spatial representation of EEG signals. (1st October 2022)

• Record Type: Journal Article
• Authors: Alazrai, Rami; Abuhijleh, Motaz; Ali, Mostafa Z.; Daoud, Mohammad I.
• Is Part Of: Expert systems with applications. Volume 203(2022)
• Publication Date: 2022-10-01
• Subjects: Electroencephalography -- Time–frequency–spatial representation -- Visual imagery -- Deep learning -- Convolutional neural networks -- Brain–computer interface; Expert systems (Computer science) -- Periodicals; Systèmes experts (Informatique) -- Périodiques; Electronic journals; 006.33
• DOI: 10.1016/j.eswa.2022.117417
• Languages: English
• ISSNs: 0957-4174

Abstract

Abstract: The recent advances in developing assistive devices have attracted researchers to use visual imagery (VI) mental tasks as a control paradigm to design brain–computer interfaces that can produce a large number of control signals. Consequently, this can facilitate the design of control mechanisms that allow locked-in individuals to interact with the surrounding world. This paper presents a two-phase approach for decoding visually imagined digits and letters using electroencephalography (EEG) signals. The first phase employs the Choi–Williams time–frequency distribution (CWD) to construct a joint time, frequency, and spatial (TFS) representation of the EEG signals. The constructed joint TFS representation characterizes the variations in the energy encapsulated within the EEG signals over the TFS domains. The second phase presents a novel deep learning (DL) framework to automatically extract features from the constructed joint TFS representation of the EEG signals and decode the imagined digits and letters. The performance of our approach is assessed using an EEG dataset that was acquired for 16 healthy participants while imagining decimal digits and uppercase English letters. Our approach achieved an average ± standard deviation accuracy of 95 . 47 ± 2 . 3 %, which is significantly outperforming the accuracies obtained when the CWD is replaced with two alternative time–frequency analysis techniques, the accuracies obtained using four pre-trained DL models, and the accuracies obtained using CWD-based handcrafted features that are classified using four conventional classifiers. Moreover, the results of our proposed approach outperform those reported by several previous studies with regard to the accuracy and number of classes. Highlights: An EEG-based two-phase approach is proposed for decoding visually imagined objects. The approach focuses on decoding visually imagined digits and letters. The first phase builds a new time–frequency–spatial representation of EEG signals. The second phase presents a new deep learning framework to decode imagined objects. The results show that the approach enables accurate decoding of imagined objects.