Spatiotemporal feature extraction method combining common spatial pattern and hybrid spatial-temporal attention-Net in motor imagery–brain-computer interface system

Abstract

In recent years, motor imagery (MI) based on brain-computer interface (BCI) has gained the attention of researchers and been widely used in many fields, such as medical rehabilitation and entertainment. The temporal and spatial features of electroencephalography (EEG) are very important for MI-BCI classification. This study proposed a better spatiotemporal feature extraction method by combining the common spatial pattern (CSP) with a hybrid spatiotemporal attention convolutional neural network (HSTA-Net) model. The method first intercepts the MI-EEG into multiple time windows and projects them into the feature space and feeds into the HSTA-Net in parallel. Subsequently, the scaled dot product attention module and multi-head attention module are used to extract spatial feature attention. Finally, the features in multiple time windows obtained through HSTA-Net are integrated and used for classification. The proposed method achieves a classification accuracy of 77.3% on the BCI Competition IV2a dataset. The experimental results show that the proposed model has better classification performance. The CSP-HSTA-Net model can quickly adapt to the distribution of MI-EEG data during the training process, which increases the interclass distance of the extracted features and decreases its intraclass distance; it can better accomplish the MI-BCI classification task.

Keywords

brain-computer interface / common spatial pattern / HSTA-Net / motor imagery

References

[1] Wolpaw JR, Birbaumer N, Heetderks WJ, et al. Brain-computer interface technology: a review of the first international meeting. IEEE Trans Rehabil Eng. 2000; 8(2): 164-173.

[2] Nasir TB, Lalin MAM, Niaz K, et al. Design and implementation of EEG based home appliance control system. In: 2021 International Conference on Information and Communication Technology for Sustainable Development (ICICT4SD). IEEE; 2021: 445-449.

[3] Mane R, Chouhan T, Guan C. BCI for stroke rehabilitation: motor and beyond. J Neural Eng. 2020; 17(4): 041001.

[4] Menon C, de Negueruela C, Millán JR, et al. Prospects of brain–machine interfaces for space system control. Acta Astronaut. 2009; 64(4): 448-456.

[5] Kohli V, Tripathi U, Chamola V, Rout BK, Kanhere SS. A review on virtual reality and augmented reality use-cases of brain computer interface based applications for smart cities. Microprocess Microsyst. 2022; 88: 104392.

[6] Zisk AH, Borgheai SB, McLinden J, Deligani RJ, Shahriari Y. Improving longitudinal P300-BCI performance for people with ALS using a data augmentation and jitter correction approach. Brain Comput Interfac. 2022; 9(1): 49-66.

[7] Chen J, Zhang Y, Pan Y, Xu P, Guan C. A transformer-based deep neural network model for SSVEP classification. Neural Network. 2023; 164: 521-534.

[8] Kleinnijenhuis M, Arns M, Spronk D, Breteler R, Duysens J. Comparison of discrete-trial-based SMR and SCP training and the interrelationship between SCP and SMR networks: implications for brain–computer interfaces and neurofeedback. J Neurother. 2008; 11(4): 19-35.

[9] Zhao X, Jin J, Xu R, et al. A regional smoothing block sparse Bayesian learning method with temporal correlation for channel selection in P300 speller. Front Hum Neurosci. 2022; 16: 875851.

[10] Lazcano-Herrera AG, Fuentes-Aguilar RQ. Chairez I, Alonso-Valerdi LM. Gonzalez-Mendoza M, Alfaro-Ponce M. Review on BCI virtual rehabilitation and remote technology based on EEG for assistive devices. Appl Sci. 2022; 12(23): 12253.

[11] Van Erp J, Lotte F, Tangermann M. Brain-computer interfaces: beyond medical applications. Computer. 2012; 45(4): 26-34.

[12] Zuo C, Jin J, Yin E, et al. Novel hybrid brain–computer interface system based on motor imagery and P300. Cognit Neurodynamics. 2020; 14(2): 253-265.

[13] Pfurtscheller G, Brunner C, Schlögl A, Lopes da Silva F. Mu rhythm (de) synchronization and EEG single-trial classification of different motor imagery tasks. Neuroimage. 2006; 31(1): 153-159.

[14] Haumann NT, Parkkonen L, Kliuchko M, Vuust P, Brattico E. Comparing the performance of popular MEG/EEG artifact correction methods in an evoked-response study. Comput Intell Neurosci. 2016; 2016: 7489108.

[15] Sun G, Hu J, Wu G. A novel frequency band selection method for common spatial pattern in motor imagery based brain computer interface In: The 2010 International Joint Conference on Neural Networks (IJCNN). IEEE; 2010: 1-6.

[16] Miao Y, Jin J, Daly I, et al. Learning common time-frequency-spatial patterns for motor imagery classification. IEEE Trans Neural Syst Rehabil Eng. 2021; 29: 699-707.

[17] Lawhern VJ, Solon AJ, Waytowich NR, Gordon SM, Hung CP, Lance BJ. EEGNet: a compact convolutional neural network for EEG-based brain–computer interfaces. J Neural Eng. 2018; 15(5): 056013.

[18] Dai G, Zhou J, Huang J, Wang N. HS-CNN: a CNN with hybrid convolution scale for EEG motor imagery classification. J Neural Eng. 2020; 17(1): 016025.

[19] Müller-Putz GR, Schwarz A, Pereira J, et al. From classic motor imagery to complex movement intention decoding: the noninvasive Graz-BCI approach. Prog Brain Res. 2016; 228: 39-70.

[20] Chin ZY, Ang KK, Wang C, et al. Multi-class filter bank common spatial pattern for four-class motor imagery BCI. In: 2009 Annual International Conference of the IEEE Engineering in Medicine and Biology Society. IEEE; 2009: 571-574.

[21] DeVore GR. Computing the Z score and centiles for cross-sectional analysis: a practical approach. J Ultrasound Med. 2017; 36(3): 459-473.

[22] Vaswani A, Shazeer N, Parmar N, et al. Attention is all you need. Adv Neural Inf Process Syst. 2017; 30: 1-11.

[23] Xie J, Zhang J, Sun J, et al. A transformer-based approach combining deep learning network and spatial-temporal information for raw EEG classification. IEEE Trans Neural Syst Rehabil Eng. 2022; 30: 2126-2136.

[24] Gordon-Rodriguez E, Loaiza-Ganem G. Pleiss G, et al. Uses and abuses of the cross-entropy loss: case studies in modern deep learning. 2020; 137: 1-10.

[25] Mammone N, Ieracitano C, Morabito FC. A deep CNN approach to decode motor preparation of upper limbs from time–frequency maps of EEG signals at source level. Neural Network. 2020; 124: 357-372.

[26] Tangermann M, Müller KR, Aertsen A, et al. Review of the BCI competition IV. Front Neurosci. 2012; 6: 55.

[27] Ang KK, Chin ZY, Wang C, Guan C, Zhang H. Filter bank common spatial pattern algorithm on BCI competition IV datasets 2a and 2b. Front Neurosci. 2012; 6: 39.

[28] Riyad M, Khalil M, Adib A. MI-EEGNET: a novel convolutional neural network for motor imagery classification. J Neurosci Methods. 2021; 353: 109037.

[29] Schirrmeister RT, Springenberg JT, Fiederer LDJ, et al. Deep learning with convolutional neural networks for EEG decoding and visualization. Hum Brain Mapp. 2017; 38(11): 5391-5420.

[30] Jiao Y, Zheng Q, Qiao D, Lang X, Xie L, Pan Y. EEG rhythm separation and time–frequency analysis of fast multivariate empirical mode decomposition for motor imagery BCI. Biol Cybern. 2024; 118(1): 21-37.

[31] Echtioui A, Zouch W, Ghorbel M, et al. Classification of BCI multiclass motor imagery task based on artificial neural network. Clin EEG Neurosci. 2024; 55(4): 455-464.

[32] Echtioui A, Zouch W, Ghorbel M, Mhiri C. Convolutional neural network with support vector machine for motor imagery EEG signal classification. Multimed Tool Appl. 2023; 82(29): 45891-45911.

RIGHTS & PERMISSIONS

2024 The Author(s). Journal of Intelligent Medicine published by John Wiley & Sons Australia, Ltd on behalf of Tianjin University.