Machine-learning applications for differentiation across states/stages of creative thinking based on time-series and time-frequency features of EEG/ERP signals

Abstract

BACKGROUND: The study presents machine-learning (ML) classification approaches for the state/stage differentiation of creative tasks using the “test-control” approach. The control tasks were considered as the initial stages of the creative activity. Time-series and time-frequency electroencephalography (EEG) data analyses were employed in three divergent thinking tasks: 1) creating endings to well-known proverbs (“PROVERBS”, event-related potential [ERP] paradigm); 2) creating stories (“STORIES”, continuous EEG); 3) free creative painting (“viART”, continuous EEG).

AIM: To compare and select effective ML classification approaches for EEG signal separation at different stages or states of creative task performance.

METHODS: In this study, 22 individuals participated in the “PROVERBS” (ERP paradigm), 15 in the “STORIES”, and 1 (a longitudinal case study) in the “viART” tasks. Linear and convolutional neural network (CNN) classifiers were used. EEG data were previous artifacts corrected and converted to current source density (CSD). Continuous EEGs were divided into 4-s intervals and 1500 ms after stimulus presentation, were used in ERPs. The EEG/ERP time-frequency maps (Morlet wavelet transformation) for 3–30 Hz were generated for 4-s intervals with 100 ms shift (continuous EEGs in “STORIES” and “viART”) or for 1500 ms after stimulus presentation (ERPs in “PROVERBS”) and consisted of combined images (224×224 px) for frontal (Fz) and parietal (Pz) brain zones. Image classification was carried out using the modified CNN (ResNet50, ResNet18 architectures).

RESULTS: The offline classification accuracy of the four-class system (description of a picture, inventing a story plot, continuation of story’s plot, and background with open eyes) in the “STORY” creation task was up to 96.4% [±8.3 SD] with ResNet architectures (ResNet50 and ResNet18). The accuracy of the three states discrimination of the artists’ creative painting (resting state with open eyes, painting on canvas, and viewing the painting) was 86.94% for kernel naive bayes and 98.2% for CNN. For the trained and tested samples given for the CNN in consecutive order (neurointerface mode), the accuracy diminished to 70.0% [11% SD] on average. In the ERP paradigm “PROVERBS”, the classification accuracy of the three-class system (creation of “new” ending, naming of semantic synonym, and remembering of the known ending) was 80.5% [±8.7 SD] for the common spatial pattern, followed by rSVM (radial kernel basis support vector machine), compared with 43.2% [±8.8 SD] for CNN.

CONCLUSION: The use of CNNs allowed better classifying of “continuous” long-term states of creative activity. In fast “transient processes” such as ERP, time-series classifiers with spatial filtering proved to be more efficient.

Keywords

neural networks / creativity / artistic creativity / supervised machine learning / electroencephalography / EEG / event-related potential / ERP / time-frequency analysis

References

[1] Mane R, Chouhan T, Guan C. BCI for stroke rehabilitation: motor and beyond. J Neural Eng. 2020;17(4):041001. doi: 10.1088/1741-2552/aba162

[2] Mane R., Chouhan T., Guan C. BCI for stroke rehabilitation: motor and beyond // J Neural Eng. 2020. Vol. 17, N 4. P. 041001. doi: 10.1088/1741-2552/aba162

[3] Tayebi H, Azadnajafabad S, Maroufi SF, et al. Applications of brain-computer interfaces in neurodegenerative diseases. Neurosurg Rev. 2023;46(1):131. doi: 10.1007/s10143-023-02038-9

[4] Tayebi H., Azadnajafabad S., Maroufi S.F., et al. Applications of brain-computer interfaces in neurodegenerative diseases // Neurosurg Rev. 2023. Vol. 46, N 1. P. 131. doi: 10.1007/s10143-023-02038-9

[5] Lazcano-Herrera AG, Fuentes-Aguilar RQ, Chairez I, et al. Review on BCI virtual rehabilitation and remote technology based on EEG for assistive devices. Appl Sci. 2022;12(23):12253. doi: 10.3390/app122312253

[6] Lazcano-Herrera A.G., Fuentes-Aguilar R.Q., Chairez I., et al. Review on BCI virtual rehabilitation and remote technology based on EEG for assistive devices // Appl Sci. 2022. Vol. 12, N 23. P. 12253. doi: 10.3390/app122312253

[7] Lotte F, Bougrain L, Cichocki A, et al. A review of classification algorithms for EEG-based brain-computer interfaces: a 10 year update. J Neural Eng. 2018;15(3):031005. doi: 10.1088/1741-2552/aab2f2

[8] Lotte F., Bougrain L., Cichocki A., et al. A review of classification algorithms for EEG-based brain-computer interfaces: a 10 year update // J Neural Eng. 2018. Vol. 15, N 3. P. 031005. doi: 10.1088/1741-2552/aab2f2

[9] Maheshwari D, Ghosh SK, Tripathy RK, et al. Automated accurate emotion recognition system using rhythm-specific deep convolutional neural network technique with multi-channel EEG signals. Comput Biol Med. 2021;134:104428. doi: 10.1016/j.compbiomed.2021.104428

[10] Maheshwari D., Ghosh S.K., Tripathy R.K., et al. Automated accurate emotion recognition system using rhythm-specific deep convolutional neural network technique with multi-channel EEG signals // Comput Biol Med. 2021. Vol. 134. P. 104428. doi: 10.1016/j.compbiomed.2021.104428

[11] Wang Y, Wang S, Xu M. Landscape perception identification and classification based on electroencephalogram (EEG) features. Int J Environ Res Public Health. 2022;19(2):629. doi: 10.3390/ijerph19020629

[12] Wang Y., Wang S., Xu M. Landscape perception identification and classification based on electroencephalogram (EEG) features // Int J Environ Res Public Health. 2022. Vol. 19, N 2. P. 629. doi: 10.3390/ijerph19020629

[13] Stevens CE Jr, Zabelina DL. Classifying creativity: applying machine learning techniques to divergent thinking EEG data. Neuroimage. 2020;219:116990. doi: 10.1016/j.neuroimage.2020.116990

[14] Stevens C.E. Jr, Zabelina D.L. Classifying creativity: applying machine learning techniques to divergent thinking EEG data // Neuroimage. 2020. Vol. 219. P. 116990. doi: 10.1016/j.neuroimage.2020.116990

[15] Shemyakina NV, Nagornova ZhV. Does the instruction “be original and create” actually affect the EEG correlates of performing creative tasks? Human Physiology. 2020;46(6):587–596. doi: 10.1134/S0362119720060092

[16] Shemyakina N.V., Nagornova Zh.V. Does the instruction “be original and create” actually affect the EEG correlates of performing creative tasks? // Физиология человека. 2020. Т. 46, № 6. С. 5–15. doi:10.1134/S0362119720060092

[17] Shemyakina NV, Potapov YG, Nagornova ZhV. Dynamics of the EEG frequency structure during sketching in ecological conditions and non-verbal tasks fulfillment by a professional artist: case study. Human Physiology. 2022;48(5):506–515. doi: 10.1134/S0362119722700050

[18] Шемякина Н.В., Потапов Ю.Г., Нагорнова Ж.В. Динамика частотной структуры ЭЭГ во время эскизирования в экологических условиях и выполнения невербальных творческих задач профессиональным художником: лонгитюдное case study // Физиология человека. 2022. Т. 48, № 5. С. 26–37. doi: 10.1134/S0362119722700050

[19] Finke RA, Smith SM, Ward TB. Creative cognition: theory, research, and applications. The MIT Press: Cambridge, Massachusetts; 1996.

[20] Finke R.A., Smith S.M., Ward T.B. Creative cognition: theory, research, and applications. Cambridge, Massachusetts: The MIT Press, 1996.

[21] Jia W, Zeng Y. EEG signals respond differently to idea generation, idea evolution and evaluation in a loosely controlled creativity experiment. Sci Rep. 2021;11(1):2119. doi: 10.1038/s41598-021-81655-0

[22] Jia W., Zeng Y. EEG signals respond differently to idea generation, idea evolution and evaluation in a loosely controlled creativity experiment // Sci Rep. 2021. Vol. 11, N 1. P. 2119. doi: 10.1038/s41598-021-81655-0

[23] Vanutelli ME, Salvadore M, Lucchiari C. BCI applications to creativity: review and future directions, from little-c to C2. Brain Sci. 2023;13(4):665. doi: 10.3390/brainsci13040665

[24] Vanutelli M.E., Salvadore M., Lucchiari C. BCI applications to creativity: review and future directions, from little-c to C2 // Brain Sci. 2023. Vol. 13, N 4. P. 665. doi: 10.3390/brainsci13040665

[25] Appriou A, Cichocki A, Lotte F. Modern machine learning algorithms to classify cognitive and affective states from electroencephalography signals. IEEE Systems, Man, and Cybernetics Magazine. 2020;6(3):29–38. doi: 10.1109/MSMC.2020.2968638

[26] Appriou A., Cichocki A., Lotte F. Modern machine learning algorithms to classify cognitive and affective states from electroencephalography signals // IEEE Systems, Man, and Cybernetics Magazine. 2020. Vol. 6, N 3. P. 29–38. doi: 10.1109/MSMC.2020.2968638

[27] Shemyakina NV, Nagornova ZhV. EEG “signs” of verbal creative task fulfillment with and without overcoming self-induced stereotypes. Behav Sci (Basel). 2019;10(1):17. doi: 10.3390/bs10010017

[28] Shemyakina N.V., Nagornova Zh.V. EEG “signs” of verbal creative task fulfillment with and without overcoming self-induced stereotypes // Behav Sci (Basel). 2019. Vol. 10, N 1. P. 17. doi: 10.3390/bs10010017

[29] O’Sullivan M, Guilford JP. Four factor tests of social intelligence (behavioral cognition): manual of instructions and interpretations. Sheridan Psychological Service, Inc.: Orange, CA, USA; 1976.

[30] O’Sullivan M., Guilford J.P. Four factor tests of social intelligence (behavioral cognition): manual of instructions and interpretations. Orange, CA, USA : Sheridan Psychological Service, Inc., 1976.

[31] Shemyakina NV, Danko SG, Nagornova ZhV, et al. Changes in the power and coherence spectra of the EEG rhythmic components during solution of a verbal creative task of overcoming a stereotype. Human Physiology. 2007;33(5):524–530. doi: 10.1134/S0362119707050027

[32] Шемякина Н.В., Данько С.Г., Нагорнова Ж.В., и др. Динамика спектров мощности и когерентности ритмических компонентов ЭЭГ при решении вербальной творческой задачи преодоления стереотипа // Физиология человека. 2007. Т. 33, № 5. С. 14–21. doi: 10.1134/S0362119707050027

[33] Raven J, Raven JC, Court JH. Manual for raven’s progressive matrices and vocabulary scales. Section 3: the standard progressive matrices. Harcourt Assessment: San Antonio, TX, USA; 2004.

[34] Raven J., Raven J.C., Court J.H. Manual for raven’s progressive matrices and vocabulary scales. Section 3: the standard progressive matrices. San Antonio, TX, USA : Harcourt Assessment, 2004.

[35] Vigário RN. Extraction of ocular artefacts from EEG using independent component analysis. Electroencephalogr Clin Neurophysiol. 1997;103(3):395–404. doi: 10.1016/s0013-4694(97)00042-8

[36] Vigário R.N. Extraction of ocular artefacts from EEG using independent component analysis // Electroencephalogr Clin Neurophysiol. 1997. Vol. 103, N 3. P. 395–404. doi: 10.1016/s0013-4694(97)00042-8

[37] Jung TP, Makeig S, Humphries C, et al. Removing electroencephalographic artifacts by blind source separation. Psychophysiology. 2000;37(2):163–178.

[38] Jung T.P., Makeig S., Humphries C., et al. Removing electroencephalographic artifacts by blind source separation // Psychophysiology. 2000. Vol. 37, N 2. P. 163–178.

[39] Tereshchenko EP, Ponomarev VA, Kropotov YD, Müller A. Comparative efficiencies of different methods for removing blink artifacts in analyzing quantitative electroencephalogram and event-related potentials. Human Physiology. 2009;35(2):241–247. doi: 10.1134/S0362119709020157

[40] Терещенко Е.П., Пономарев В.А., Кропотов Ю.Д., Мюллер А. Сравнение эффективности различных методов удаления артефактов морганий при анализе количественной электроэнцефалограммы и вызванных потенциалов // Физиология человека. 2009. Т. 35, № 2. С. 124–131. doi: 10.1134/S0362119709020157

[41] Perrin F, Pernier J, Bertrand O, Echallier JF. Spherical splines for scalp potential and current density mapping. Electroencephalogr Clin Neurophysiol. 1989;72(2):184–187. doi: 10.1016/0013-4694(89)90180-6

[42] Perrin F., Pernier J., Bertrand O., Echallier J.F. Spherical splines for scalp potential and current density mapping // Electroencephalogr Clin Neurophysiol. 1989. Vol. 72, N 2. P. 184–187. doi: 10.1016/0013-4694(89)90180-6

[43] Tenke CE, Kayser J. Generator localization by current source density (CSD): implications of volume conduction and field closure at intracranial and scalp resolutions. Clin Neurophysiol. 2012;123(12):2328–2345. doi: 10.1016/j.clinph.2012.06.005

[44] Tenke C.E., Kayser J. Generator localization by current source density (CSD): implications of volume conduction and field closure at intracranial and scalp resolutions // Clin Neurophysiol. 2012. Vol. 123, N 12. P. 2328–2345. doi: 10.1016/j.clinph.2012.06.005

[45] Kayser J, Tenke CE. Issues and considerations for using the scalp surface Laplacian in EEG/ERP research: a tutorial review. Int J Psychophysiol. 2015;97(3):189–209. doi: 10.1016/j.ijpsycho.2015.04.012

[46] Kayser J., Tenke C.E. Issues and considerations for using the scalp surface Laplacian in EEG/ERP research: a tutorial review // Int J Psychophysiol. 2015. Vol. 97, N 3. P. 189–209. doi: 10.1016/j.ijpsycho.2015.04.012

[47] Ponomarev VA, Mueller A, Candrian G, et al. Group independent component analysis (gICA) and current source density (CSD) in the study of EEG in ADHD adults. Clin Neurophysiol. 2014;125(1):83–97. doi: 10.1016/j.clinph.2013.06.015

[48] Ponomarev V.A., Mueller A., Candrian G., et al. Group independent component analysis (gICA) and current source density (CSD) in the study of EEG in ADHD adults // Clin Neurophysiol. 2014. Vol. 125, N 1. P. 83–97. doi: 10.1016/j.clinph.2013.06.015

[49] Lilly JM, Olhede SC. Generalized Morse wavelets as a superfamily of analytic wavelets. IEEE Trans Signal Process. 2012;60(11):6036–6041. doi: 10.1109/TSP.2012.2210890

[50] Lilly J.M., Olhede S.C. Generalized Morse wavelets as a superfamily of analytic wavelets // IEEE Trans Signal Process. 2012. Vol. 60, N 11. P. 6036–6041. doi: 10.1109/TSP.2012.2210890

[51] Koles ZJ, Lazar MS, Zhou SZ. Spatial patterns underlying population differences in the background EEG. Brain Topogr. 1990;2(4):275–284. doi: 10.1007/BF01129656

[52] Koles Z.J., Lazar M.S., Zhou S.Z. Spatial patterns underlying population differences in the background EEG // Brain Topogr. 1990. Vol. 2, N 4. P. 275–284. doi: 10.1007/BF01129656

[53] Blankertz B, Tomioka R, Lemm S, et al. Optimizing spatial filters for robust EEG single-trial analysis. IEEE Signal Processing Magazine. 2008;25(1):41–56. doi: 10.1109/MSP.2008.4408441

[54] Blankertz B, Tomioka R, Lemm S, et al. Optimizing spatial filters for robust EEG single-trial analysis // IEEE Signal Processing Magazine. 2008. Vol. 25, N 1. P. 41–56. doi: 10.1109/MSP.2008.4408441

[55] Grosse-Wentrup M, Buss M. Multiclass common spatial patterns and information theoretic feature extraction. IEEE Trans Biomed Eng. 2008;55(8):1991–2000. doi: 10.1109/TBME.2008.921154

[56] Grosse-Wentrup M., Buss M. Multiclass common spatial patterns and information theoretic feature extraction // IEEE Trans Biomed Eng. 2008. Vol. 55, N 8. P. 1991–2000. doi: 10.1109/TBME.2008.921154

[57] Fang Y, Yang H, Zhang X, et al. Multi-feature input deep forest for EEG-based emotion recognition. Front Neurorobot. 2021;14:617531. doi: 10.3389/fnbot.2020.617531

[58] Fang Y., Yang H., Zhang X., et al. Multi-feature input deep forest for EEG-based emotion recognition // Front Neurorobot. 2021. Vol. 14. P. 617531. doi: 10.3389/fnbot.2020.617531

[59] Cheng J, Chen M, Li C, et al. Emotion recognition from multi-channel EEG via deep forest. IEEE J Biomed Health Inform. 2021;25(2):453–464. doi: 10.1109/JBHI.2020.2995767

[60] Cheng J., Chen M., Li C., et al. Emotion recognition from multi-channel EEG via deep forest // IEEE J Biomed Health Inform. 2021. Vol. 25, N 2. P. 453–464. doi: 10.1109/JBHI.2020.2995767

[61] Wang F, Wu S, Zhang W, et al. Emotion recognition with convolutional neural network and EEG-based EFDMs. Neuropsychologia. 2020;146:107506. doi: 10.1016/j.neuropsychologia.2020.107506

[62] Wang F., Wu S., Zhang W., et al. Emotion recognition with convolutional neural network and EEG-based EFDMs // Neuropsychologia. 2020. Vol. 146. P. 107506. doi: 10.1016/j.neuropsychologia.2020.107506

[63] Kim S, Kim TS, Lee WH. Accelerating 3D convolutional neural network with channel bottleneck module for EEG-based emotion recognition. Sensors (Basel). 2022;22(18):6813. doi: 10.3390/s22186813

[64] Kim S., Kim T.S., Lee W.H. Accelerating 3D convolutional neural network with channel bottleneck module for EEG-based emotion recognition // Sensors (Basel). 2022. Vol. 22, N 18. P. 6813. doi: 10.3390/s22186813

[65] Sasaki M, Iversen J, Callan DE. Music improvisation is characterized by increase EEG spectral power in prefrontal and perceptual motor cortical sources and can be reliably classified from non-improvisatory performance. Front Hum Neurosci. 2019;13:435. doi: 10.3389/fnhum.2019.00435

[66] Sasaki M., Iversen J., Callan D.E. Music improvisation is characterized by increase EEG spectral power in prefrontal and perceptual motor cortical sources and can be reliably classified from non-improvisatory performance // Front Hum Neurosci. 2019. Vol. 13. P. 435. doi: 10.3389/fnhum.2019.00435

[67] Tian Y, Zhang H, Pang Y, Lin J. Classification for single-trial N170 during responding to facial picture with emotion. Front Comput Neurosci. 2018;12:68. doi: 10.3389/fncom.2018.00068

[68] Tian Y., Zhang H., Pang Y., Lin J. Classification for single-trial N170 during responding to facial picture with emotion // Front Comput Neurosci. 2018. Vol. 12. P. 68. doi: 10.3389/fncom.2018.00068

[69] Santamaria-Vazquez E, Martinez-Cagigal V, Vaquerizo-Villar F, Hornero R. EEG-inception: a novel deep convolutional neural network for assistive ERP-based brain-computer interfaces. IEEE Trans Neural Syst Rehabil Eng. 2020;28(12):2773–2782. doi: 10.1109/TNSRE.2020.3048106

[70] Santamaria-Vazquez E., Martinez-Cagigal V., Vaquerizo-Villar F., Hornero R. EEG-inception: a novel deep convolutional neural network for assistive ERP-based brain-computer interfaces // IEEE Trans Neural Syst Rehabil Eng. 2020. Vol. 28, N 12. P. 2773–2782. doi: 10.1109/TNSRE.2020.3048106

[71] Kumar A, Pirogova E, Mahmoud SS, Fang Q. Classification of error-related potentials evoked during stroke rehabilitation training. J Neural Eng. 2021;18(5):10.1088/1741-2552/ac1d32. doi: 10.1088/1741-2552/ac1d32

[72] Kumar A., Pirogova E., Mahmoud S.S., Fang Q. Classification of error-related potentials evoked during stroke rehabilitation training // J Neural Eng. 2021. Vol. 18, N 5. P. 10. doi: 10.1088/1741-2552/ac1d32

RIGHTS & PERMISSIONS

Eco-Vector