Spoken language-based automatic cognitive assessment of stroke survivors

Abstract

Stroke survivors (SSs) often experience cognitive decline following their initial stroke, necessitating repeat post-stroke cognitive assessments. Current methods of assessment, such as the pen-and-paper-based Montreal Cognitive Assessment (MoCA), is time-consuming and often reliant on seeing skilled clinicians in person. This is at a time when patients have a lot of often diverse rehabilitation needs. To address these challenges, our paper introduces the first system of its kind to be used for this cohort. CognoSpeak is an automated cognitive assessment system that people can use initially on the ward immediately post-stroke (baseline) and subsequently at home (follow-ups). CognoSpeak assesses cognitive decline by asking users to engage with a virtual agent by answering questions and completing clinically-motivated tasks and cognitive tests. The system then uses AI to extract and process speech, language, and interactional cues for cognitive decline. The system was originally developed for dementia; here, we show that it can successfully predict MoCA scores (regression) and identify cognitive decline predicated on a MoCA-based threshold (classification) in the stroke survivor cohort. We explore an extensive set of acoustic- and text-based features as well as different machine learning models. Leveraging a unique dataset of 55 SS CognoSpeak interactions, our findings show excellent performance for both regression and classification style prediction with the best regression result (Normalised Root Mean Squared Error (N-RMSE)) of 0.092. In addition, we show that direct classification of the MoCA score cutoff of 26 yields an F1-score of 0.74 (Specificity: 0.73, Sensitivity: 0.75) using a Logistic Regression Classifier. This demonstrates the first evidence of the system ’ s robustness and clinical potential.

Keywords

Speech Technology / Post-stroke Rehabilitation / Cognitive decline assessment

Declaration of competing interest

The authors declare the following financial interests/personal relationships which may be considered as potential competing interests: Bahman Mirheidari reports financial support was provided by The Rosetrees Trust and the Stoneygate Trust (COMPASS, Grant Agreement No. M934), and NIHR Academic Clinical Lectureship in Neurology (CL-2020-04-00). If there are other authors, they declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Data availability

The data that has been used is confidential.

Acknowledgements

This work is supported by the Rosetrees Trust and the Stoneygate Trust (COMPASS, Grant Agreement No. M934). An NIHR Academic Clinical Lectureship in Neurology CL-2020-04-004 NIHR supports SMB. This summarises independent research at the NIHR Sheffield Biomedical Research Centre(Translational Neuroscience)

References

[1] Baevski A., Zhou Y., Mohamed A., & Auli M. (2020). wav2vec 2.0: A framework for self-supervised learning of speech representations. Advances in Neural Information Processing Systems, 33, 12449-12460.

[2] Bandini, A., Green, J.R., Richburg, B., Yunusova, Y., (2018). Automatic detection of orofacial impairment in stroke.In: Interspeech, pp. 1711-1715.

[3] Becker J. T., Boiler F., Lopez O. L., Saxton J., & McGonigle K. L. (1994). The natural history of alzheimer ’ s disease: Description of study cohort and accuracy of diagnosis. Archives of Neurology, 51(6), 585-594.

[4] Blackburn D. J., Bafadhel L., Randall M., & Harkness K. A. (2013). Cognitive screening in the acute stroke setting. Age and Ageing, 42(1), 113-116.

[5] Chen C., Dong Y., Venketasubramanian N., Sharma V., Chan B., Teoh H., Seet R., Slavin M., Sachdev P.,Collinson, S., et al. (2011). A comparison of the baseline montreal cognitive assessment (moca) and the baseline mini-mental state examination (mmse) in predicting moderate to severe poststroke cognitive impairment. Cerebrovascular Diseases, 32, 44-45.

[6] Coen R. F., Robertson D. A., Kenny R. A., & King-Kallimanis B. L. (2016). Strengths and limitations of the MoCA for assessing cognitive functioning: Findings from a large representative sample of irish older adults. Journal of Geriatric Psychiatry and Neurology, 29(1), 18-24.

[7] Devlin, J., Chang, M.-W., Lee, K., Toutanova, K., (2018). Bert: Pre-training of deep bidirectional transformers for language understanding, arXiv preprint arXiv: 1810.04805.

[8] Dong Y., Sharma V. K., Chan, B. P.-L., Venketasubramanian N., Teoh H. L., Seet R. C. S., Tanicala S., Chan Y. H., & Chen C. (2010). The montreal cognitive assessment (moca) is superior to the mini-mental state examination (mmse) for the detection of vascular cognitive impairment after acute stroke. Journal of the Neurological Sciences, 299(1-2), 15-18.

[9] Elsey C., Drew P., Jones D., Blackburn D., Wakefield S., Harkness K., Venneri A., & Reuber M. (2015). Towards diagnostic conversational profiles of patients presenting with dementia or functional memory disorders to memory clinics. Patient Education and Counseling, 98, 1071-1077.

[10] Eyben, F., W ¨ ollmer, M., Schuller, B., (2010). Opensmile: The Munich versatile and fast open-source audio feature extractor, In:Proceedings of the 18th ACM International Conference on Multimedia, pp. 1459-1462.

[11] Eyben F., Scherer K. R., Schuller B. W., Sundberg J., André E., Busso C., Devillers L. Y., Epps J., Laukka P.,Narayanan, S. S., et al. (2015). The Geneva minimalistic acoustic parameter set (GeMAPS) for voice research and affective computing. IEEE Transactions on Affective Computing, 7(2), 190-202.

[12] Folstein M. F., Folstein S. E., & McHugh P. R. (1975). Mini-mental state: A practical method for grading the cognitive state of patients for the clinician. Journal of Psychiatric Research, 12(3), 189-198.

[13] Fu, Z., Haider, F., Luz, S., (2020). Predicting mini-mental status examination scores through paralinguistic acoustic features of spontaneous speech, In:Proceedings of the 2020 42nd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC). IEEE, 5548-5552.

[14] Godefroy O., Fickl A., Roussel M., Auribault C., Bugnicourt J. M., Lamy C., Canaple S., & Petitnicolas G. (2011). Is the montreal cognitive assessment superior to the mini-mental state examination to detect poststroke cognitive impairment? A study with neuropsychological evaluation. Stroke, 42(6), 1712-1716.

[15] Gosztolya G., Vincze V., Tóth L., Pákáski M., Kálmán J., & Hoffmann I. (2019). Identifying mild cognitive impairment and mild alzheimer ’ s disease based on spontaneous speech using asr and linguistic features. Computer Speech & Language, 53, 181-197.

[16] Kantithammakorn P., Punyabukkana P., Pratanwanich P. N., Hemrungrojn S., Chunharas C., & Wanvarie D. (2022). Using automatic speech recognition to assess Thai speech language fluency in the Montreal cognitive assessment (MoCA). Sensors, 22(4), 1583.

[17] Katz M. J., Wang C., Nester C. O., Derby C. A., Zimmerman M. E., Lipton R. B., Sliwinski M. J., & Rabin L. A. (2021). T-moca: A valid phone screen for cognitive impairment in diverse community samples. Alzheimer’s & Dementia: Diagnosis, Assessment & Disease Monitoring, 13(1), Article e12144.

[18] Laranjo L., Dunn A. G., Tong H. L., Kocaballi A. B., Chen J., Bashir R., Surian D., Gallego B., Magrabi F.,Lau, A. Y., et al. (2018). Conversational agents in healthcare: A systematic review. Journal of the American Medical Informatics Association, 25(9), 1248-1258.

[19] Liu J., Du X., Lu S., Zhang Y.-M., An-ming H., Ng M. L., Su R., Wang L., & Yan N. (2023). Audio-video database from subacute stroke patients for dysarthric speech intelligence assessment and preliminary analysis. Biomedical Signal Processing and Control, 79, Article 104161.

[20] Livingston G., Huntley J., Sommerlad A., Ames D., Ballard C., Banerjee S., Brayne C., Burns A., Cohen-Mansfield J.,Cooper, C., et al. (2020). Dementia prevention, intervention, and care: 2020 report of the lancet commission. The Lancet, 396(10248), 413-446.

[21] Luz, S., Haider, F., De la Fuente, S., Fromm, D., MacWhinney, B., (2020). Alzheimer ’ s dementia recognition through spontaneous speech: The ADReSS challenge, arXiv preprint arXiv:2004.06833, 2020.

[22] Luz, S., Haider, F., De La Fuente, S., Fromm, D., MacWhinney, B., (2021). Detecting cognitive decline using speech only: The ADReSSo challenge, arXiv preprint arXiv: 2104.09356, 2021.

[23] Mai L. M., Sposato L. A., Rothwell P. M., Hachinski V., & Pendlebury S. T. (2016). A comparison between the moca and the mmse visuoexecutive sub-tests in detecting abnormalities in tia/stroke patients. International Journal of Stroke, 11(4), 420-424.

[24] Manohar, V., Povey, D., Khudanpur, S., (2017). JHU Kaldi system for Arabic MGB-3 ASR challenge using diarization, audio-transcript alignment and transfer learning, In:Proceedings of the 2017 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU). IEEE, pp. 346-352.

[25] Mirheidari, B., Blackburn, D., Christensen, H., (2022). Automatic cognitive assessment: Combining sparse datasets with disparate cognitive scores, in Proc. Interspeech. ISCA, 2022.

[26] Mirheidari B., Pan Y., Blackburn D., O ’ Malley R., Christensen H., (2021). Identifying cognitive impairment using sentence representation vectors, Proc. Interspeech, pp. 2941-2945.

[27] Mirheidari B., Blackburn D., Harkness K., Walker T., Venneri A., Reuber M., & Christensen H. (2017). Toward the automation of diagnostic conversation analysis in patients with memory complaints. Journal of Alzheimer’s Disease, 1-15.

[28] Mirheidari, B., Blackburn, D., O ’ Malley, R., Walker, T., Venneri, A., Reuber, M., Christensen, H., (2019). Computational cognitive assessment: Investigating the use of an intelligent virtual agent for the detection of early signs of dementia, In: Proceedings of the ICASSP. IEEE, pp. 2732-2736.

[29] Mirheidari B., Blackburn D., O ’ Malley R., Venneri A., Walker T., Reuber M., & Christensen H. (2020). Improving cognitive impairment classification by generative neural network-based feature augmentation. Proc. Interspeech, 2527-2531.

[30] Mitchell A. J. (2017). The mini-mental state examination (MMSE):update on its diagnostic accuracy and clinical utility for cognitive disorders. Cognitive screening instruments (pp.37-48). Springer,.

[31] Nasreddine Z. S., Phillips N. A., Bédirian V., Charbonneau S., Whitehead V., Collin I., Cummings J. L., & Chertkow H. (2005). The Montreal cognitive assessment, MoCA: A brief screening tool for mild cognitive impairment. Journal of the American Geriatrics Society, 53(4), 695-699.

[32] Ostrand R., & Gunstad J. (2021). Using automatic assessment of speech production to predict current and future cognitive function in older adults. Journal of Geriatric Psychiatry and Neurology, 34(5), 357-369.

[33] Pan, Y., Mirheidari, B., Harris, J.M., Thompson, J.C., Jones, M., Snowden, J.S., Blackburn, D., Christensen, H., (2021). Using the outputs of different automatic speech recognition paradigms for acoustic-and bert-based alzheimer ’ s dementia detection through spontaneous speech, Proc. Interspeech, pp. 3810-3814.

[34] Pappagari R., Cho J., Joshi S., Moro-Velazquez L., Zelasko P., Villalba J., & Dehak N. (2021). Automatic detection and assessment of Alzheimer Disease using speech and language technologies in low-resource scenarios. Proc. Interspeech.

[35] Pedregosa F., & Varoquaux G. (2011). Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12, 2825-2830.

[36] Pendlebury S. T., Cuthbertson F. C., Welch S. J., Mehta Z., & Rothwell P. M. (2010). Underestimation of cognitive impairment by mini-mental state examination versus the montreal cognitive assessment in patients with transient ischemic attack and stroke: A population-based study. Stroke, 41(6), 1290-1293.

[37] Pendlebury S. T., Wadling S., Silver L. E., Mehta Z., & Rothwell P. M. (2011). Transient cognitive impairment in tia and minor stroke. Stroke, 42(11), 3116-3121.

[38] Pendlebury S. T., Mariz J., Bull L., Mehta Z., & Rothwell P. M. (2012). Moca, ace-r and mmse versus the ninds-csn vci harmonisation standards neuropsychological battery after tia and stroke. Stroke, 43(2), 464.

[39] Pennington, J., Socher, R., Manning, C., (2014). Glove:Global vectors for word representation, In: Proc. EMNLP, pp. 1532-1543.

[40] D. Povey, A. Ghoshal, G. Boulianne, L. Burget, O. Glembek, N. Goel, M. Hannemann, P. Motlicek, Y. Qian, P. Schwarz, J. Silovsky, G. Stemmer, and K. Vesely,The Kaldi speech recognition toolkit, In: Proceedings of the IEEE 2011 Workshop on Automatic Speech Recognition and Understanding, 2011.

[41] Riepe M. W., Riss S., Bittner D., & Huber R. (2003). Screening for cognitive impairment in patients with acute stroke. Dementia and Geriatric Cognitive Disorders, 17(1-2), 49-53.

[42] Romana, A., Bandon, J., Perez, M., Gutierrez, S., Richter, R., Roberts, A., Provost, E. M., (2021). Automatically detecting errors and disfluencies in read speech to predict cognitive impairment in people with Parkinson ’ s Disease, In:Proceedings of the INTERSPEECH 2021. International Speech Communication Association, pp. 156-160.

[43] Rudd A. G., Jenkinson D., Grant R. L., & Hoffman A. (2009). Staffing levels and patient dependence in english stroke units. Clinical Medicine, 9(2), 110.

[44] Sahathevan R., Brodtmann A., & Donnan G. A. (2012). Dementia, stroke, and vascular risk factors; A review. International Journal of Stroke, 7(1), 61-73.

[45] Salvadori E., Pasi M., Poggesi A., Chiti G., Inzitari D., & Pantoni L. (2013). Predictive value of moca in the acute phase of stroke on the diagnosis of mid-term cognitive impairment. Journal of Neurology, 260(9), 2220-2227.

[46] Schuller, B., Steidl, S., Batliner, A., Hirschberg, J., Burgoon, J.K., Baird, A., Elkins, A., Zhang, Y., Coutinho, E., Evanini, K., (2016). The interspeech 2016 computational paralinguistics challenge: Deception, sincerity & native language, In:Proceedings of the 17TH Annual Conference of the International Speech Communication Association ( Interspeech 2016), Vols 1-5, vol. 8. ISCA, vol. 8. ISCA, 2001-2005.

[47] Sun J.-H., Tan L., & Yu J.-T. (2014). Post-stroke cognitive impairment: epidemiology, mechanisms and management. Annals of Translational Medicine, 2(8).

[48] Sun, L., Zheng, J., Li, J., Qian, C., (2022). Exploring mmse score prediction model based on spontaneous speech.In: SEKE, 347-350.

[49] Triantafyllopoulos, A., Keren, G., Wagner, J., Steiner, I., Schuller, B., (2019). Towards robust speech emotion recognition using deep residual networks for speech enhancement.

[50] Tudor Car L., Dhinagaran D. A., Kyaw B. M., Kowatsch T., Joty S., Theng Y.-L., & Atun R. (2020). Conversational agents in health care: Scoping review and conceptual analysis. Journal of Medical Internet Research, 22(8), Article e17158.

[51] Valstar, M., Schuller, B., Smith, K., Eyben, F., Jiang, B., Bilakhia, S., Schnieder, S., Cowie, R., Pantic, M., (2013). Avec 2013: the continuous audio/visual emotion and depression recognition challenge, In: Proceedings of the 3rd ACM international workshop on Audio/visual emotion challenge, 3-10.

[52] Yancheva, M., Fraser, K.C., Rudzicz, F., (2015). Using linguistic features longitudinally to predict clinical scores for alzheimer ’ s disease and related dementias, In: Proceedings of SLPAT 2015: 6th Workshop on Speech and Language Processing for Assistive Technologies, 134-139.

Funding

the Rosetrees Trust and the Stoneygate Trust (COMPASS, Grant Agreement No. M934). An NIHR Academic Clinical Lectureship in Neurology CL-2020-04-004 NIHR supports SMB. This summarises independent research at the NIHR Sheffield Biomedical Research Centre(Translational Neuroscience)