How people with brain injury run and evaluate a SLAM-based smartphone augmented reality application to assess object-location memory

Abstract

Augmented reality (AR) technology allows virtual objects to be superimposed on the real-world environment, offering significant potential for improving cognitive assessments and rehabilitation processes in the field of visuospatial learning. This study examines how patients with acquired brain injury (ABI) evaluate the functions and usability of a SLAM-based smartphone AR app to assess object-location skills. Ten ABI patients performed a task for the spatial recall of four objects using an AR app. The data collected from 10 healthy participants provided reference values for the best performance. Their perceptions of the AR app/technology and its usability were investigated. The results indicate lower effectiveness in solving the task in the patient group, as the time they needed to complete it was related to their level of impairment. The patients showed lower, yet positive, scores in factors related to app usability and acceptance (e.g., mental effort and satisfaction, respectively). There were more patients reported on entertainment as a positive aspect of the app. Patients’ perceived enjoyment was related to concentration and calm, whereas usability was associated with perceived competence, expertise, and a lower level of physical effort. For patients, the sensory aspects of the objects were related to their presence, while for healthy participants, they were related to enjoyment and required effort. The results show that AR seems to be a promising tool to assess spatial orientation in the target patient population.

Keywords

augmented reality / disability / neuropsychological practice / spatial memory / usability

References

[1] Astur, R. S., Taylor, L. B., Mamelak, A. N., Philpott, L., & Sutherland, R. J. (2002). Humans with hippocampus damage display severe spatial memory impairments in a virtual Morris water task. Behavioural Brain Research, 132(1), 77–84.

[2] Azuma, R., Baillot, Y., Behringer, R., Feiner, S., Julier, S., & MacIntyre, B. (2001). Recent advances in augmented reality. IEEE Computer Graphics and Applications, 21(6), 34–47.

[3] Bakır, Ç. N., Abbas, S. O., Sever, E., Özcan Morey, A., Aslan Genç H., & Mutluer, T. (2023). Use of augmented reality in mental health-related conditions: A systematic review. Digital Health, 9, 1–22.

[4] Barker-Collo, S., & Feigin, V. (2006). The impact of neuropsychological deficits on functional stroke outcomes. Neuropsychology Review, 16(2), 53–64.

[5] Barrett, A. M. M., & Muzaffar, T. (2014). Spatial cognitive rehabilitation and motor recovery after stroke. Current Opinion in Neurology, 27(6), 653–658.

[6] Bohil, C. J., Alicea, B., & Biocca, F. A. (2011). Virtual reality in neuroscience research and therapy. Nature Reviews Neuroscience, 12(12), 752–762.

[7] Bonato, M. (2012). Neglect and extinction depend greatly on task demands: A review. Frontiers in Human Neuroscience, 17(6), 195.

[8] Braun, V., & Clarke, V. (2006). Using thematic analysis in psychology. Qualitative Research in Psychology, 3(2), 77–101.

[9] Brooke, J. (1996). SUS: A ‘quick and dirty’ usability scale. In P. Jordan, B. Thomas, B. Weerdmeester, & A. McClelland (Eds.), Usability evaluation in industry (pp. 189–194). Taylor & Francis.

[10] Burgess, N., Becker, S., King, J. A., & O’Keefe, J. (2001). Memory for events and their spatial context: Models and experiments. Philosophical Transactions of the Royal Society of London, 356(1413), 1493–1503.

[11] Calle-Bustos, A. M., Juan, M. C., García-García, I., & Abad, F. (2017). An augmented reality game to support therapeutic education for children with diabetes. PLoS One, 12(9), e0184645.

[12] Carelli, L., Rusconi, M. L., Scarabelli, C., Stampatori, C., Mattioli, F., & Riva, G. (2011). The transfer from survey (map-like) to route representations into virtual reality mazes: Effect of age and cerebral lesion. Journal of Neuroengineering and Rehabilitation, 8, 6.

[13] Cicerone, K. D., Dahlberg, C., Kalmar, K., Langenbahn, D. M., Malec, J. F., Bergquist, T. F., Felicetti, T., Giacino, J. T., Harley, J. P., Harrington, D. E., Herzog, J., Kneipp, S., Laatsch, L., & Morse, P. A. (2000). Evidence-based cognitive rehabilitation: Recommendations for clinical practice. Archives of Physical Medicine and Rehabilitation, 81(12), 1596–1615.

[14] Cimadevilla, J. M., Lizana, J. R., Roldán, M. D., Cánovas, R., & Rodríguez, E. (2014). Spatial memory alterations in children with epilepsy of genetic origin or unknown cause. Epileptic Disorders: International Epilepsy Journal with Videotape, 16(2), 203–207.

[15] Cohen, M. A. (1998). The monetary value of saving a high-risk youth. Journal of Quantitative Criminology, 14(1), 5–33.

[16] Czaja, S. J., Charness, N., Fisk, A. D., Hertzog, C., Nair, S. N., Rogers, W. A., & Sharit, J. (2006). Factors predicting the use of technology: Findings from the Center for Research and Education on Aging and Technology Enhancement (CREATE). Psychology and Aging, 21(2), 333–352.

[17] de Assis, G. A., Corrêa, A. G. D., Martins, M. B. R., Pedrozo, W. G., & de Lopes, R. (2016). An augmented reality system for upper-limb post-stroke motor rehabilitation: A feasibility study. Disability and Rehabilitation: Assistive Technology, 11(6), 521–528.

[18] Evald, L. (2015). Prospective memory rehabilitation using smartphones in patients with TBI: What do participants report? Neuropsychological Rehabilitation, 25(2), 283–297.

[19] Fernandez-Baizan, C., Diaz-Caceres, E., Arias, J. L., & Mendez, M. (2019). Egocentric and allocentric spatial memory in healthy aging: Performance on real-world tasks. Brazilian Journal of Medical and Biological Research, 52(4), e8041.

[20] Furió D., González-Gancedo, S., Juan, M.-C., Seguí I., & Costa, M. (2013). The effects of the size and weight of a mobile device on an educational game. Computers & Education, 64, 24–41.

[21] Hampstead, B. M., Gillis, M. M., & Stringer, A. Y. (2014). Cognitive rehabilitation of memory for mild cognitive impairment: A methodological review and model for future research. Journal of the International Neuropsychological Society, 20(2), 135–151.

[22] Hampstead, B. M., Stringer, A. Y., Stilla, R. F., Amaraneni, A., & Sathian, K. (2011). Where did I put that? Patients with amnestic mild cognitive impairment demonstrate widespread reductions in activity during the encoding of ecologically relevant object-location associations. Neuropsychologia, 49(9), 2349–2361.

[23] Hart, T., Buchhofer, R., & Vaccaro, M. (2004). Portable electronic devices as memory and organizational aids after traumatic brain injury: A consumer survey study. The Journal of Head Trauma Rehabilitation, 19(5), 351–365.

[24] Hervas, R., Bravo, J., & Fontecha, J. (2014). An assistive navigation system based on augmented reality and context awareness for people with mild cognitive impairments. IEEE Journal of Biomedical and Health Informatics, 18(1), 368–374.

[25] Hugues, O., Fuchs, P., & Nannipieri, O. (2011). New augmented reality taxonomy: Technologies and features of augmented environment. In B. Furht (Ed.), Handbook of augmented reality (pp. 47–63). Springer.

[26] Johnson, D. M. (2005). Introduction to and review of simulator sickness research. Rotary-Wing Aviation Research Unit, U.S. Army Research Institute for the Behavioral and Social Sciences.

[27] Juan, M. C., Alcaniz, M., Monserrat, C., Botella, C., Banos, R., & Guerrero, B. (2005). Using augmented reality to treat phobias. IEEE Computer Graphics and Applications, 25(6), 31–37.

[28] Juan, M. C., Mendez-Lopez, M., Perez-Hernandez, E., & Albiol-Perez, S. (2014). Augmented reality for the assessment of children’s spatial memory in real settings. PLoS One, 9(12), e113751.

[29] Kang, Y. J., Ku, J., Han, K., Kim, S. I., Yu, T. W., Lee, J. H., & Park, C. I. (2008). Development and clinical trial of virtual reality-based cognitive assessment in people with stroke: Preliminary study. Cyberpsychology & Behavior, 11(3), 329–339.

[30] Kessels, R. P. C., Jaap Kappelle, L., de Haan, E. H. F., & Postma, A. (2002). Lateralization of spatial-memory processes: Evidence on spatial span, maze learning, and memory for object locations. Neuropsychologia, 40(8), 1465–1473.

[31] Kessels, R. P. C., Postma, A., & de Haan, E. H. F. (1999). Object relocation: A program for setting up, running, and analyzing experiments on memory for object locations. Behavior Research Methods, Instruments, & Computers, 31(3), 423–428.

[32] Koen, J. D., Borders, A. A., Petzold, M. T., & Yonelinas, A. P. (2017). Visual short-term memory for high resolution associations is impaired in patients with medial temporal lobe damage. Hippocampus, 27(2), 184–193.

[33] Lindén, A., Lexell, J., & Lund, M. L. (2010). Perceived difficulties using everyday technology after acquired brain injury: Influence on activity and participation. Scandinavian Journal of Occupational Therapy, 17(4), 267–275.

[34] Lobo, A., Ezquerra, J., Gómez, F., Sala, J., & Seva, A. (1979). El Mini Examen Cognoscitivo: un test sencillo, práctico, para detectar alteraciones intelectivas en pacientes médicos. Actas Luso-Españolas de Neurología, Psiquiatría y Ciencias Afines, 3, 189–202.

[35] Lovgren Engstrom, A.-L., Lexell, J., & Larsson Lund, M. (2010). Difficulties in using everyday technology after acquired brain injury: A qualitative analysis. Scandinavian Journal of Occupational Therapy, 17(3), 1–11.

[36] Luque-Moreno, C., Ferragut-Garcías, A., Rodríguez-Blanco, C., Heredia-Rizo, A. M., Oliva-Pascual-Vaca, J., Kiper, P., & Oliva-Pascual-Vaca, Á. (2015). A decade of progress using virtual reality for poststroke lower extremity rehabilitation: Systematic review of the intervention methods. BioMed Research International, 2015, 342529.

[37] Mahoney, F. I., & Barthel, D. W. (1965). Functional evaluation: The Barthel index: A simple index of independence useful in scoring improvement in the rehabilitation of the chronically ill. Maryland State Medical Journal, 14, 61–65.

[38] McAuley, E. D., Duncan, T., & Tammen, V. V. (1989). Psychometric properties of the intrinsic motivation inventory in a competitive sport setting: A confirmatory factor analysis. Research Quarterly for Exercise and Sport, 60(1), 48–58.

[39] McDonald, B. C., Flashman, L. A., & Saykin, A. J. (2002). Executive dysfunction following traumatic brain injury: Neural substrates and treatment strategies. NeuroRehabilitation, 17(4), 333–344.

[40] Mendez-Lopez, M., Perez-Hernandez, E., & Juan, M. C. (2016). Learning in the navigational space: Age differences in a short-term memory for objects task. Learning and Individual Differences, 50, 11–22.

[41] Meza-Kubo, V., & Morán, A. L. (2013). UCSA: A design framework for usable cognitive systems for the worried-well. Personal and Ubiquitous Computing, 17(6), 1135–1145.

[42] Moffat, S. D., Kennedy, K. M., Rodrigue, K. M., & Raz, N. (2007). Extrahippocampal contributions to age differences in human spatial navigation. Cerebral Cortex, 17(6), 1274–1282.

[43] Monacelli, A. M., Cushman, L. A., Kavcic, V., & Duffy, C. J. (2003). Spatial disorientation in Alzheimer’s disease: The remembrance of things passed. Neurology, 61(11), 1491–1497.

[44] Mousavi Hondori, H., Khademi, M., Dodakian, L., McKenzie, A., Lopes, C. V., & Cramer, S. C. (2016). Choice of human–computer interaction mode in stroke rehabilitation. Neurorehabilitation and Neural Repair, 30(3), 258–265.

[45] Munoz-Montoya, F., Fidalgo, C., Juan, M.-C., & Mendez-Lopez, M. (2019). Memory for object location in augmented reality: The role of gender and the relationship among spatial and anxiety outcomes. Frontiers in Human Neuroscience, 13, 113.

[46] Munoz-Montoya, F., Juan, M.-C., Mendez-Lopez, M., & Fidalgo, C. (2019). Augmented reality based on SLAM to assess spatial short-term memory. IEEE Access, 7, 2453–2466.

[47] Osumi, M., Ichinose, A., Sumitani, M., Wake, N., Sano, Y., Yozu, A., Kumagaya, S., Kuniyoshi, Y., & Morioka, S. (2017). Restoring movement representation and alleviating phantom limb pain through short-term neurorehabilitation with a virtual reality system. European Journal of Pain, 21(1), 140–147.

[48] Pallavicini, F., Pedroli, E., Serino, S., Dell’Isola, A., Cipresso, P., Cisari, C., & Riva, G. (2015). Assessing unilateral spatial neglect using advanced technologies: The potentiality of mobile virtual reality. Technology and Health Care, 23(6), 795–807.

[49] Peleg-Adler, R., Lanir, J., & Korman, M. (2018). The effects of aging on the use of handheld augmented reality in a route planning task. Computers in Human Behavior, 81, 52–62.

[50] Piccardi, L., Iaria, G., Bianchini, F., Zompanti, L., & Guariglia, C. (2011). Dissociated deficits of visuo-spatial memory in near space and navigational space: Evidence from brain-damaged patients and healthy older participants. Aging, Neuropsychology, and Cognition, 18(3), 362–384.

[51] Ponce, D., Torres, C., Mendez-Lopez, M., Molla, R., & Juan, M. C. (2024). Augmented reality to assess short-term spatial memory. A comparative study of olfactory, visual, and tactile stimuli. IEEE Access, 12, 47041–47056.

[52] Pugnetti, L., Mendozzi, L., Attree, E. A., Barbieri, E., Brooks, B. M., Cazzullo, C. L., Motta, A., & Rose, F. D. (1998). Probing memory and executive functions with virtual reality: Past and present studies. CyberPsychology and Behavior, 1(2), 151–161.

[53] Raspelli, S., Pallavicini, F., Carelli, L., Morganti, F., Cipresso, P., Pedroli, E., Poletti, B., Corra, B., Sangalli, D., Silani, V., & Riva, G. (2012). Validating the neuro VR-based virtual version of the multiple errands test: Preliminary results. Presence, 21(1), 31–42.

[54] Regenbrecht, H., & Schubert, T. (2002). Measuring presence in augmented reality environments: Design and a first test of a questionnaire. In Proceedings of the 5th Annual International Workshop on Presence. arXiv.org.

[55] Rose, F. D., Brooks, B. M., Attree, E. A., Parslow, D. M., Leadbetter, A. G., McNeil, J. E., Jayawardena, S., Greenwood, R., & Potter, J. (1999). A preliminary investigation into the use of virtual environments in memory retraining after vascular brain injury: Indications for future strategy? Disability and Rehabilitation, 21(12), 548–554.

[56] Salar, R., Arici, F., Caliklar, S., & Yilmaz, R. M. (2020). A model for augmented reality immersion experiences of university students studying in science education. Journal of Science Education and Technology, 29, 257–271.

[57] Slater, M., Usoh, M., & Steed, A. (1994). Depth of presence in virtual environments. Presence, 3(2), 130–144.

[58] Tieri, G., Morone, G., Paolucci, S., & Iosa, M. (2018). Virtual reality in cognitive and motor rehabilitation: Facts, fiction and fallacies. Expert Review of Medical Devices, 15(2), 107–117.

[59] Tinetti, M. E., Franklin Williams, T., & Mayewski, R. (1986). Fall risk index for elderly patients based on number of chronic disabilities. The American Journal of Medicine, 80(3), 429–434.

[60] Toma, M. V., Turcu, C. E., Turcu, C. O., Vlad, S., Tiliute, D. E., & Pascu, P. (2024). Extended reality-based mobile app solutions for the therapy of children with autism spectrum disorders: Systematic literature review. JMIR Serious Games, 12(1), e49906.

[61] Tunur, T., DeBlois, A., Yates-Horton, E., Rickford, K., & Columna, L. A. (2019). Augmented reality-based dance intervention for individuals with Parkinson’s disease: A pilot study. Disability and Health Journal, 13(2), 100848.

[62] van der Kuil, M. N. A., Visser-Meily, J. M. A., Evers, A. W. M., & van der Ham, I. J. M. (2018). A usability study of a serious game in cognitive rehabilitation: A compensatory navigation training in acquired brain injury patients. Frontiers in Psychology, 9, 846.

[63] Venkatesh, V., Morris, M. G., Davis, G. B., & Davis, F. D. (2003). User acceptance of information technology: Toward a unified view. MIS Quarterly, 27(3), 425–478.

[64] Wald, J. L., Liu, L., & Reil, S. (2000). Concurrent validity of a virtual reality driving assessment for persons with brain injury. Cyberpsychology & Behavior, 3(4), 643–654.

[65] Weniger, G., Ruhleder, M., Wolf, S., Lange, C., & Irle, E. (2009). Egocentric memory impaired and allocentric memory intact as assessed by virtual reality in subjects with unilateral parietal cortex lesions. Neuropsychologia, 47(1), 59–69.

[66] Wenk, N., Buetler, K. A., Penalver-Andres, J., Müri, R. M., & Marchal-Crespo, L. (2022). Naturalistic visualization of reaching movements using head-mounted displays improves movement quality compared to conventional computer screens and proves high usability. Journal of Neuroengineering and Rehabilitation, 19(1), 137.

[67] Wilson, B. A., Baddeley, A. D., & Cockburn, J. M. (1989). How do old dogs learn new tricks: Teaching a technological skill to brain injured people. Cortex, 25(1), 115–119.

[68] Witmer, B. G., & Singer, M. J. (1998). Measuring presence in virtual environments: A presence questionnaire. Presence, 7(3), 225–240.

[69] Wong, D., Sinclair, K., Seabrook, E., McKay, A., & Ponsford, J. (2017). Smartphones as assistive technology following traumatic brain injury: A preliminary study of what helps and what hinders. Disability and Rehabilitation, 39(23), 2387–2394.

[70] Zaidel, D. W. (2014). Creativity, brain, and art: Biological and neurological considerations. Frontiers in Human Neuroscience, 2(8), 389.

[71] Zhang, J. Y., & Feinstein, A. (2016). Screening for cognitive impairments after traumatic brain injury: A comparison of a brief computerized battery with the Montreal cognitive assessment. The Journal of Neuropsychiatry and Clinical Neurosciences, 28(4), 328–331.

RIGHTS & PERMISSIONS

2024 The Author(s). PsyCh Journal published by Institute of Psychology, Chinese Academy of Sciences and John Wiley & Sons Australia, Ltd.