Emotion recognition in virtual reality: Toward adaptive and responsible technologies

Abstract

Affective Computing aims to build systems that can sense, interpret, and influence human emotions. In parallel, virtual reality (VR) has matured into a powerful solution for immersion, presence, and interaction. When combined, these two technologies enable a new generation of emotionally adaptive systems that can recognize users’ inner states and react in real time. In this article, we explore the opportunities and challenges of emotion recognition and emotion elicitation in VR, focusing on the Virtual Emotion, Elicitation and Recognition Loop, a closed-loop framework where virtual environments adapt based on recognized and targeted emotions. We discuss applications in manufacturing, product design, and education, and highlight key ethical and privacy concerns, especially in light of recent regulations such as the European Commission released the proposed Regulation on Artificial Intelligence (EU AI Act). Finally, we outline open research directions toward responsible, privacy-aware, and scalable affective VR systems, suggesting that the convergence between VR and Affective Computing can become a crucial foundation for the next generation of human-centric interactive technologies.

Keywords

Affective computing / emotion recognition / emotion elicitation / virtual reality / adaptive systems / privacy / responsible artificial intelligence

References

[1] Wang Y, Song W, Tao W, Liotta A, Yang D, Li X, et al. A systematic review on affective computing: Emotion models, databases, and recent advances. Inf Fusion. 2022; 83: 19-52.

[2] Shehu HA, Browne WN, Eisenbarth H. Emotion categorization from facial expressions: A review of datasets, methods, and research directions. Neurocomputing. 2025; 624: 129367.

[3] Dzardanova E, Kasapakis V. Virtual reality: A journey from vision to commodity. IEEE Ann Hist Comput. 2023; 45(1): 18-30.

[4] Papandrea M, Peternier A, Frei D, La Porta N, Gelsomini M, Allegri D, et al. V-cockpit: A platform for the design, testing, and validation of car infotainment systems through virtual reality. Appl Sci. 2024; 14(18): 8160.

[5] Klein Schaarsberg RE, van Dam L, Widdershoven GAM, Lindauer RJL, Popma A. Ethnic representation within virtual reality: A co-design study in a forensic youth care setting. BMC Digit Health. 2024; 2: 25.

[6] Standen B, Anderson J, Sumich A, Heym N. Effects of system- and media-driven immersive capabilities on presence and affective experience. Virtual Real. 2023; 27(1): 371-384.

[7] Lemmens JS, Simon M, Sumter SR. Fear and loathing in VR: The emotional and physiological effects of immersive games. Virtual Real. 2022; 26(1): 223-234.

[8] Linares-Vargas BGP, Cieza-Mostacero SE. Interactive virtual reality environments and emotions: A systematic review. Virtual Real. 2025; 29: 3.

[9] Mousavi SMH, Khaertdinov B, Jeuris P, Hortal E, Andreoletti D, Giordano S. Emotion recognition in adaptive virtual reality settings: Challenges and opportunities. In: 2023 Workshop on Advances of Mobile and Wearable Biometrics; 2023 Sep 26; Athens, Greece. Aachen: CEUR-WS.org; 2023. p. 1-20. Available from: https://ceur-ws.org/Vol-3517/paper1.pdf

[10] Andreoletti D, Luceri L, Peternier A, Leidi T, Giordano S. The virtual emotion loop: Towards emotion-driven product design via virtual reality. In: 2021 16th Conference on Computer Science and Intelligence Systems (FedCSIS); 2021 Sep 02-05; Sofia, Bulgaria. Piscataway: IEEE; 2021. p. 371-378.

[11] Andreoletti D, Paoliello M, Luceri L, Leidi T, Peternier A, Giordano S. A framework for emotion-driven product design through virtual reality. In: Ziemba E, Chmielarz W, editors. Information technology for management: Business and social issues. Cham: Springer; 2022. p. 42-61.

[12] Giaretta A. Security and privacy in virtual reality: A literature survey. Virtual Real. 2025; 29: 10.

[13] McStay A. Emotional AI, soft biometrics and the surveillance of emotional life: An unusual consensus on privacy. Big Data Soc. 2020; 7(1): 2053951720904386.

[14] Fox E. Perspectives from affective science on understanding the nature of emotion. Brain Neurosci Adv. 2018; 2: 2398212818812628.

[15] Kisley MA, Shulkin J, Meza-Whitlatch MV, Pedler RB. Emotion beliefs: Conceptual review and compendium. Front Psychol. 2024; 14: 1271135.

[16] Calvo RA, D’Mello S, Gratch J, Kappas A, editors. The oxford handbook of affective computing . Oxford: Oxford University Press; 2015.

[17] Somarathna R, Bednarz T, Mohammadi G. Virtual reality for emotion elicitation-A review. IEEE Trans Affect Comput. 2022; 14(4): 2626-2645.

[18] Jinan UA, Heidarikohol N, Borst CW, Billinghurst M, Jung S. A systematic review of using immersive technologies for empathic computing from 2000-2024. Empath Comput. 2025; 1: 202501.

[19] Cornelio P, Velasco C, Obrist M. Multisensory integration as per technological advances: A review. Front Neurosci. 2021; 15: 652611.

[20] Castiblanco Jimenez IA, Olivetti EC, Vezzetti E, Moos S, Celeghin A, Marcolin F. Effective affective EEG-based indicators in emotion-evoking VR environments: An evidence from machine learning. Neural Comput Applic. 2024; 36(35): 22245-22263.

[21] Gupta K, Lazarevic J, Pai YS, Billinghurst M. AffectivelyVR: Towards VR personalized emotion recognition. In: VRST ’20: Proceedings of the 26th ACM Symposium on Virtual Reality Software and Technology; 2020 Nov 01-04; Virtual event, Canada. New York: Association for Computing Machinery; 2020. p. 1-3.

[22] Polo EM, Iacomi F, Rey AV, Ferraris D, Paglialonga A, Barbieri R. Advancing emotion recognition with Virtual Reality: A multimodal approach using physiological signals and machine learning. Comput Biol Med. 2025; 193: 110310.

[23] Romero-Ramos L, González-Serna G, López-Sánchez M, González-Franco N, Valenzuela-Robles B. Emotion recognition in virtual reality learning environments: A multimodal machine learning approach. In: Martínez-Villaseñor L, Ochoa-Ruiz G, Montes Rivera M, Barrón-Estrada ML, Acosta-Mesa HG, editors. Advances in computational intelligence. MICAI 2024 International Workshops. Cham: Springer; 2025. p. 45-52.

[24] Kim MK, Kim M, Oh E, Kim SP. A review on the computational methods for emotional state estimation from the human EEG. Comput Math Meth Med. 2013; 2013: 573734.

[25] Warsinke M, Kojić T, Vergari M, Spang R, Voigt-Antons JN, Möller S. Comparing continuous and retrospective emotion ratings in remote VR study. In: 2024 16th International Conference on Quality of Multimedia Experience (QoMEX); 2024 Jun 18-20; Karlshamn, Sweden. Piscataway: IEEE; 2024. p. 139-145.

[26] Maksatbekova A, Argan M. Virtual reality’s dual edge: Navigating mental health benefits and addiction risks across timeframes. Curr Psychol. 2025; 44(7): 6469-6480.

[27] Kim H, Kim DJ, Chung WH, Park KA, Kim JDK, Kim D, et al. Clinical predictors of cybersickness in virtual reality (VR) among highly stressed people. Sci Rep. 2021; 11: 12139.

[28] Cebeci B, Celikcan U, Capin TK. A comprehensive study of the affective and physiological responses induced by dynamic virtual reality environments. Comput Anim Virtual Worlds. 2019; 30: e1893.

[29] Lavoie R, Main K, King C, King D. Virtual experience, real consequences: The potential negative emotional consequences of virtual reality gameplay. Virtual Real. 2021; 25(1): 69-81.

[30] Mousavi SMH, Besenzoni M, Andreoletti D, Peternier A, Giordano S. The magic XRoom: A flexible VR platform for controlled emotion elicitation and recognition. In: Proceedings of the 25th International Conference on Mobile Human-Computer Interaction; 2023 Sep 26-29; Athens, Greece. New York: Association for Computing Machinery; 2023. p. 1-5.

[31] Henrich J, Heine SJ, Norenzayan A. The weirdest people in the world? Behav Brain Sci. 2010; 33: 61-83.

[32] Baghaei N, Chitale V, Hlasnik A, Stemmet L, Liang HN, Porter R. Virtual reality for supporting the treatment of depression and anxiety: Scoping review. JMIR Ment Health. 2021; 8(9): e29681.

[33] Omisore OM, Odenigbo I, Orji J, Beltran AIH, Meier S, Baghaei N, et al. Extended reality for mental health evaluation: Scoping review. JMIR Serious Games. 2024; 12: e38413.

[34] Fabiano N. Affective computing and emotional data: Challenges and implications in privacy regulations, the AI act, and ethics in large language models. arXiv:2509.20153v2 [Preprint]. 2025.

[35] Wu Y, Shi C, Zhang T, Walker P, Liu J, Saxena N, et al. Privacy leakage via unrestricted motion-position sensors in the age of virtual reality: A study of snooping typed input on virtual keyboards. In: 2023 IEEE Symposium on Security and Privacy (SP); 2023 May 21-25; San Francisco, USA. Piscataway: IEEE; 2023. p. 3382-3398.

[36] Chitale V, Henry JD, Liang HN, Matthews B, Baghaei N. Virtual reality analytics map (VRAM): A conceptual framework for detecting mental disorders using virtual reality data. New Ideas Psychol. 2025; 76: 101127.

[37] Di Pasquale V, Cutolo P, Esposito C, Franco B, Iannone R, Miranda S. Virtual reality for training in assembly and disassembly tasks: A systematic literature review. Machines. 2024; 12(8): 528.

[38] Kulal S, Li Z, Tian X. Security and privacy in virtual reality: A literature review. Issues Inf Syst. 2022; 23(2): 185-192.

[39] Olade I, Fleming C, Liang HN. BioMove: Biometric user identification from human kinesiological movements for virtual reality systems. Sensors. 2020; 20(10): 2944.

[40] Nair V, Guo W, Mattern J, Wang R, O’Brien JF, Rosenberg L, et al. Unique identification of 50,000+ virtual reality users from head & hand motion data. In: 32nd USENIX Security Symposium; 2023 Aug 09-11; Anaheim, USA. Berkeley: USENIX Association; 2023. p. 895-910.

[41] Regulation of the European Parliament and of the Council Laying Down Harmonised Rules on Artificial Intelligence (EU AI Act) [Internet]. Luxembourg: Publications Office of the European Union; 2024.

[42] Kokina J, Blanchette S, Davenport TH, Pachamanova D. Challenges and opportunities for artificial intelligence in auditing: Evidence from the field. Int J Account Inf Syst. 2025; 56: 100734.

[43] Chesterman S. From ethics to law: Why, when, and how to regulate AI. In: Gunkel DJ, editor. Handbook on the ethics of artificial intelligence. Cheltenham: Edward Elgar Publishing; 2023. p. 113-127.

[44] Barker D, Tippireddy MKR, Farhan A, Ahmed B. Ethical considerations in emotion recognition research. Psychol Int. 2025; 7(2): 43.

[45] Sigfrids A, Leikas J, Salo-Pöntinen H, Koskimies E. Human-centricity in AI governance: A systemic approach. Front Artif Intell. 2023; 6: 976887.

[46] Podschwadt R, Takabi D, Hu P, Rafiei MH, Cai Z. A survey of deep learning architectures for privacy-preserving machine learning with fully homomorphic encryption. IEEE Access. 2022; 10: 117477-117500.

[47] Knott B, Venkataraman S, Hannun A, Sengupta S, Ibrahim M, van der Maaten L. CrypTen: Secure multi-party computation meets machine learning. Adv Neural Inf Process Syst. 2021; 34: 4961-4973.

[48] Dwork C. The promise of differential privacy: A tutorial on algorithmic techniques. In: 2011 IEEE 52nd Annual Symposium on Foundations of Computer Science; 2011 Oct 22-25; Palm Springs, USA. Los Alamitos: IEEE Computer Society; 2011. p. 1-2.

[49] Guo W, Zhuang F, Zhang X, Tong Y, Dong J. A comprehensive survey of federated transfer learning: Challenges, methods and applications. Front Comput Sci. 2024; 18(6): 186356.

[50] Zhu J. Synthetic data generation by diffusion models. Natl Sci Rev. 2024; 11(8): nwae276.

[51] Hine E, Rezende IN, Roberts H, Wong D, Taddeo M, Floridi L. Safety and privacy in immersive extended reality: An analysis and policy recommendations. Digit Soc. 2024; 3(2): 33.

[52] Raja US, Al-Baghli R. Ethical concerns in contemporary virtual reality and frameworks for pursuing responsible use. Front Virtual Real. 2025; 6: 1451273.

[53] Boemer F, Lao Y, Cammarota R, Wierzynski C. nGraph-HE: A graph compiler for deep learning on homomorphically encrypted data. In: Proceedings of the 16th ACM International Conference on Computing Frontiers; 2019 Apr 30-May 02; Alghero, Italy. New York: Association for Computing Machinery; 2019. p. 3-13.

[54] Gilad-Bachrach R, Dowlin N, Laine K, Lauter K, Naehrig M, Wernsing J. Cryptonets: Applying neural networks to encrypted data with high throughput and accuracy. In: Proceedings of the 33rd International Conference on Machine Learning; 2016 Jun 19-24; New York, USA. New York: PMLR; 2016. p. 201-210. Available from: https://proceedings.mlr.press/v48/gilad-bachrach16.html?utm_source=chatgpt.com

[55] Wei K, Li J, Ding M, Ma C, Yang HH, Farokhi F, et al. Federated learning with differential privacy: Algorithms and performance analysis. IEEE Trans Inf Forensics Secur. 2020; 15: 3454-3469.

[56] Dhall A, Singh M, Goecke R, Gedeon T, Zeng D, Wang Y, et al. EmotiW 2023: Emotion recognition in the wild challenge. In: Proceedings of the 25th International Conference on Multimodal Interaction; 2023 Oct 09-13; Paris, France. New York: Association for Computing Machinery; 2023. p. 746-749.

[57] Wang X, Chen C, Yang F, Gong X, Zhao S. SRMER: Synthetic-to-real multimodal emotion recognition. Inf Fusion. 2026; 127: 103869.

[58] Hao S, Han W, Jiang T, Li Y, Wu H, Zhong C, et al. Synthetic data in AI: Challenges, applications, and ethical implications. arXiv:2401.01629v1 [Preprint]. 2024.

[59] Ning X, Zhuo Y, Wang X, Sio CID, Lee LH. When generative artificial intelligence meets extended reality: A systematic review. Int J Hum. 2025.

[60] Freiknecht J, Effelsberg W. A survey on the procedural generation of virtual worlds. Multimodal Technol Interact. 2017; 1(4): 27.