Virtual humans’ facial expressions, gestures, and voices impact user empathy

Abstract

Aims:Users’ empathy towards artificial agents can be influenced by the agent’s expression of emotion. To date, most studies have used a Wizard of Oz design or manually programmed agents’ expressions. This study investigated whether autonomously animated emotional expression and neural voices could increase user empathy towards a Virtual Human. Methods:158 adults participated in an online experiment, where they watched videos of six emotional stories generated by ChatGPT. For each story, participants were randomly assigned to a virtual human (VH) called Carina, telling the story with either (1) autonomous expressive or non-expressive animation, and (2) a neural or standard text-to-speech (TTS) voice. After each story, participants rated how well the animation and voice matched the story, and their cognitive, affective, and subjective empathy towards Carina were evaluated. Qualitative data were collected on how well participants thought Carina expressed emotion. Results:Autonomous emotional expression enhanced the alignment between the animation and voice, and improved subjective, cognitive, and affective empathy. The standard voice was rated as matching the fear and sad stories better, the sad animation was better, creating greater subjective and cognitive empathy for the sad story. Trait empathy and ratings of how well the animation and voice matched the story predicted subjective empathy. Qualitative analysis revealed that the animation conveyed emotions more effectively than the voice, and emotional expression was associated with increased empathy. Conclusion:Autonomous emotional animation of VHs can improve empathy towards AI-generated stories. Further research is needed on voices that can dynamically change to express different emotions.

Keywords

Embodied conversational agent / empathy / emotion / autonomous / voice / face / gesture

References

[1] Jami PY, Walker DI, Mansouri B. Interaction of empathy and culture: A review. Curr Psychol. 2024; 43(4): 2965-2980. [DOI]

[2] Riess H. The science of empathy. J Patient Exp. 2017; 4(2): 74-77. [DOI]

[3] Keshtkar L, Madigan CD, Ward A, Ahmed S, Tanna V, Rahman I, et al. The effect of practitioner empathy on patient satisfaction: A systematic review of randomized trials. Ann Intern Med. 2024; 177(2): 196-209. [DOI]

[4] Broadbent E, Johanson D, Shah J. A new model to enhance robot-patient communication: Applying insights from the medical world. In: International Conference on Social Robotics; 2018 Nov 28-30; Qingdao, China. Cham: Springer; 2018. p. 308-317. [DOI]

[5] Johanson DL, Ahn HS, Broadbent E. Improving interactions with healthcare robots: A review of communication behaviours in social and healthcare contexts. Int J Soc Robot. 2021; 13(8): 1835-1850. [DOI]

[6] Loveys K, Sebaratnam G, Sagar M, Broadbent E. The effect of design features on relationship quality with embodied conversational agents: A systematic review. Int J Soc Robot. 2020; 12(6): 1293-1312. [DOI]

[7] Christov-Moore L, Reggente N, Vaccaro A, Schoeller F, Pluimer B, Douglas PK, et al. Preventing antisocial robots: A pathway to artificial empathy. Sci Robot. 2023; 8(80): eabq3658. [DOI]

[8] Raamkumar AS, Yang Y. Empathetic conversational systems: A review of current advances, gaps, and opportunities. IEEE Trans Affect Comput. 2022; 14(4): 2722-2739. [DOI]

[9] Park S, Whang M. Empathy in human-robot interaction: Designing for social robots. Int J Environ Res Public Health. 2022; 19(3): 1889. [DOI]

[10] Rosenthal-Von der Pütten AM, Schulte FP, Eimler SC, Sobieraj S, Hoffmann L, Maderwald S, et al. Investigations on empathy towards humans and robots using fMRI. Comput Hum Behav. 2014; 33: 201-212. [DOI]

[11] Mattiassi AD, Sarrica M, Cavallo F, Fortunati L. Degrees of empathy: Humans’ empathy toward humans, animals, robots and objects. In: Casiddu N, Porfirione C, Monteriù A, Cavallo F, editors. Ambient assisted living. Cham: Springer; 2017. p. 101-113. [DOI]

[12] Costa S, Brunete A, Bae BC, Mavridis N. Emotional storytelling using virtual and robotic agents. Int J Humanoid Robot. 2018; 15(03): 1850006. [DOI]

[13] Spitale M, Okamoto S, Gupta M, Xi H, Matarić MJ. Socially assistive robots as storytellers that elicit empathy. ACM Trans Hum-Robot Interact. 2022; 11(4): 1-29. [DOI]

[14] Appel M, Lugrin B, Kühle M, Heindl C. The emotional robotic storyteller: On the influence of affect congruency on narrative transportation, robot perception, and persuasion. Comput Hum Behav. 2021; 120: 106749. [DOI]

[15] Paiva A, Leite I, Boukricha H, Wachsmuth I. Empathy in virtual agents and robots: A survey. ACM Trans Interact Intell Syst. 2017; 7(3): 1-40. [DOI]

[16] Rawal N, Stock-Homburg RM. Facial emotion expressions in human-robot interaction: A survey. Int J Soc Robot. 2022; 14(7): 1583-1604. [DOI]

[17] Ferstl Y, Neff M, McDonnell R. ExpressGesture: Expressive gesture generation from speech through database matching. Comput Animat Virtual Worlds. 2021; 32(3-4):e2016. [DOI]

[18] Nyatsanga S, Kucherenko T, Ahuja C, Henter GE, Neff M. A comprehensive review of data-driven co-speech gesture generation. Comput Graph Forum. 2023; 42(2): 569-596. [DOI]

[19] Liu Y, Mohammadi G, Song Y, Johal W. Speech-based gesture generation for robots and embodied agents: A scoping review. In: Proceedings of the 9th International Conference on Human-Agent Interaction; 2021 Nov 09-11; Virtual Event, Japan. New York: Association for Computing Machinery; 2021. p. 31-38. [DOI]

[20] Llanes-Jurado J, Gómez-Zaragozá L, Minissi ME, Alcañiz M, Marín-Morales J. Developing conversational virtual humans for social emotion elicitation based on large language models. Expert Syst Appl. 2024; 246: 123261. [DOI]

[21] Seaborn K, Miyake NP, Pennefather P, Otake-Matsuura M. Voice in human-agent interaction: A survey. ACM Comput Surv. 2021; 54(4): 1-43. [DOI]

[22] James J, Watson CI, MacDonald B. Artificial empathy in social robots: An analysis of emotions in speech. In: 2018 27th IEEE international symposium on robot and human interactive communication (RO-MAN); Aug 27-31; Nanjing, China. Piscataway: IEEE; 2018. p. 632-637. [DOI]

[23] James J, Balamurali B, Watson CI, MacDonald B. Empathetic speech synthesis and testing for healthcare robots. Int J Soc Robot. 2021; 13(8): 2119-2137. [DOI]

[24] Beatman A. Announcing new voices and emotions to Azure Neural Text to Speech. Redmond: Microsoft Azure; 2022. Available from: https://azure.microsoft.com/en-us/blog/announcing-new-voices-and-emotions-to-azure-neural-text-to-speech/

[25] Yun C. A new generative engine and three voices are now generally available on Amazon Polly. Seattle: Amazon Web Services; 2024. Available from: https://aws.amazon.com/cn/blogs/aws/a-new-generative-engine-and-three-voices-are-now-generally-available-on-amazon-polly/

[26] Tan X. Neural text-to-speech synthesis. In: O’Sullivan B, Wooldridge M, editors. Artificial intelligence: Foundations, theory, and algorithms. Singapore: Springer Nature; 2023. [DOI]

[27] Du H, Chhatre K, Peters C, Keegan B, McDonnell R, Ennis C. Synthetically expressive: Evaluating gesture and voice for emotion and empathy in VR and 2D scenarios. In: Proceedings of the 25th ACM International Conference on Intelligent Virtual Agents; 2025 Sep 16-19; Berlin, Germany. New York: Association for Computing Machinery; 2025. p. 1-10. [DOI]

[28] Velentza AM, Fachantidis N, Pliasa S. Which one? Choosing favorite robot after different styles of storytelling and robots’ conversation. Front Robot AI. 2021; 8: 700005. [DOI]

[29] Davis M. A multidimensional approach to individual differences in empathy. JSAS Catalog Sel Doc Psychol. 1980; 10: 85. Available from: https://www.uv.es/friasnav/Davis_1980.pdf

[30] Darling K, Nandy P, Breazeal C. Empathic concern and the effect of stories in human-robot interaction. In: 2015 24th IEEE international symposium on robot and human interactive communication (RO-MAN); 2015 Aug31-Sep 04; Kobe, Japan. Piscataway: IEEE; 2015. p. 770-775. [DOI]

[31] Jauniaux J, Tessier MH, Regueiro S, Chouchou F, Fortin-Côté A, Jackson PL. Emotion regulation of others’ positive and negative emotions is related to distinct patterns of heart rate variability and situational empathy. PLoS One. 2020; 15(12): e0244427. [DOI]

[32] Braun V, Clarke V. Using thematic analysis in psychology. Qual Res Psychol. 2006; 3(2): 77-101. [DOI]

[33] Shen J, DiPaola D, Ali S, Sap M, Park HW, Breazeal C. Empathy toward artificial intelligence versus human experiences and the role of transparency in mental health and social support chatbot design: Comparative study. JMIR Ment Health. 2024; 11: e62679. [DOI]

[34] Stolarski Ł. Pitch patterns in vocal expression of ‘happiness’ and ‘sadness’ in the reading aloud of prose on the basis of selected audiobooks. Res Lang. 2015; 13(2): 140-161. [DOI]

[35] Lawson AP, Mayer RE. The power of voice to convey emotion in multimedia instructional messages. Int J Artif Intell Educ. 2022; 32(4): 971-990. [DOI]

[36] Shih MT, Hsu MY, Lee SC. Empathy-GPT: Leveraging large language models to enhance emotional empathy and user engagement in embodied conversational agents. In: Adjunct Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology; 2024 Oct 13-16; Pittsburgh, USA. New York: Association for Computing Machinery; 2024. p. 1-3. [DOI]

[37] Steinhaeusser SC, Piller R, Lugrin B. Combining emotional gestures, sound effects, and background music for robotic storytelling-effects on storytelling experience, emotion induction, and robot perception. In: Proceedings of the 2024 ACM/IEEE international conference on human-robot interaction; 2024 Mar 11-15; Boulder, USA. New York: Association for Computing Machinery; 2024. p. 687-696. [DOI]