Reframing human-robot interaction through extended reality: Unlocking safer, smarter, and more empathic interactions with virtual robots and foundation models

Abstract

This perspective reframes human-robot interaction (HRI) through extended reality (XR), arguing that virtual robots powered by large foundation models (FMs) can serve as cognitively grounded, empathic agents. Unlike physical robots, XR-native agents are unbound by hardware constraints and can be instantiated, adapted, and scaled on demand, while still affording embodiment and co-presence. We synthesize work across XR, HRI, and cognitive AI to show how such agents can support safety-critical scenarios, socially and cognitively empathic interaction across domains, and extend physical capabilities with XR and AI integration. We then discuss how multimodal large FMs (e.g., large language models, large vision models, and vision-language models) enable context-aware reasoning, affect-sensitive situations, and long-term adaptation, positioning virtual robots as cognitive and empathic mediators rather than mere simulation assets. At the same time, we highlight challenges and potential risks, including overtrust, cultural and representational bias, privacy concerns around biometric sensing, and data governance and transparency. The paper concludes by outlining a research agenda for human-centered, ethically grounded XR agents, emphasizing multi-layered evaluation frameworks, multi-user ecosystems, mixed virtual-physical embodiment, and societal and ethical design practices to envision XR-based virtual agents powered by FMs as reshaping future HRI into a more efficient and adaptive paradigm.

Keywords

Human-robot interaction / extended reality / large foundation models / cognitively empathic interaction

References

[1] Xia H, Zhang Y, Rajabi N, Taleb F, Yang Q, Kragic D, et al. Shaping high-performance wearable robots for human motor and sensory reconstruction and enhancement. Nat Commun. 2024; 15(1): 1760. [DOI]

[2] Zhang Y, Ma Y, Kragic D. Vision beyond boundaries: An initial design space of domain-specific large vision models in human-robot interaction. In: Dingler T, Salim F, Koelle M, van Berkel N, Waycott J, editors. Adjunct Proceedings of the 26th International Conference on Mobile Human-Computer Interaction; 2024 Sep 30-Oct 3; Melbourne, Australia. New York: Association for Computing Machinery; 2024. p. 1-8. [DOI]

[3] Ringwald M, Theben P, Gerlinger K, Hedrich A, Klein B. How should your assistive robot look like? A scoping review on embodiment for assistive robots. J Intell Rob Syst. 2023; 107(1): 12. [DOI]

[4] Karabegović I, Karabegović E, Mahmić M, Husak E. The application of service robots for logistics in manufacturing processes. Adv Prod Eng Manag. 2015; 10(4): 185-194. [DOI]

[5] Schmitt F, Piccin O, Barbé L, Bayle B. Soft robots manufacturing: A review. Front Robot AI. 2018; 5: 84. [DOI]

[6] Zhang Y, Rajabi N, Taleb F, Matviienko A, Ma Y, Björkman M, et al. Mind meets robots: A review of EEG-based brain-robot interaction systems. Int J Hum. 2025; 41(20): 12784-12815. [DOI]

[7] Trevelyan J, Hamel WR, Kang SC. Robotics in hazardous applications. In: Siciliano B, Khatib O, editors. Springer handbook of robotics. Cham: Springer; 2016. p. 1521-1548. [DOI]

[8] Willemse CJ, Toet A, van Erp JB. Affective and behavioral responses to robot-initiated social touch: Toward understanding the opportunities and limitations of physical contact in human-robot interaction. Front ICT. 2017; 4: 12. [DOI]

[9] Zhang Y, Nowak A, Xuan Y, Romanowski A, Fjeld M. See or hear? Exploring the effect of visual/audio hints and gaze-assisted instant post-task feedback for visual search tasks in AR. In: 2023 IEEE International Symposium on Mixed and Augmented Reality (ISMAR); 2023 Oct 16-20; Sydney, Australia. Piscataway: IEEE; 2023. p. 1113-1122. [DOI]

[10] Kim D, Jo D. Effects on co-presence of a virtual human: A comparison of display and interaction types. Electronics. 2022; 11(3): 367. [DOI]

[11] Lee H. A unified conceptual model of immersive experience in extended reality. Comput Hum Behav Rep. 2025; 18: 100663. [DOI]

[12] Román-Ibáñez V, Pujol-López FA, Mora-Mora H, Pertegal-Felices ML, Jimeno-Morenilla A. A low-cost immersive virtual reality system for teaching robotic manipulators programming. Sustainability. 2018; 10(4): 1102. [DOI]

[13] Solanes JE, Muñoz A, Gracia L, Tornero J. Virtual reality-based interface for advanced assisted mobile robot teleoperation. Appl Sci. 2022; 12(12): 6071. [DOI]

[14] Inamura T. Digital twin of experience for human-robot collaboration through virtual reality. Int J Autom Technol. 2023; 17(3): 284-291. [DOI]

[15] Covaciu F, Iordan AE. Control of a drone in virtual reality using MEMS sensor technology and machine learning. Micromachines. 2022; 13(4): 521. [DOI]

[16] Plümer JH, Yu K, Eck U, Kalkofen D, Steininger P, Navab N, et al. XR prototyping of mixed reality visualizations: Compensating interaction latency for a medical imaging robot. In: 2024 IEEE International Symposium on Mixed and Augmented Reality (ISMAR); 2024 Oct 21-25; Bellevue, USA. Piscataway: IEEE; 2024. p. 1-10. [DOI]

[17] Cleaver A, Tang D, Chen V, Sinapov J. HAVEN: A unity-based virtual robot environment to showcase HRI-based augmented reality. arXiv:2011.03464 [Preprint]. 2020. [DOI]

[18] Awais M, Naseer M, Khan S, Anwer RM, Cholakkal H, Shah M, et al. Foundation models defining a new era in vision: A survey and outlook. IEEE Trans Pattern Anal Mach Intell. 2025; 47(4): 2245-2264. [DOI]

[19] Lu P, Bansal H, Xia T, Liu J, Li C, Hajishirzi H, et al. MathVista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv:2310.02255 [Preprint]. 2023. [DOI]

[20] Li C, Gan Z, Yang Z, Yang J, Li L, Wang L, et al. Multimodal foundation models: From specialists to general-purpose assistants. Found Trends Comput Graph Vis. 2024; 16: 1-214. [DOI]

[21] Ma Y, Nordberg OE, Zhang Y, Rongve A, Bachinski M, Fjeld M. Understanding dementia speech: Towards an adaptive voice assistant for enhanced communication. In: Companion Proceedings of the 16th ACM SIGCHI Symposium on Engineering Interactive Computing Systems. New York: Association for Computing Machinery; 2024. p. 15-21. [DOI]

[22] Lin J, Ma Z, Gomez R, Nakamura K, He B, Li G. A review on interactive reinforcement learning from human social feedback. IEEE Access. 2020; 8: 120757-120765. [DOI]

[23] Kupcsik A, Hsu D, Lee WS. Learning dynamic robot-to-human object handover from human feedback. In: Bicchi A, Burgard W, editors. Robotics research. Cham: Springer; 2017. p. 161-176. [DOI]

[24] Nowak A, Zhang Y, Romanowski A, Fjeld M. Augmented reality with industrial process tomography: To support complex data analysis in 3D space. In: Adjunct Proceedings of the 2021 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2021 ACM International Symposium on Wearable Computers; 2021 Sep 21-26; USA. New York: Association for Computing Machinery; 2021. p. 56-58. [DOI]

[25] Park KB, Choi SH, Lee JY, Ghasemi Y, Mohammed M, Jeong H. Hands-free human-robot interaction using multimodal gestures and deep learning in wearable mixed reality. IEEE Access. 2021; 9: 55448-55464. [DOI]

[26] Qi J, Ma L, Cui Z, Yu Y. Computer vision-based hand gesture recognition for human-robot interaction: A review. Complex Intell Syst. 2024; 10(1): 1581-1606. [DOI]

[27] Lynch C, Wahid A, Tompson J, Ding T, Betker J, Baruch R, et al. Interactive language: Talking to robots in real time. IEEE Robot Autom Lett. 2024. [DOI]

[28] Ruhland K, Peters CE, Andrist S, Badler JB, Badler NI, Gleicher M, et al. A review of eye gaze in virtual agents, social robotics and HCI: Behaviour generation, user interaction and perception. Computer Graph Forum. 2015; 34(6): 299-326. [DOI]

[29] Zhang Y, Kassem K, Gong Z, Mo F, Ma Y, Kirjavainen E, et al. Human-centered AI technologies in human-robot interaction for social settings. In: Matviienko A, Niess J, Kosch T, editors. Proceedings of the International Conference on Mobile and Ubiquitous Multimedia; 2024 Dec 1-4; Stockholm, Sweden. New York: Association for Computing Machinery; 2024. p. 501-505. [DOI]

[30] Walker ME, Hedayati H, Szafir D. Robot teleoperation with augmented reality virtual surrogates. In: 2019 14th ACM/IEEE International Conference on Human-Robot Interaction (HRI); 2019 March 11-14; Daegu, Korea (South). Piscataway: IEEE; 2019. p. 202-210. [DOI]

[31] Dianatfar M, Pöysäri S, Latokartano J, Siltala N, Lanz M. Virtual reality-based safety training in human-robot collaboration scenario: User experiences testing. AIP Conf Proc. 2024; 2989(1): 020002. [DOI]

[32] Karpichev Y, Charter T, Hong J, Soufi Enayati AM, Honari H, Tamizi MG, et al. Extended reality for enhanced human-robot collaboration: A human-in-the-loop approach. In: 2024 33rd IEEE International Conference on Robot and Human Interactive Communication (ROMAN); 2024 Aug 26-30; Pasadena, USA. Piscataway: IEEE; 2024. p. 1991-1998. [DOI]

[33] Zhang B, Soh H. Large language models as zero-shot human models for human-robot interaction. In: 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS); 2023 Oct 1-5; Detroit, USA. Piscataway: IEEE; 2023. p. 7961-7968. [DOI]

[34] Wang J, Shi E, Hu H, Ma C, Liu Y, Wang X, et al. Large language models for robotics: Opportunities, challenges, and perspectives. J Autom Intell. 2025; 4(1): 52-64. [DOI]

[35] Duguleana M, Barbuceanu FG, Mogan G. Evaluating human-robot interaction during a manipulation experiment conducted in immersive virtual reality. In: Shumaker R, editor. Virtual and mixed reality-new trends. Heidelberg: Springer; 2011. p. 164-173. [DOI]

[36] Zhang Y, Xuan Y, Yadav R, Omrani A, Fjeld M. Playing with data: An augmented reality approach to interact with visualizations of industrial process tomography. In: Abdelnour Nocera J, Kristín Lárusdóttir M, Petrie H, Piccinno A, Winckler M, editors. Human-computer interaction-INTERACT 2023. Cham: Springer; 2023. p. 123-144. [DOI]

[37] Weistroffer V, Paljic A, Fuchs P, Hugues O, Chodacki JP, Ligot P, et al. Assessing the acceptability of human-robot co-presence on assembly lines: A comparison between actual situations and their virtual reality counterparts. In: The 23rd IEEE International Symposium on Robot and Human Interactive Communication; 2014 Aug 25-29; Edinburgh, UK. Piscataway: IEEE; 2014. p. 377-384. [DOI]

[38] Bayro A, Ghasemi Y, Jeong H. Subjective and objective analyses of collaboration and co-presence in a virtual reality remote environment. In: 2022 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW); 2022 Mar 12-16; Christchurch, New Zealand. Piscataway: IEEE; 2022. p. 485-487. [DOI]

[39] Zhang Y, Nowak A, Romanowski A, Fjeld M. An initial exploration of visual cues in head-mounted display augmented reality for book searching. In. p. 273-275. [DOI]

[40] Wang X, Shen L, Lee LH. A systematic review of XR-enabled remote human-robot interaction systems. ACM Comput Surv. 2025; 57(11): 1-37. [DOI]

[41] Tanaka Y, Ruiz CM. Immersive technologies for remote human-robot interaction: A systematic literature review. Eur J Emerg Comput Vis Nat Lang Process. 2024; 1(1): 1-20. Available from: https://parthenonfrontiers.com/index.php/ejecvnlp/article/view/74

[42] San Martin A, Kildal J, Lazkano E. Mixed reality representation of hazard zones while collaborating with a robot: Sense of control over own safety. Virtual Real. 2025; 29(1): 43. [DOI]

[43] Bolano G, Juelg C, Roennau A, Dillmann R. Transparent robot behavior using augmented reality in close human-robot interaction. In: 2019 28th IEEE International Conference on Robot and Human Interactive Communication (RO-MAN); 2019 Oct 14-18; New Delhi, India. Piscataway: IEEE; 2019. p. 1-7. [DOI]

[44] Walker M, Hedayati H, Lee J, Szafir D. Communicating robot motion intent with augmented reality. In: 2018 13th ACM/IEEE International Conference on Human-Robot Interaction (HRI); 2018 Mar 5-8; Chicago, USA. Piscataway: IEEE; 2018. p. 316-324. Available from: https://ieeexplore.ieee.org/document/9473546

[45] Ahmad Malik A, Brem A. Digital twins for collaborative robots: A case study in human-robot interaction. Robot Comput Integr Manuf. 2021; 68: 102092. [DOI]

[46] Xie J, Liu Y, Wang X, Fang S, Liu S. A new XR-based human‐robot collaboration assembly system based on industrial metaverse. J Manuf Syst. 2024; 74: 949-964. [DOI]

[47] Choi SH, Park KB, Roh DH, Lee JY, Ghasemi Y, Jeong H. An XR-based approach to safe human-robot collaboration. In: 2022 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW); 2022 Mar 12-16; Christchurch, New Zealand. Piscataway: IEEE; 2022. p. 481-482. [DOI]

[48] Pan M, Wong MO, Lam CC, Pan W. Integrating extended reality and robotics in construction: A critical review. Adv Eng Inform. 2024; 62: 102795. [DOI]

[49] Green SA, Billinghurst M, Chen X, Chase JG. Human robot collaboration: An augmented reality approach-a literature review and analysis. In: ASME 2007 International Design Engineering Technical Conferences and Computers and Information in Engineering Conference; 2007 Sep 4-7; Las Vegas, USA. New York: ASME; 2007. p. 117-126. [DOI]

[50] Higgins P, Kebe GY, Berlier A, Darvish K, Engel D, Ferraro F, et al. Towards making virtual human-robot interaction a reality. In: Proceedings of the 3rd International Workshop on Virtual, Augmented, and Mixed-Reality for Human-Robot Interactions (VAM-HRI); 2020 Mar 23; Cambridge, UK. New York: Association for Computing Machinery; 2021. Available from: https://vam-hri.github.io/previous/2021/papers/10_towards_making_virtual_human_robot_interaction_a_reality.pdf

[51] Murnane M, Higgins P, Saraf M, Ferraro F, Matuszek C, Engel D. A simulator for human-robot interaction in virtual reality. In: 2021 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW); 2021 Mar 27-Apr 1; Lisbon, Portugal. Piscataway: IEEE; 2021. p. 470-471. [DOI]

[52] Arntz A, Helgert A, Straßmann C, Eimler SC. Enhancing human-robot interaction research by using a virtual reality lab approach. In: 2024 IEEE International Conference on Artificial Intelligence and eXtended and Virtual Reality (AIxVR); 2024 Jan 17-19; Los Angeles, USA. Piscataway: IEEE; 2024. p. 340-344. [DOI]

[53] Surve S, Guo J, Menezes JC, Tate C, Jin Y, Walker J, et al. UnRealTHASC-a cyber-physical XR testbed for underwater real-time human autonomous systems collaboration. In: 2024 33rd IEEE International Conference on Robot and Human Interactive Communication (ROMAN); 2024 Aug 26-30; Pasadena, USA. Piscataway: IEEE; 2024. p. 196-203. [DOI]

[54] Bustamante S, Peters J, Scholkopf B, Grosse-Wentrup M, Jayaram V. ArmSym: A virtual human-robot interaction laboratory for assistive robotics. IEEE Trans Human-Mach Syst. 2021; 51(6): 568-577. [DOI]

[55] Villani V, Capelli B, Sabattini L. Use of virtual reality for the evaluation of human-robot interaction systems in complex scenarios. In: 2018 27th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN); 2018 Aug 27-31; Nanjing, China. Piscataway: IEEE; 2018. p. 422-427. [DOI]

[56] Fratczak P, Goh YM, Kinnell P, Soltoggio A, Justham L. Understanding human behaviour in industrial human-robot interaction by means of virtual reality. In: Proceedings of the Halfway to the Future Symposium 2019 Nov 19-20; Nottingham, United Kingdom. New York: Association for Computing Machinery; 2019. p. 1-7. [DOI]

[57] Yamamoto T, Abolhassani N, Jung S, Okamura AM, Judkins TN. Augmented reality and haptic interfaces for robot-assisted surgery. Int J Med Robot Comput Assist Surg. 2012; 8(1): 45-56. [DOI]

[58] Fang HC, Ong SK, Nee AYC. A novel augmented reality-based interface for robot path planning. Int J Interact Des Manuf (IJIDeM). 2014; 8(1): 33-42. [DOI]

[59] Hernández JD, Sobti S, Sciola A, Moll M, Kavraki LE. Increasing robot autonomy via motion planning and an augmented reality interface. IEEE Robot Autom Lett. 2020; 5(2): 1017-1023. [DOI]

[60] Kousi N, Stoubos C, Gkournelos C, Michalos G, Makris S. Enabling Human Robot Interaction in flexible robotic assembly lines: An Augmented Reality based software suite. Procedia CIRP. 2019; 81: 1429-1434. [DOI]

[61] Michalos G, Karagiannis P, Makris S, Tokçalar Ö, Chryssolouris G. Augmented reality (AR) applications for supporting human-robot interactive cooperation. Procedia CIRP. 2016; 41: 370-375. [DOI]

[62] Milgram P, Zhai S, Drascic D, Grodski J. Applications of augmented reality for human-robot communication. In: Proceedings of 1993 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS ‘93); 1993 Jul 26-30; Yokohama, Japan. Piscataway: IEEE; 1993. p. 1467-1472. [DOI]

[63] Malý I, Sedláček D, Leitão P. Augmented reality experiments with industrial robot in industry 4.0 environment. In: 2016 IEEE 14th International Conference on Industrial Informatics (INDIN); 2016 Jul 19-21; Poitiers, France. Piscataway: IEEE; 2016. p. 176-181. [DOI]

[64] Ostanin M, Yagfarov R, Devitt D, Akhmetzyanov A, Klimchik A. Multi robots interactive control using mixed reality. Int J Prod Res. 2021; 59(23): 7126-7138. [DOI]

[65] Fang B, Ding W, Sun F, Shan J, Wang X, Wang C, et al. Brain-computer interface integrated with augmented reality for human-robot interaction. IEEE Trans Cogn Dev Syst. 2023; 15(4): 1702-1711. [DOI]

[66] Guhl J, Hügle J, Krüger J. Enabling human-robot-interaction via virtual and augmented reality in distributed control systems. Procedia CIRP. 2018; 76: 167-170. [DOI]

[67] Wang X, Guo S, Xu Z, Zhang Z, Sun Z, Xu Y. A robotic teleoperation system enhanced by augmented reality for natural human-robot interaction. Cyborg Bionic Syst. 2024; 5: 0098. [DOI]

[68] Bischoff R, Kazi A. Perspectives on augmented reality based human-robot interaction with industrial robots. In: 2004 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS); 2004 Sep 28-Oct 2; Sendai, Japan. Piscataway: IEEE; 2004. p. 3226-3231. [DOI]

[69] Hedayati H, Walker M, Szafir D. Improving collocated robot teleoperation with augmented reality. In: Proceedings of the 2018 ACM/IEEE International Conference on Human-Robot Interaction; 2018 March 5-8; Chicago, USA. New York: Association for Computing Machinery; 2018. p. 78-86. [DOI]

[70] Choi SH, Park KB, Roh DH, Lee JY, Mohammed M, Ghasemi Y, et al. An integrated mixed reality system for safety-aware human-robot collaboration using deep learning and digital twin generation. Robot Comput Integr Manuf. 2022; 73: 102258. [DOI]

[71] Li C, Zheng P, Yin Y, Pang YM, Huo S. An AR-assisted Deep Reinforcement Learning-based approach towards mutual-cognitive safe human-robot interaction. Robot Comput Integr Manuf. 2023; 80: 102471. [DOI]

[72] Chacko SM, Kapila V. An augmented reality interface for human-robot interaction in unconstrained environments. In: 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS); 2019 Nov 3-8; Macau, China. Piscataway: IEEE; 2019. p. 3222-3228. [DOI]

[73] Ostanin M, Yagfarov R, Klimchik A. Interactive robots control using mixed reality. IFAC-PapersOnLine. 2019; 52(13): 695-700. [DOI]

[74] Manring L, Pederson J, Potts D, Boardman B, Mascarenas D, Harden T, et al. Augmented reality for interactive robot control. In: Dervilis N, editor. Special topics in structural dynamics & experimental techniques, volume 5. Cham: Springer; 2020. p. 11-18. [DOI]

[75] Buldu KB, Özdel S, Lau KHC, Wang M, Saad D, Schönborn S, et al. CUIfy the XR: An open-source package to embed LLM-powered conversational agents in XR. In: 2025 IEEE International Conference on Artificial Intelligence and eXtended and Virtual Reality (AIxVR); 2025 Jan 27-29; Lisbon, Portugal. Piscataway: IEEE; 2025. p. 192-197. [DOI]

[76] Konenkov M, Lykov A, Trinitatova D, Tsetserukou D. VR-GPT: Visual language model for intelligent virtual reality applications. arXiv:2405.11537 [Preprint]. 2024. [DOI]

[77] Afzal MZ, Ali SA, Stricker D, Eisert P, Hilsmann A, Perez-Marcos D, et al. Next generation XR systems: Large language models meet augmented and virtual reality. IEEE Comput Grap Appl. 2025; 45(1): 43-55. [DOI]

[78] Li X, Liu M, Zhang H, Yu C, Xu J, Wu H, et al. Vision-language foundation models as effective robot imitators. arXiv:2311.01378 [Preprint]. 2023. [DOI]

[79] Lakhnati Y, Pascher M, Gerken J. Exploring a GPT-based large language model for variable autonomy in a VR-based human-robot teaming simulation. Front Robot AI. 2024; 11: 1347538. [DOI]

[80] Zhang C, Chen J, Li J, Peng Y, Mao Z. Large language models for human-robot interaction: A review. Biomim Intell Robot. 2023; 3(4): 100131. [DOI]

[81] Wang Z, Giannopoulos E, Slater M, Peer A. Handshake: Realistic human-robot interaction in haptic enhanced virtual reality. Presence. 2011; 20(4): 371-392. [DOI]

[82] Wang I, Smith J, Ruiz J. Exploring virtual agents for augmented reality. In: Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems; 2019 May 4-9; Glasgow, UK. New York: Association for Computing Machinery; 2019. p. 1-12. [DOI]

[83] Suzuki R, Karim A, Xia T, Hedayati H, Marquardt N. Augmented reality and robotics: A survey and taxonomy for AR-enhanced human-robot interaction and robotic interfaces. In: CHI conference on human factors in computing systems; 2022 Apr 29-May 5; New Orleans, USA. New York: Association for Computing Machinery; 2022. p. 1-33. [DOI]

[84] Lee YK, Jung Y, Kang G, Hahn S. Developing social robots with empathetic non-verbal cues using large language models. arXiv:2308.16529 [Preprint]. 2023. [DOI]

[85] Darwesh L, Singh J, Marian M, Alexa E, Hindriks K, Baraka K. Exploring the effect of robotic embodiment and empathetic tone of LLMs on empathy elicitation. In: Social robotics. 16th International Conference, ICSR + AI 2024; 2024 Oct 23-26; Odense, Denmark. Singapore: Springer; 2024. p. 1-11. [DOI]

[86] Chen X, Ge J, Dai H, Zhou Q, Feng Q, Hu J, et al. Empathyagent: Can embodied agents conduct empathetic actions? arXiv:2503.16545 [Preprint]. 2025. [DOI]

[87] Walker M, Phung T, Chakraborti T, Williams T, Szafir D. Virtual, augmented, and mixed reality for human-robot interaction: A survey and virtual design element taxonomy. ACM Trans Hum-Robot Interact. 2023; 12(4): 1-39. [DOI]

[88] Groechel TR, Walker ME, Chang CT, Rosen E, Forde JZ. A tool for organizing key characteristics of virtual, augmented, and mixed reality for human-robot interaction systems: Synthesizing VAM-HRI trends and takeaways. IEEE Robot Automat Mag. 2022; 29(1): 35-44. [DOI]

[89] Peters C, Yang F, Saikia H, Li C, Skantze G. Towards the use of mixed reality for HRI design via virtual robots. In: 1st International Workshop on Virtual, Augmented, and Mixed Reality for HRI (VAM-HRI); 2020 Mar 23; Cambridge, UK. New York: Association for Computing Machinery; 2020. Available from: https://www.researchgate.net/publication/335756897

[90] Wang N, Pynadath DV, Unnikrishnan KV, Shankar S, Merchant C. Intelligent agents for virtual simulation of human-robot interaction. In: Shumaker R, Lackey S, editors. Virtual, augmented and mixed reality. Cham: Springer; 2015. p. 228-239. [DOI]

[91] Szafir D. Mediating human-robot interactions with virtual, augmented, and mixed reality. In: Chen JYC, Fragomeni G, editors. Virtual, augmented and mixed reality. Applications and case studies. Cham: Springer; 2019. p. 124-149. [DOI]

[92] Qiu S, Liu H, Zhang Z, Zhu Y, Zhu SC. Human-robot interaction in a shared augmented reality workspace. In: 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS); 2021 Oct 24-Jan 24; Las Vegas, USA. Piscataway: IEEE; 2020. p. 11413-11418. [DOI]

[93] Sabbella SR, Kaszuba S, Leotta F, Nardi D. Virtual reality applications for enhancing human-robot interaction: A gesture recognition perspective. In: Proceedings of the 23rd ACM International Conference on Intelligent Virtual Agents; 2023 Sep 19-22; Würzburg, Germany. New York: Association for Computing Machinery; 2023. p. 1-4. [DOI]

[94] Arntz A, Eimler SC, Hoppe HU. “The robot-arm talks back to me”-human perception of augmented human-robot collaboration in virtual reality. In: 2020 IEEE International Conference on Artificial Intelligence and Virtual Reality (AIVR); 2020 Dec 14-18; Utrecht, Netherlands. Piscataway: IEEE; 2020. p. 307-312. [DOI]

[95] Zhang Y, Omrani A, Yadav R, Fjeld M. Supporting visualization analysis in industrial process tomography by using augmented reality-a case study of an industrial microwave drying system. Sensors. 2021; 21(19): 6515. [DOI]

[96] Paiva A, Leite I, Boukricha H, Wachsmuth I. Empathy in virtual agents and robots: A survey. ACM Trans Interact Intell Syst. 2017; 7(3): 1-40. [DOI]

[97] Zhao S. Toward a taxonomy of copresence. Presence. 2003; 12(5): 445-455. [DOI]

[98] Szczurek KA, Prades RM, Matheson E, Rodriguez-Nogueira J, Di Castro M. Multimodal multi-user mixed reality human-robot interface for remote operations in hazardous environments. IEEE Access. 2023; 11: 17305-17333. [DOI]

[99] Simaan N, Taylor RH, Choset H. Intelligent surgical robots with situational awareness. Mech Eng. 2015; 137(9): S3-S6. [DOI]

[100] Roldán JJ, Peña-Tapia E, Martín-Barrio A, Olivares-Méndez MA, Del Cerro J, Barrientos A. Multi-robot interfaces and operator situational awareness: Study of the impact of immersion and prediction. Sensors. 2017; 17(8): 1720. [DOI]

[101] Zhang Y, Nowak A, Rao G, Romanowski A, Fjeld M. Is industrial tomography ready for augmented reality? A need-finding study of how augmented reality can be adopted by industrial tomography experts. In: Chen JYC, Fragomeni G, editors. Virtual, augmented and mixed reality. Cham: Springer; 2023. p. 523-535. [DOI]

[102] Van Haastregt J, Welle MC, Zhang Y, Kragic D. Puppeteer your robot: Augmented reality leader-follower teleoperation. In: 2024 IEEE-RAS 23rd International Conference on Humanoid Robots (Humanoids); 2024 Nov 22-24; Nancy, France. Piscataway: IEEE; 2024. p. 1019-1026. [DOI]

[103] Zhang Y, Orthmann B, Welle MC, Van Haastregt J, Kragic D. LLM-driven augmented reality puppeteer: Controller-free voice-commanded robot teleoperation. In: Coman A, Vasilache S, editors. Social computing and social media. Cham: Springer; 2025. p. 97-112. [DOI]

[104] Zhang Y, Orthmann B, Ji S, Welle M, Van Haastregt J, Kragic D. Multimodal “puppeteer”: Exploring robot teleoperation via virtual counterpart with LLM-driven voice and gesture interaction in augmented reality. arXiv:2506.13189 [Preprint]. 2025. [DOI]

[105] Kim CY, Lee CP, Mutlu B. Understanding large-language model (LLM)-powered human-robot interaction. In: Proceedings of the 2024 ACM/IEEE International Conference on Human-Robot Interaction; 2024 Mar 11-15; Boulder, USA. New York: Association for Computing Machinery; 2024. p. 371-380. [DOI]

[106] Kawaharazuka K, Matsushima T, Gambardella A, Guo J, Paxton C, Zeng A. Real-world robot applications of foundation models: A review. Adv Robot. 2024; 38(18): 1232-1254. [DOI]

[107] Costa GM, Petry MR, Moreira AP. Augmented reality for human-robot collaboration and cooperation in industrial applications: A systematic literature review. Sensors. 2022; 22(7): 2725. [DOI]

[108] Robins B, Dautenhahn K, Te Boekhorst R, Billard A. Robotic assistants in therapy and education of children with autism: Can a small humanoid robot help encourage social interaction skills? Univ Access Inf Soc. 2005; 4(2): 105-120. [DOI]

[109] Shamsuddin S, Yussof H, Ismail L, Hanapiah FA, Mohamed S, Ali Piah H, et al. Initial response of autistic children in human-robot interaction therapy with humanoid robot NAO. In: 2012 IEEE 8th International Colloquium on Signal Processing and Its Applications; 2012 Mar 23-25; Malacca, Malaysia. Piscataway: IEEE; 2012. p. 188-193. [DOI]

[110] Mohebbi A. Human-robot interaction in rehabilitation and assistance: A review. Curr Robot Rep. 2020; 1(3): 131-144. [DOI]

[111] Leite I, Pereira A, Mascarenhas S, Martinho C, Prada R, Paiva A. The influence of empathy in human-robot relations. Int J Hum Comput Stud. 2013; 71(3): 250-260. [DOI]

[112] Baytas MA, Çay D, Zhang Y, Obaid M, Yantaç AE, Fjeld M. The design of social drones: A review of studies on autonomous flyers in inhabited environments. In: Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems; 2019 May 4-9; Glasgow, UK. New York: Association for Computing Machinery; 2019. p. 1-13. [DOI]

[113] Shin D. Empathy and embodied experience in virtual environment: To what extent can virtual reality stimulate empathy and embodied experience? Comput Hum Behav. 2018; 78: 64-73. [DOI]

[114] Rueda J, Lara F. Virtual reality and empathy enhancement: Ethical aspects. Front Robot AI. 2020; 7: 506984. [DOI]

[115] Lv N, Gong J. The application of virtual reality technology in the efficiency optimisation of students’ online interactive learning. Int J Contin Eng Educ Life Long Learn. 2022; 32(1): 35. [DOI]

[116] Zhang Y, Ma Y, Fu D, Portales SZ, Fjeld M, Kragic D. Personalizing emotion-aware conversational agents? Exploring user traits-driven conversational strategies for enhanced interaction. arXiv:2511.06954 [Preprint]. 2025. [DOI]

[117] Nakazawa A, Iwamoto M, Kurazume R, Nunoi M, Kobayashi M, Honda M. Augmented reality-based affective training for improving care communication skill and empathy. PLoS One. 2023; 18(7): e0288175. [DOI]

[118] Billinghurst M. Using augmented reality to create empathic experiences. In: Proceedings of the 19th International Conference on Intelligent User Interfaces; 2014 Feb 24-27; Haifa, Israel. New York: Association for Computing Machinery; 2014. p. 5-6. [DOI]

[119] Ma Y, Zhang Y, Bachinski M, Fjeld M. Emotion-aware voice assistants: Design, implementation, and preliminary insights. In: Proceedings of the Eleventh International Symposium of Chinese CHI; 2023 Nov 13-16; Denpasar, Indonesia. New York: Association for Computing Machinery; 2023. p. 527-532. [DOI]

[120] Sorin V, Brin D, Barash Y, Konen E, Charney A, Nadkarni G, et al. Large language models and empathy: Systematic review. J Med Internet Res. 2024; 26: e52597. [DOI]

[121] Hasan M, Ozel C, Potter S, Hoque E. SAPIEN: Affective virtual agents powered by large language models. In: 2023 11th International Conference on Affective Computing and Intelligent Interaction Workshops and Demos (ACIIW); 2023 Sep 10-13; Cambridge, USA. Piscataway: IEEE; 2023. p. 1-3. [DOI]

[122] Shih MT, Hsu MY, Lee SC. Empathy-GPT: Leveraging large language models to enhance emotional empathy and user engagement in embodied conversational agents. In. p. 1-3. [DOI]

[123] Lee KM, Jung Y, Kim J, Kim SR. Are physically embodied social agents better than disembodied social agents? The effects of physical embodiment, tactile interaction, and people’s loneliness in human-robot interaction. Int J Hum Comput Stud. 2006; 64(10): 962-973. [DOI]

[124] Wainer J, Feil-seifer DJ, Shell DA, Mataric MJ. The role of physical embodiment in human-robot interaction. In: ROMAN 2006-the 15th IEEE International Symposium on Robot and Human Interactive Communication; 2006 Sep 6-8; Hatfield, UK. Piscataway: IEEE; 2006. p. 117-122. [DOI]

[125] Kilteni K, Groten R, Slater M. The sense of embodiment in virtual reality. Presence Teleop Virt Environ. 2012; 21(4): 373-387. [DOI]

[126] Han Z, Zhu Y, Phan A, Garza FS, Castro A, Williams T. Crossing reality: Comparing physical and virtual robot deixis. In: Proceedings of the 2023 ACM/IEEE International Conference on Human-Robot Interaction; 2023 Mar 13-16; Stockholm, Sweden. New York: Association for Computing Machinery; 2023. p. 152-161. [DOI]

[127] Wolf E, Fiedler ML, Döllinger N, Wienrich C, Latoschik ME. Exploring presence, avatar embodiment, and body perception with a holographic augmented reality mirror. In: 2022 IEEE Conference on Virtual Reality and 3D User Interfaces (VR); 2022 Mar 12-16; Piscataway: IEEE; 2022. p. 350-359. [DOI]

[128] Nimcharoen C, Zollmann S, Collins J, Regenbrecht H. Is that me?-Embodiment and body perception with an augmented reality mirror. In: 2018 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct); 2018 Oct 16-20; Munich, Germany. Piscataway: IEEE; 2018. p. 158-163. [DOI]

[129] Genay A, Lecuyer A, Hachet M. Being an avatar “for real”: A survey on virtual embodiment in augmented reality. IEEE Trans Vis Comput Graph. 2022; 28(12): 5071-5090. [DOI]

[130] Zhang Y, Nowak A, Romanowski A, Fjeld M. On-site or remote working? An initial solution on how COVID-19 pandemic may impact augmented reality users. In: Proceedings of the 2022 International Conference on Advanced Visual Interfaces; 2022 Jun 6-10; Frascati, Italy. New York: Association for Computing Machinery; 2022. p. 1-3. [DOI]

[131] Ghafurian M, Budnarain N, Hoey J. Improving humanness of virtual agents and users’ cooperation through emotions. IEEE Trans Affect Comput. 2023; 14(2): 1461-1471. [DOI]

[132] Lisetti C, Amini R, Yasavur U, Rishe N. I can help you change! An empathic virtual agent delivers behavior change health interventions. ACM Trans Manag Inf Syst. 2013; 4(4): 1-28. [DOI]

[133] Parmar D, Olafsson S, Utami D, Murali P, Bickmore T. Designing empathic virtual agents: Manipulating animation, voice, rendering, and empathy to create persuasive agents. Auton Agent Multi Agent Syst. 2022; 36(1): 17. [DOI]

[134] Zhang Y, Nowak A, Romanowski A, Fjeld M. Virtuality or physicality? Supporting memorization through augmented reality gamification. In: Companion Proceedings of the 2023 ACM SIGCHI Symposium on Engineering Interactive Computing Systems; 2023 Jun 27-30; Swansea, United Kingdom. New York: Association for Computing Machinery; 2023. p. 53-58. [DOI]

[135] Wallace B, van Otterdijk M, Zhang Y, Rajabi N, Marin-Bucio D, Kragic D, et al. Imitation or innovation? Translating features of expressive motion from humans to robots. In: Proceedings of the 12th International Conference on Human-Agent Interaction; 2024 Nov 24-27; Swansea, United Kingdom. New York: Association for Computing Machinery; 2024. p. 296-304. [DOI]

[136] Suzuki R, Gonzalez-Franco M, Sra M, Lindlbauer D. XR and AI: AI-enabled virtual, augmented, and mixed reality. In: Adjunct Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology; 2023 Oct 29-Nov 1; San Francisco, USA. New York: Association for Computing Machinery; 2023. p. 1-3. [DOI]

[137] Van de Merwe DB, Van Maanen L, Ter Haar FB, Van Dijk RJE, Hoeba N, Van der Stap N. Human-robot interaction during virtual reality mediated teleoperation: How environment information affects spatial task performance and operator situation awareness. In: Chen JYC, Fragomeni G, editors. Virtual, augmented and mixed reality. Applications and case studies. Cham: Springer; 2019. p. 163-177. [DOI]

[138] Dini A, Murko C, Yahyanejad S, Augsdörfer U, Hofbaur M, Paletta L. Measurement and prediction of situation awareness in human-robot interaction based on a framework of probabilistic attention. In: 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS); 2017 Sep 24-28; Vancouver, Canada. Piscataway: IEEE; 2017. p. 4354-4361. [DOI]

[139] Senaratne H, Tian L, Sikka P, Williams J, Howard D, Kulić D, et al. A framework for dynamic situational awareness in human robot teams: An interview study. ACM Trans Hum-Robot Interact. 2025; 15: 1-37. [DOI]

[140] Müller M, Ruppert T, Jazdi N, Weyrich M. Self-improving situation awareness for human-robot-collaboration using intelligent Digital Twin. J Intell Manuf. 2024; 35(5): 2045-2063. [DOI]

[141] Lopes JC, Lopes RP. Computer vision in augmented, virtual, mixed and extended reality environments: A bibliometric review. Vis Inform. 2024; 8(4): 13-22. [DOI]