Towards spatial computing: recent advances in multimodal natural interaction for Extended Reality headsets

Abstract

With the widespread adoption of Extended Reality (XR) headsets, spatial computing technologies are gaining increasing attention. Spatial computing enables interaction with virtual elements through natural input methods such as eye tracking, hand gestures, and voice commands, thus placing natural human-computer interaction at its core. While previous surveys have reviewed conventional XR interaction techniques, recent advancements in natural interaction, particularly driven by artificial intelligence (AI) and large language models (LLMs), have introduced new paradigms and technologies. In this paper, we review research on multimodal natural interaction for wearable XR, focusing on papers published since 2022 in six top venues: ACM CHI, UIST, IMWUT (Ubicomp), IEEE VR, ISMAR, and TVCG. We classify and analyze these studies based on application scenarios, operation types, and interaction modalities. This analysis provides a structured framework for understanding how researchers are designing advanced natural interaction techniques in XR. Based on these findings, we discuss the challenges in natural interaction techniques and suggest potential directions for future research. This review provides valuable insights for researchers aiming to design natural and efficient interaction systems for XR, ultimately contributing to the advancement of spatial computing.

Graphical abstract

Keywords

extended reality / multimodal / natural interaction / eye / hand / speech

References

[1] Spittle B, Frutos-Pascual M, Creed C, Williams I . A review of interaction techniques for immersive environments. IEEE Transactions on Visualization and Computer Graphics, 2023, 29( 9): 3900–3921

[2] Aros M, Tyger C L, Chaparro B S. Unraveling the meta quest 3: an out-of-box experience of the future of mixed reality headsets. In: Proceedings of the 26th International Conference on Human-Computer Interaction. 2024, 3−8

[3] Jing A, Lee G A, Billinghurst M. Using speech to visualise shared gaze cues in MR remote collaboration. In: Proceedings of 2022 IEEE Conference on Virtual Reality and 3D User Interfaces. 2022, 250−259

[4] Quere C, Menin A, Julien R, Wu H Y, Winckler M. HandyNotes: using the hands to create semantic representations of contextually aware real-world objects. In: Proceedings of 2024 IEEE Conference Virtual Reality and 3D User Interfaces. 2024, 265−275

[5] Barteit S, Lanfermann L, Bärnighausen T, Neuhann F, Beiersmann C . Augmented, mixed, and virtual reality-based head-mounted devices for medical education: systematic review. JMIR Serious Games, 2021, 9( 3): e29080

[6] Hrycak C, Lewakis D, Krüger J. Investigating the apple vision pro spatial computing platform for GPU-based volume visualization. In: Proceedings of 2024 IEEE Visualization and Visual Analytics. 2024, 181−185

[7] Yenduri G, M R, Maddikunta P K R, Gadekallu T R, Jhaveri R H, Bandi A, Chen J, Wang W, Shirawalmath A A, Ravishankar R, Wang W. Spatial computing: concept, applications, challenges and future directions. 2024, arXiv preprint arXiv: 2402.07912

[8] Hackl C, Cronin I. Spatial Computing: An AI-Driven Business Revolution. Hoboken: John Wiley & Sons, 2024

[9] Bao Y, Wang J, Wang Z, Lu F. Exploring 3D interaction with gaze guidance in augmented reality. In: Proceedings of 2023 IEEE Conference Virtual Reality and 3D User Interfaces. 2023, 22−32

[10] Bérard F, Ip J, Benovoy M, El-Shimy D, Blum J R, Cooperstock J R. Did ”minority report” get it wrong? Superiority of the mouse over 3D input devices in a 3D placement task. In: Proceedings of the 12th IFIP TC 13 International Conference on Human-Computer Interaction. 2009, 400−414

[11] Matthews B J, Thomas B H, Von Itzstein G S, Smith R T. Shape aware haptic retargeting for accurate hand interactions. In: Proceedings of 2022 IEEE Conference on Virtual Reality and 3D User Interfaces. 2022, 625−634

[12] Giunchi D, Numan N, Gatti E, Steed A. DreamCodeVR: towards democratizing behavior design in virtual reality with speech-driven programming. In: Proceedings of 2024 IEEE Conference Virtual Reality and 3D User Interfaces. 2024, 579−589

[13] Huang Y, Yang L, Chen G, Zhang H, Lu F, Sato Y . Matching compound prototypes for few-shot action recognition. International Journal of Computer Vision, 2024, 132( 9): 3977–4002

[14] Wang Z, Gu X, Lu F. DEAMP: dominant-eye-aware foveated rendering with multi-parameter optimization. In: Proceedings of 2023 IEEE International Symposium on Mixed and Augmented Reality. 2023, 632−641

[15] Chaconas N, Höllerer T. An evaluation of bimanual gestures on the Microsoft HoloLens. In: Proceedings of 2018 IEEE Conference on Virtual Reality and 3D User Interfaces. 2018, 33−40

[16] Hincapié-Ramos J D, Guo X, Moghadasian P, Irani P. Consumed endurance: a metric to quantify arm fatigue of mid-air interactions. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 2014, 1063−1072

[17] Cao Y, Kazi R H, Wei L Y, Aneja D, Xia H. Elastica: adaptive live augmented presentations with elastic mappings across modalities. In: Proceedings of 2024 CHI Conference on Human Factors in Computing Systems. 2024, 599

[18] De La Torre F, Fang C M, Huang H, Banburski-Fahey A, Fernandez J A, Lanier J. LLMR: real-time prompting of interactive worlds using large language models. In: Proceedings of 2024 CHI Conference on Human Factors in Computing Systems. 2024, 600

[19] Sidenmark L, Parent M, Wu C H, Chan J, Glueck M, Wigdor D, Grossman T, Giordano M . Weighted pointer: error-aware gaze-based interaction through fallback modalities. IEEE Transactions on Visualization and Computer Graphics, 2022, 28( 11): 3585–3595

[20] Lystbæk M N, Rosenberg P, Pfeuffer K, Grønbæk J E, Gellersen H . Gaze-hand alignment: combining eye gaze and mid-air pointing for interacting with menus in augmented reality. Proceedings of the ACM on Human-Computer Interaction, 2022, 6( ETRA): 145

[21] Li J N, Xu Y, Grossman T, Santosa S, Li M. OmniActions: predicting digital actions in response to real-world multimodal sensory inputs with LLMs. In: Proceedings of 2024 CHI Conference on Human Factors in Computing Systems. 2024

[22] Wang H, Dong X, Chen Z, Shi B E. Hybrid gaze/EEG brain computer interface for robot arm control on a pick and place task. In: Proceedings of the 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society. 2015, 1476−1479

[23] Tran T T M, Brown S, Weidlich O, Billinghurst M, Parker C . Wearable augmented reality: research trends and future directions from three major venues. IEEE Transactions on Visualization and Computer Graphics, 2023, 29( 11): 4782–4793

[24] Hertel J, Karaosmanoglu S, Schmidt S, Bräker J, Semmann M, Steinicke F. A taxonomy of interaction techniques for immersive augmented reality based on an iterative literature review. In: Proceedings of 2021 IEEE International Symposium on Mixed and Augmented Reality. 2021, 431−440

[25] Pirker J, Dengel A . The potential of 360° virtual reality videos and real VR for education—A literature review. IEEE Computer Graphics and Applications, 2021, 41( 4): 76–89

[26] Katona J . A review of human−computer interaction and virtual reality research fields in cognitive InfoCommunications. Applied Sciences, 2021, 11( 6): 2646

[27] Zhang Y, Wang Z, Zhang J, Shan G, Tian D . A survey of immersive visualization: focus on perception and interaction. Visual Informatics, 2023, 7( 4): 22–35

[28] Guan H, Song C, Zhang Z . GRAMO: geometric resampling augmentation for monocular 3D object detection. Frontiers of Computer Science, 2024, 18( 5): 185706

[29] Liu F, Zheng Z, Shi Y, Tong Y, Zhang Y . A survey on federated learning: a perspective from multi-party computation. Frontiers of Computer Science, 2024, 18( 1): 181336

[30] Tang J, Song R, Huang Y, Gao S, Yu Z . Semantic-aware entity alignment for low resource language knowledge graph. Frontiers of Computer Science, 2024, 18( 4): 184319

[31] Li J, Lei Y, Bian Y, Cheng D, Ding Z, Jiang C . RA-CFGPT: Chinese financial assistant with retrieval-augmented large language model. Frontiers of Computer Science, 2024, 18( 5): 185350

[32] Lu F, Zhao Q . Towards cobodied/symbodied AI: concept and eight scientific and technical problems. SCIENTIA SINICA Informationis, 2025, 55( 2): 444–448

[33] Ghamandi R K, Hmaiti Y, Nguyen T T, Ghasemaghaei A, Kattoju R K, Taranta E M, LaViola J J. What and how together: a taxonomy on 30 years of collaborative human-centered XR tasks. In: Proceedings of 2023 IEEE International Symposium on Mixed and Augmented Reality. 2023, 322−335

[34] Chowdhury S, Ullah A K M A, Pelmore N B, Irani P, Hasan K. WriArm: leveraging wrist movement to design wrist+arm based teleportation in VR. In: Proceedings of 2022 IEEE International Symposium on Mixed and Augmented Reality. 2022, 317−325

[35] Schmitz M, Günther S, Schön D, Müller F. Squeezy-feely: investigating lateral thumb-index pinching as an input modality. In: Proceedings of 2022 CHI Conference on Human Factors in Computing Systems. 2022, 61

[36] Ban R, Matsumoto K, Narumi T, Kuzuoka H. Wormholes in VR: teleporting hands for flexible passive haptics. In: Proceedings of 2022 IEEE International Symposium on Mixed and Augmented Reality. 2022, 748−757

[37] Yu D, Zhou Q, Dingler T, Velloso E, Gonçalves J. Blending on-body and mid-air interaction in virtual reality. In: Proceedings of 2022 IEEE International Symposium on Mixed and Augmented Reality. 2022, 637−646

[38] Lee G, Healey J, Manocha D. VRDoc: gaze-based interactions for VR reading experience. In: Proceedings of 2022 IEEE International Symposium on Mixed and Augmented Reality. 2022, 787−796

[39] Choi M, Sakamoto D, Ono T. Kuiper belt: utilizing the ”out-of-natural angle” region in the eye-gaze interaction for virtual reality. In: Proceedings of 2022 CHI Conference on Human Factors in Computing Systems. 2022, 357

[40] Kim T, Ham A, Ahn S, Lee G. Lattice menu: a low-error gaze-based marking menu utilizing target-assisted gaze gestures on a lattice of visual anchors. In: Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems. 2022, 277

[41] Yi X, Lu Y, Cai Z, Wu Z, Wang Y, Shi Y. GazeDock: gaze-only menu selection in virtual reality using auto-triggering peripheral menu. In: Proceedings of 2022 IEEE Conference on Virtual Reality and 3D User Interfaces. 2022, 832−842

[42] Wang Z, Zhao Y, Lu F . Gaze-vergence-controlled see-through vision in augmented reality. IEEE Transactions on Visualization and Computer Graphics, 2022, 28( 11): 3843–3853

[43] Chen T, Li T, Yang X, Zhu K . EFRing: enabling thumb-to-index-finger microgesture interaction through electric field sensing using single smart ring. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 2022, 6( 4): 161

[44] Sendhilnathan N, Zhang T, Lafreniere B, Grossman T, Jonker T R. Detecting input recognition errors and user errors using gaze dynamics in virtual reality. In: Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology. 2022, 38

[45] Liao J, Karim A, Jadon S S, Kazi R H, Suzuki R. RealityTalk: real-time speech-driven augmented presentation for AR live storytelling. In: Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology. 2022, 17

[46] Yi X, Qiu L, Tang W, Fan Y, Li H, Shi Y. DEEP: 3D gaze pointing in virtual reality leveraging eyelid movement. In: Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology. 2022, 3

[47] Xu W, Meng X, Yu K, Sarcar S, Liang H N. Evaluation of text selection techniques in virtual reality head-mounted displays. In: Proceedings of 2022 IEEE International Symposium on Mixed and Augmented Reality. 2022, 131−140

[48] Shen X, Yan Y, Yu C, Shi Y . ClenchClick: hands-free target selection method leveraging teeth-clench for augmented reality. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 2022, 6( 3): 139

[49] Meng X, Xu W, Liang H N. An exploration of hands-free text selection for virtual reality head-mounted displays. In: Proceedings of 2022 IEEE International Symposium on Mixed and Augmented Reality. 2022, 74−81

[50] Das S, Nasser A, Hasan K. FingerButton: enabling controller-free transitions between real and virtual environments. In: Proceedings of 2023 IEEE International Symposium on Mixed and Augmented Reality. 2023, 533−542

[51] Zhu F, Sidenmark L, Sousa M, Grossman T. PinchLens: applying spatial magnification and adaptive control-display gain for precise selection in virtual reality. In: Proceedings of 2023 IEEE International Symposium on Mixed and Augmented Reality. 2023, 1221−1230

[52] Song Z, Dudley J J, Kristensson P O . HotGestures: complementing command selection and use with delimiter-free gesture-based shortcuts in virtual reality. IEEE Transactions on Visualization and Computer Graphics, 2023, 29( 11): 4600–4610

[53] Tseng W J, Huron S, Lecolinet E, Gugenheimer J. FingerMapper: mapping finger motions onto virtual arms to enable safe virtual reality interaction in confined spaces. In: Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems. 2023, 874

[54] Chen Y S, Hsieh C E, Jie M T Y, Han P H, Hung Y P. Leap to the eye: implicit gaze-based interaction to reveal invisible objects for virtual environment exploration. In: Proceedings of 2023 IEEE International Symposium on Mixed and Augmented Reality. 2023, 214−222

[55] Sidenmark L, Clarke C, Newn J, Lystbæk M N, Pfeuffer K, Gellersen H. Vergence matching: inferring attention to objects in 3D environments for gaze-assisted selection. In: Proceedings of 2023 CHI Conference on Human Factors in Computing Systems. 2023, 257

[56] Wagner U, Lystbæk M N, Manakhov P, Grønbæk J E S, Pfeuffer K, Gellersen H. A fitts’ law study of gaze-hand alignment for selection in 3D user interfaces. In: Proceedings of 2023 CHI Conference on Human Factors in Computing Systems. 2023, 252

[57] Shi R, Wei Y, Qin X, Hui P, Liang H N . Exploring gaze-assisted and hand-based region selection in augmented reality. Proceedings of the ACM on Human-Computer Interaction, 2023, 7( ETRA): 160

[58] Caillet A C, Goguey A, Nigay L. 3D selection in mixed reality: designing a two-phase technique to reduce fatigue. In: Proceedings of 2023 IEEE International Symposium on Mixed and Augmented Reality. 2023, 800−809

[59] Chen X, Guo D, Feng L, Chen B, Liu W. Compass+ring: a multimodal menu to improve interaction performance and comfortability in one-handed scenarios. In: Proceedings of 2023 IEEE International Symposium on Mixed and Augmented Reality. 2023, 473−482

[60] Wei Y, Shi R, Yu D, Wang Y, Li Y, Yu L, Liang H N. Predicting gaze-based target selection in augmented reality headsets based on eye and head endpoint distributions. In: Proceedings of 2023 CHI Conference on Human Factors in Computing Systems. 2023, 283

[61] Hou B J, Newn J, Sidenmark L, Khan A A, Bækgaard P, Gellersen H. Classifying head movements to separate head-gaze and head gestures as distinct modes of input. In: Proceedings of 2023 CHI Conference on Human Factors in Computing Systems. 2023, 253

[62] Yan Y, Liu H, Shi Y, Wang J, Guo R, Li Z, Xu X, Yu C, Wang Y, Shi Y . ConeSpeech: exploring directional speech interaction for multi-person remote communication in virtual reality. IEEE Transactions on Visualization and Computer Graphics, 2023, 29( 5): 2647–2657

[63] Sindhupathiraja S R, Ullah A K M A, Delamare W, Hasan K. Exploring bi-manual teleportation in virtual reality. In: Proceedings of 2024 IEEE Conference Virtual Reality and 3D User Interfaces. 2024, 754−764

[64] Dupré C, Appert C, Rey S, Saidi H, Pietriga E. TriPad: touch input in AR on ordinary surfaces with hand tracking only. In: Proceedings of 2024 CHI Conference on Human Factors in Computing Systems. 2024, 754

[65] Orlosky J, Liu C, Sakamoto K, Sidenmark L, Mansour A. EyeShadows: peripheral virtual copies for rapid gaze selection and interaction. In: Proceedings of 2024 IEEE Conference Virtual Reality and 3D User Interfaces. 2024, 681−689

[66] Zhang C, Chen T, Shaffer E, Soltanaghai E. FocusFlow: 3D gaze-depth interaction in virtual reality leveraging active visual depth manipulation. In: Proceedings of 2024 CHI Conference on Human Factors in Computing Systems. 2024, 372

[67] Turkmen R, Gelmez Z E, Batmaz A U, Stuerzlinger W, Asente P, Sarac M, Pfeuffer K, Machuca M D B. EyeGuide & EyeConGuide: gaze-based visual guides to improve 3D sketching systems. In: Proceedings of 2024 CHI Conference on Human Factors in Computing Systems. 2024, 178

[68] Zenner A, Karr C, Feick M, Ariza O, Krüger A. Beyond the blink: investigating combined saccadic & blink-suppressed hand redirection in virtual reality. In: Proceedings of 2024 CHI Conference on Human Factors in Computing Systems. 2024, 750

[69] Rodriguez R, Sullivan B T, Barrera Machuca M D, Batmaz A U, Tornatzky C, Ortega F R. An artists’ perspectives on natural interactions for virtual reality 3D sketching. In: Proceedings of 2024 CHI Conference on Human Factors in Computing Systems. 2024, 163

[70] Lai Y, Sun M, Li Z. GazePuffer: hands-free input method leveraging puff cheeks for VR. In: Proceedings of 2024 IEEE Conference Virtual Reality and 3D User Interfaces. 2024, 331−341

[71] Marquardt A, Steininger M, Trepkowski C, Weier M, Kruijff E. Selection performance and reliability of eye and head gaze tracking under varying light conditions. In: Proceedings of 2024 IEEE Conference Virtual Reality and 3D User Interfaces. 2024, 546−556

[72] Pei S, Chen A, Lee J, Zhang Y. Hand interfaces: using hands to imitate objects in AR/VR for expressive interactions. In: Proceedings of 2022 CHI Conference on Human Factors in Computing Systems. 2022, 429

[73] Satriadi K A, Smiley J, Ens B, Cordeil M, Czauderna T, Lee B, Yang Y, Dwyer T, Jenny B. Tangible globes for data visualisation in augmented reality. In: Proceedings of 2022 CHI Conference on Human Factors in Computing Systems. 2022, 505

[74] Xu X, Zhou Y, Shao B, Feng G, Yu C . GestureSurface: VR sketching through assembling scaffold surface with non-dominant hand. IEEE Transactions on Visualization and Computer Graphics, 2023, 29( 5): 2499–2507

[75] Zhan L, Xiong T, Zhang H, Guo S, Chen X, Gong J, Lin J, Qin Y . TouchEditor: interaction design and evaluation of a flexible touchpad for text editing of head-mounted displays in speech-unfriendly environments. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 2023, 7( 4): 198

[76] Williams A S, Ortega F R . The impacts of referent display on gesture and speech elicitation. IEEE Transactions on Visualization and Computer Graphics, 2022, 28( 11): 3885–3895

[77] Deng C L, Sun L, Zhou C, Kuai S G . Dual-gain mode of head-gaze interaction improves the efficiency of object positioning in a 3D virtual environment. International Journal of Human−Computer Interaction, 2024, 40( 8): 2067–2082

[78] Hombeck J, Voigt H, Heggemann T, Datta R R, Lawonn K. Tell me where to go: voice-controlled hands-free locomotion for virtual reality systems. In: Proceedings of 2023 IEEE Conference Virtual Reality and 3D User Interfaces. 2023, 123−134

[79] Mortezapoor S, Vasylevska K, Vonach E, Kaufmann H. CoboDeck: a large-scale haptic VR system using a collaborative mobile robot. In: Proceedings of 2023 IEEE Conference Virtual Reality and 3D User Interfaces. 2023, 297−307

[80] Sin Z P T, Jia Y, Li R C, Va Leong H, Li Q, Ng P H F. Illumotion: an optical-illusion-based VR locomotion technique for long-distance 3D movement. In: Proceedings of 2024 IEEE Conference Virtual Reality and 3D User Interfaces. 2024, 924−934

[81] Wang Z, Yuan L P, Wang L, Jiang B, Zeng W. VirtuWander: enhancing multi-modal interaction for virtual tour guidance through large language models. In: Proceedings of 2024 CHI Conference on Human Factors in Computing Systems. 2024, 612

[82] Kang S, Jeong J, Lee G A, Kim S H, Yang H J, Kim S. The RayHand navigation: a virtual navigation method with relative position between hand and gaze-ray. In: Proceedings of 2024 CHI Conference on Human Factors in Computing Systems. 2024, 634

[83] Lee H S, Weidner F, Sidenmark L, Gellersen H. Snap, pursuit and gain: virtual reality viewport control by gaze. In: Proceedings of 2024 CHI Conference on Human Factors in Computing Systems. 2024, 375

[84] Stemasov E, Wagner T, Gugenheimer J, Rukzio E. ShapeFindAR: exploring in-situ spatial search for physical artifact retrieval using mixed reality. In: Proceedings of 2022 CHI Conference on Human Factors in Computing Systems. 2022, 292

[85] Song Z, Dudley J J, Kristensson P O. Efficient special character entry on a virtual keyboard by hand gesture-based mode switching. In: Proceedings of 2022 IEEE International Symposium on Mixed and Augmented Reality. 2022, 864−871

[86] He Z, Lutteroth C, Perlin K. TapGazer: text entry with finger tapping and gaze-directed word selection. In: Proceedings of 2022 CHI Conference on Human Factors in Computing Systems. 2022, 337

[87] Shen J, Dudley J J, Kristensson P O . Fast and robust mid-air gesture typing for AR headsets using 3D trajectory decoding. IEEE Transactions on Visualization and Computer Graphics, 2023, 29( 11): 4622–4632

[88] Zhao M, Pierce A M, Tan R, Zhang T, Wang T, Jonker T R, Benko H, Gupta A. Gaze speedup: eye gaze assisted gesture typing in virtual reality. In: Proceedings of the 28th International Conference on Intelligent User Interfaces. 2023, 595−606

[89] Cui W, Liu R, Li Z, Wang Y, Wang A, Zhao X, Rashidian S, Baig F, Ramakrishnan I V, Wang F, Bi X. GlanceWriter: writing text by glancing over letters with gaze. In: Proceedings of 2023 CHI Conference on Human Factors in Computing Systems. 2023, 719

[90] Zhang R, Li K, Hao Y, Wang Y, Lai Z, Guimbretière F, Zhang C. EchoSpeech: continuous silent speech recognition on minimally-obtrusive eyewear powered by acoustic sensing. In: Proceedings of 2023 CHI Conference on Human Factors in Computing Systems. 2023, 852

[91] Shen J, Boldu R, Kalla A, Glueck M, Surale H B, Karlson A . RingGesture: a ring-based mid-air gesture typing system powered by a deep-learning word prediction framework. IEEE Transactions on Visualization and Computer Graphics, 2024, 30( 11): 7441–7451

[92] Hu J, Dudley J J, Kristensson P O. SkiMR: dwell-free eye typing in mixed reality. In: Proceedings of 2024 IEEE Conference Virtual Reality and 3D User Interfaces. 2024, 439−449

[93] Cai Z, Ma Y, Lu F . Robust dual-modal speech keyword spotting for XR headsets. IEEE Transactions on Visualization and Computer Graphics, 2024, 30( 5): 2507–2516

[94] Ren Y, Zhang Y, Liu Z, Xie N . Eye-hand typing: eye gaze assisted finger typing via Bayesian processes in AR. IEEE Transactions on Visualization and Computer Graphics, 2024, 30( 5): 2496–2506

[95] Jadon S S, Faridan M, Mah E, Vaish R, Willett W, Suzuki R. Augmented conversation with embedded speech-driven on-the-fly referencing in AR. 2024, arXiv preprint arXiv: 2405.18537

[96] Wang Z, Shi Y, Wang Y, Yao Y, Yan K, Wang Y, Ji L, Xu X, Yu C . G-VOILA: gaze-facilitated information querying in daily scenarios. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 2024, 8( 2): 78

[97] Lee J, Wang J, Brown E, Chu L, Rodriguez S S, Froehlich J E. GazePointAR: a context-aware multimodal voice assistant for pronoun disambiguation in wearable augmented reality. In: Proceedings of 2024 CHI Conference on Human Factors in Computing Systems. 2024, 408

[98] Wang X, Lafreniere B, Zhao J. Exploring visualizations for precisely guiding bare hand gestures in virtual reality. In: Proceedings of 2024 CHI Conference on Human Factors in Computing Systems. 2024, 636

[99] Shen J, Dudley J J, Mo G B, Kristensson P O . Gesture spotter: a rapid prototyping tool for key gesture spotting in virtual and augmented reality applications. IEEE Transactions on Visualization and Computer Graphics, 2022, 28( 11): 3618–3628

[100] Lee C J, Zhang R, Agarwal D, Yu T C, Gunda V, Lopez O, Kim J, Yin S, Dong B, Li K, Sakashita M, Guimbretiere F, Zhang C. EchoWrist: continuous hand pose tracking and hand-object interaction recognition using low-power active acoustic sensing on a wristband. In: Proceedings of 2014 CHI Conference on Human Factors in Computing Systems. 2024, 403

[101] Rupp D, GrieBer P, Bonsch A, Kuhlen T W. Authentication in immersive virtual environments through gesture-based interaction with a virtual agent. In: Proceedings of 2024 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops. 2024, 54−60

[102] Liu T, Xiao Y, Hu M, Sha H, Ma S, Gao B, Guo S, Liu Y, Song W . AudioGest: gesture-based interaction for virtual reality using audio devices. IEEE Transactions on Visualization and Computer Graphics, 2025, 31( 2): 1569–1581

[103] Xu C, Zhou B, Krishnan G, Nayar S K. AO-finger: hands-free fine-grained finger gesture recognition via acoustic-optic sensor fusing. In: Proceedings of 2023 CHI Conference on Human Factors in Computing Systems. 2023, 306

[104] Kitamura R, Yamamoto T, Sugiura Y. TouchLog: finger micro gesture recognition using photo-reflective sensors. In: Proceedings of the 2023 International Symposium on Wearable Computers. 2023, 92−97

[105] Li T, Liu Y, Ma S, Hu M, Liu T, Song W. NailRing: an intelligent ring for recognizing micro-gestures in mixed reality. In: Proceedings of 2022 IEEE International Symposium on Mixed and Augmented Reality. 2022, 178−186

[106] Pöhlmann K M T, Li G, McGill M, Markoff R, Brewster S A. You spin me right round, baby, right round: examining the impact of multi-sensory self-motion cues on motion sickness during a VR reading task. In: Proceedings of 2023 CHI Conference on Human Factors in Computing Systems. 2023, 712

[107] Medlar A, Lehtikari M T, Glowacka D. Behind the scenes: adapting cinematography and editing concepts to navigation in virtual reality. In: Proceedings of 2024 CHI Conference on Human Factors in Computing Systems. 2024, 545

[108] Feick M, Regitz K P, Tang A, Krüger A. Designing visuo-haptic illusions with proxies in virtual reality: exploration of grasp, movement trajectory and object mass. In: Proceedings of 2022 CHI Conference on Human Factors in Computing Systems. 2022, 635

[109] Wu G, Qian J, Quispe S C, Chen S, Rulff J, Silva C. ARTiST: automated text simplification for task guidance in augmented reality. In: Proceedings of 2024 CHI Conference on Human Factors in Computing Systems. 2024, 939

[110] Yang J J, Qiu L, Corona-Moreno E A, Shi L, Bui H, Lam M S, Landay J A. AMMA: adaptive multimodal assistants through automated state tracking and user model-directed guidance planning. In: Proceedings of 2024 IEEE Conference Virtual Reality and 3D User Interfaces. 2024, 892−902

[111] Elsharkawy A I A M, Ataya A A S, Yeo D, An E, Hwang S, Kim S. SYNC-VR: synchronizing your senses to conquer motion sickness for enriching in-vehicle virtual reality. In: Proceedings of 2024 CHI Conference on Human Factors in Computing Systems. 2024, 257

[112] Li Y, Liu Z, Yuan L, Tang H, Fan Y, Xie N. Dynamic scene adjustment mechanism for manipulating user experience in VR. In: Proceedings of 2024 IEEE Conference Virtual Reality and 3D User Interfaces. 2024, 179−188

[113] Rasch J, Rusakov V D, Schmitz M, Müller F. Going, going, gone: exploring intention communication for multi-user locomotion in virtual reality. In: Proceedings of 2023 CHI Conference on Human Factors in Computing Systems. 2023, 785

[114] Tan F F Y, Xu P, Ram A, Suen W Z, Zhao S, Huang Y, Hurter C. AudioXtend: assisted reality visual accompaniments for audiobook storytelling during everyday routine tasks. In: Proceedings of 2024 CHI Conference on Human Factors in Computing Systems. 2024, 83

[115] Wang X, Zhang W, Fu H. A3RT: attention-aware AR teleconferencing with life-size 2.5D video avatars. In: Proceedings of 2024 IEEE Conference Virtual Reality and 3D User Interfaces. 2024, 211−221

[116] Tao Y, Lopes P. Integrating real-world distractions into virtual reality. In: Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology. 2022, 5

[117] Lee G, Lee D Y, Su G M, Manocha D . “May I speak?”: multi-modal attention guidance in social VR group conversations. IEEE Transactions on Visualization and Computer Graphics, 2024, 30( 5): 2287–2297

[118] Wang Z, Lu F . Tasks reflected in the eyes: egocentric gaze-aware visual task type recognition in virtual reality. IEEE Transactions on Visualization and Computer Graphics, 2024, 30( 11): 7277–7287

[119] Shen V, Shultz C, Harrison C. Mouth haptics in VR using a headset ultrasound phased array. In: Proceedings of 2022 CHI Conference on Human Factors in Computing Systems. 2022, 275

[120] Tatzgern M, Domhardt M, Wolf M, Cenger M, Emsenhuber G, Dinic R, Gerner N, Hartl A. AirRes mask: a precise and robust virtual reality breathing interface utilizing breathing resistance as output modality. In: Proceedings of 2022 CHI Conference on Human Factors in Computing Systems. 2022, 274

[121] Kim M J, Ofek E, Pahud M, Sinclair M J, Bianchi A. Big or small, it’s all in your head: visuo-haptic illusion of size-change using finger-repositioning. In: Proceedings of 2024 CHI Conference on Human Factors in Computing Systems. 2024, 751

[122] Yamazaki Y, Hasegawa S. Providing 3D guidance and improving the music-listening experience in virtual reality shooting games using musical vibrotactile feedback. In: Proceedings of 2023 IEEE Conference Virtual Reality and 3D User Interfaces. 2023, 276−285

[123] Shen V, Rae-Grant T, Mullenbach J, Harrison C, Shultz C. Fluid reality: high-resolution, untethered haptic gloves using electroosmotic pump arrays. In: Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology. 2023, 8

[124] Jingu A, Withana A, Steimle J. Double-sided tactile interactions for grasping in virtual reality. In: Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology. 2023, 9

[125] Saint-Aubert J, Argelaguet F, Macé M, Pacchierotti C, Amedi A, Lécuyer A. Persuasive vibrations: effects of speech-based vibrations on persuasion, leadership, and co-presence during verbal communication in VR. In: Proceedings of 2023 IEEE Conference Virtual Reality and 3D User Interfaces. 2023, 552−560

[126] Shi C, Chen J, Liu J, Yang C . Graph foundation model. Frontiers of Computer Science, 2024, 18( 6): 186355

[127] Guo W, Zhuang F, Zhang X, Tong Y, Dong J . A comprehensive survey of federated transfer learning: challenges, methods and applications. Frontiers of Computer Science, 2024, 18( 6): 186356

[128] Hedeshy R, Kumar C, Menges R, Staab S. Hummer: text entry by gaze and hum. In: Proceedings of 2021 CHI Conference on Human Factors in Computing Systems. 2021, 741

[129] Schneider D, Biener V, Otte A, Gesslein T, Gagel P, Campos C, Pucihar K Č, Kljun M, Ofek E, Pahud M, Kristensson P O, Grubert J. Accuracy evaluation of touch tasks in commodity virtual and augmented reality head-mounted displays. In: Proceedings of 2021 ACM Symposium on Spatial User Interaction. 2021, 7

[130] Cai M, Lu F, Sato Y. Generalizing hand segmentation in egocentric videos with uncertainty-guided model adaptation. In: Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020, 14392−14401

[131] Cai M, Lu F, Gao Y . Desktop action recognition from first-person point-of-view. IEEE Transactions on Cybernetics, 2019, 49( 5): 1616–1628

[132] Cheng Y, Wang H, Bao Y, Lu F . Appearance-based gaze estimation with deep learning: a review and benchmark. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024, 46( 12): 7509–7528

[133] Wang Z, Zhao Y, Liu Y, Lu F. Edge-guided near-eye image analysis for head mounted displays. In: Proceedings of 2021 IEEE International Symposium on Mixed and Augmented Reality. 2021, 11−20

[134] Guestrin E D, Eizenman M . General theory of remote gaze estimation using the pupil center and corneal reflections. IEEE Transactions on Biomedical Engineering, 2006, 53( 6): 1124–1133

[135] Wu Z, Rajendran S, van As T, Zimmermann J, Badrinarayanan V, Rabinovich A. MagicEyes: a large scale eye gaze estimation dataset for mixed reality. 2020, arXiv preprint arXiv: 2003.08806

[136] Santini T, Niehorster D C, Kasneci E. Get a grip: slippage-robust and glint-free gaze estimation for real-time pervasive head-mounted eye tracking. In: Proceedings of the 11th ACM Symposium on Eye Tracking Research & Applications. 2019, 17

[137] Narcizo F B, de Queiroz J E R, Gomes H M. Remote eye tracking systems: technologies and applications. In: Proceedings of the 26th Conference on Graphics, Patterns and Images Tutorials. 2013, 15−22

[138] Dierkes K, Kassner M, Bulling A. A fast approach to refraction-aware eye-model fitting and gaze prediction. In: Proceedings of the 11th ACM Symposium on Eye Tracking Research & Applications. 2019, 23

[139] Kytö M, Ens B, Piumsomboon T, Lee G A, Billinghurst M. Pinpointing: precise head- and eye-based target selection for augmented reality. In: Proceedings of 2018 CHI Conference on Human Factors in Computing Systems. 2018, 81

[140] Wang X, Su Z, Rekimoto J, Zhang Y. Watch your mouth: silent speech recognition with depth sensing. In: Proceedings of 2024 CHI Conference on Human Factors in Computing Systems. 2024, 323

[141] Wang Z, Sun J, Hu M, Rao M, Song W, Lu F. GazeRing: enhancing hand-eye coordination with pressure ring in augmented reality. In: Proceedings of 2024 IEEE International Symposium on Mixed and Augmented Reality. 2024, 534−543

[142] Wang Z, Wang H, Yu H, Lu F . Interaction with gaze, gesture, and speech in a flexibly configurable augmented reality system. IEEE Transactions on Human-Machine Systems, 2021, 51( 5): 524–534

[143] Burdea G C, Coiffet P. Virtual Reality Technology. 2nd ed. Hoboken: Wiley, 2003

[144] Jang J, Frier W, Park J. Multimodal volume data exploration through mid-air haptics. In: Proceedings of 2022 IEEE International Symposium on Mixed and Augmented Reality. 2022, 243−251

[145] Shiota T, Takagi M, Kumagai K, Seshimo H, Aono Y. Egocentric action recognition by capturing hand-object contact and object state. In: Proceedings of 2024 IEEE/CVF Winter Conference on Applications of Computer Vision. 2024, 6527−6537

[146] Khan A A, Newn J, Bailey J, Velloso E. Integrating gaze and speech for enabling implicit interactions. In: Proceedings of 2022 CHI Conference on Human Factors in Computing Systems. 2022, 349

RIGHTS & PERMISSIONS

The Author(s) 2025. This article is published with open access at link.springer.com and journal.hep.com.cn