Beyond the surface: a scoping review of vision-based underwater experience technologies and user studies

Abstract

Exploring the underwater world presents significant physical and financial barriers. Telepresence technologies offer a potential solution by a providing a more accessible version of an underwater experience. Our research involved a scoping review to consolidate previous findings on vision-based technologies that aim to recreate underwater experiences and their user evaluations. We searched 5 academic databases for papers describing or evaluating technologies providing visual underwater experiences without actual submersion. We systematically searched YouTube to include immersive experiences not documented in academic publications. Our review included 45 academic papers and 23 YouTube videos classified by their level of ‘reality’, ‘degrees of freedom’, and presence of interactive elements. The technologies reviewed included virtual reality, 360

{}^{\circ }

\documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$^{\circ }$$\end{document}

video and imaging, augmented and mixed reality, head-mounted displays, mobile devices, cameras, sensors, remotely operated vehicles (ROVs), and games. Half of the selected papers featured user evaluations (with sample sizes ranging from 5 to 1006 participant); these methods included interviews, performance tracking, and questionnaires. We identified six main application areas for these technologies: (i) general environmental awareness, (ii) formal education about marine life, (iii) therapeutic interventions, (iv) access to underwater heritage sites, (v) ROV teleoperation and simulations, and (vi) entertainment. Immersive technologies, such as head-mounted displays and augmented reality, were prevalent across all application categories, though their usability varied. Cost considerations were also diverse, with costs ranging from expensive ROVs and simulated environments to cheaper 360

{}^{\circ }

\documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$^{\circ }$$\end{document}

videos. Our findings indicate a need for more robust user studies, including long-term research and comparisons among real-time, pre-recorded, and simulated experiences. A better understanding of entertainment-driven applications could benefit education, environmental conservation, and healthcare. The findings of the scoping review are discussed with respect to the technologies identified and the corresponding user studies.

Keywords

Underwater telepresence / Immersive technology / Human-computer interaction / Underwater experience

References

[1] Agrawal AK, Robertson S, Litwin L, Tringale E, Treadwell M, Hoppe C et al (2019) Virtual reality as complementary pain therapy in hospitalized patients with sickle cell disease. Pediatr Blood Cancer 66(2):e27525. https://doi.org/10.1002/pbc.27525

[2] Agrawal S, Simon AMD, Bech S, Bærentsen KB, Forchhammer S. Defining immersion: literature review and implications for research on audiovisual experiences. J Audio Eng Soc, 2020, 68(6): 404-417,

[3] Anderson KO, Dowds BN, Pelletz RE, Edwards WT, Peeters-Asdourian C. Development and initial validation of a scale to measure self-efficacy beliefs in patients with chronic pain. Pain, 1995, 63(1): 77-83,

[4] Attkisson CC, Zwick R. The client satisfaction questionnaire: psychometric properties and correlations with service utilization and psychotherapy outcome. Eval Program Plan, 1982, 5(3): 233-237,

[5] Baños RM, Botella C, Garcia-Palacios A, Villa H, Perpiñá C, Alcaniz M. Presence and reality judgment in virtual environments: a unitary construct?. CyberPsychology Behav, 2000, 3(3): 327-335,

[6] Barrile V, Fotia A, Bernardo E, Candela G (2020) Geomatics techniques for submerged heritage: a mobile app for tourism. WSEAS Trans Environ Dev 16:586–597. https://doi.org/10.37394/232015.2020.16.60

[7] Barrile V, Pucinotti R, Bilotta G. . Geomatics and virtual reality techniques for underwater heritage., 2021,

[8] Berlyne DE. Novelty, complexity, and hedonic value. Percept Psychophys, 1970, 8(5): 279-286,

[9] Brennan ML, Cantelas F, Elliott K, Delgado JP, Bell KLC, Coleman D et al (2018) Telepresence-enabled maritime archaeological exploration in the deep. J Marit Archaeol 13(2):97–121

[10] Brooke J. Sus: a “quick and dirty" usability. Usability Eval Ind, 1996, 189(3): 189-194

[11] Bruce J, Dennis M, Edwards A, Parker-Holder J, Shi Y, Hughes E et al (2024) Genie: generative interactive environments. Preprint at arXiv:2402.15391

[12] Bruno F, Barbieri L, Lagudi A, Cozza M, Cozza A, Peluso R, Muzzupappa M. Virtual dives into the underwater archaeological treasures of south Italy. Virtual Reality, 2018, 22(1): 47-60,

[13] Bruno F, Barbieri L, Muzzupappa M, Tusa S, Fresina A, Oliveri F et al (2019) Enhancing learning and access to underwater cultural heritage through digital technologies: the case study of the Cala Minnola shipwreck site. Digit Appl Archaeol Cult Herit 13:e00103. https://doi.org/10.1016/j.daach.2019.e00103

[14] Bruno F, Lagudi A, Barbieri L, Muzzupappa M, Mangeruga M, Pupo F et al (2017) Virtual diving in the underwater archaeological site of Cala Minnola. Int Arch Photogramm Remote Sens Spat Inf Sci XLII-2/W3:121–126. https://doi.org/10.5194/isprs-archives-XLII-2-W3-121-2017

[15] Bruzzone G, Bono R, Caccia M, Coletta P, Veruggio G (2003) Internet-based teleoperation of the Romeo ROV in the Arctic region. IFAC Proc Volumes 36(21):265–269. https://doi.org/10.1016/S1474-6670(17)37818-7

[16] Burdea CG, Coiffet P (2003) Virtual reality technology, 2nd edition. Wiley-IEEE Press

[17] Calvi L, Santos CP, Relouw J, Endrovski B, Rothwell C, Sara A et al (2017) A VR game to teach underwater sustainability while diving. In: 2017 Fifth IFIP Conference on Sustainable Internet and ICT for Sustainability (SustainIT 2017), Funchal, pp 114–117

[18] Capocci R, Dooly G, Omerdić E, Coleman J, Newe T, Toal D. Inspection-class remotely operated vehicles-a review. J Mar Sci Eng, 2017, 5(1): 13,

[19] Chen G, Li B, Tian F, Ji P, Li W (2012) Design and implementation of a 3D ocean virtual reality and visualization engine. J Ocean Univ China 11(4):481–487

[20] Chouiten M, Domingues C, Didier JY, Otmane S, Mallem M (2012) Distributed mixed reality for remote underwater telerobotics exploration. In: Proceedings of the 2012 Virtual Reality International Conference, New York, pp 1–6. https://doi.org/10.1145/2331714.2331716

[21] Christensen L, de Gea Fernández J, Hildebrandt M, Koch CES, Wehbe B. Recent advances in AI for navigation and control of underwater robots. Curr Robot Rep, 2022, 3(4): 165-175,

[22] Cleeland C, Ryan K. Pain assessment: global use of the Brief Pain Inventory. Ann Acad Med Singap, 1994, 23(2): 129-138

[23] Corker K, Mishkin A, Lyman J (1980) Achievement of a sense of operator presence in remote manipulation. In: Technical report, California University Los Angeles Biotechnology Laboratory, Defense Technical Information Center, report No. ADA109849. https://apps.dtic.mil/sti/citations/ADA109849. Accessed 19 Dec 2023

[24] de la Cruz M, Casan GA, Sanz PJ, Marin R (2020a) A new virtual reality interface for underwater intervention missions. In: 21st IFAC World Congress on Automatic Control-Meeting Societal Challenges, pp 14600–14607. https://doi.org/10.1016/j.ifacol.2020.12.1468

[25] de la Cruz M, Casañ GA, Sanz PJ, Marín R (2020b) Preliminary work on a virtual reality interface for the guidance of underwater robots. Robotics 9(4):81. https://doi.org/10.3390/robotics9040081

[26] Delgado DA, Lambert BS, Boutris N, McCulloch PC, Robbins AB, Moreno MR et a (2018) Validation of digital visual analog scale pain scoring with a traditional paper-based visual analog scale in adults. J Am Acad Orthop Surg Glob Res Rev 2(3):e088. https://doi.org/10.5435/JAAOSGlobal-D-17-00088

[27] Diener E, Wirtz D, Tov W, Kim-Prieto C, Dw Choi, Oishi S, Biswas-Diener R. New well-being measures: Short scales to assess flourishing and positive and negative feelings. Soc Indic Res, 2010, 97: 143-156,

[28] Dunlap RE, Van Liere KD, Mertig AG, Jones RE. New trends in measuring environmental attitudes: measuring endorsement of the new ecological paradigm: a revised NEP scale. J Soc Issues, 2000, 56(3): 425-442,

[29] Elor A, Thang T, Hughes BP, Crosby A, Phung A, Gonzalez E et al (2021) Catching jellies in immersive virtual reality: a comparative teleoperation study of ROVs in underwater capture tasks. In: Proceedings of the 27th ACM Symposium on Virtual Reality Software and Technology, Osaka, pp 1–10. https://doi.org/10.1145/3489849.3489861

[30] Fahlman SA, Mercer-Lynn KB, Flora DB, Eastwood JD. Development and validation of the multidimensional state boredom scale. Assessment, 2013, 20(1): 68-85,

[31] Gerringer ME, Ismail Y, Cannon KA, Hernández AC, Peralta FG, Bohen R et al (2023) Deep-sea biology in undergraduate classrooms: open access data from remotely operated vehicles provide impactful research experiences. Front Mar Sci 9:1033274. https://doi.org/10.3389/fmars.2022.1033274

[32] Hagen O, Varga M, Baxter R, Jones R, Aly A, Bazazian D et al (2024) Insights from co-design of underwater telepresence and extended reality technologies with digitally excluded older adults. In: Proceedings of the 10th International Conference on Information and Communication Technologies for Ageing Well and e-Health ICT4AWE, Angers, pp 298–303. https://doi.org/10.5220/0012741900003699

[33] Hart SG, Staveland LE (1988) Development of NASA-TLX (task load index): results of empirical and theoretical research. In: Hancock PA, Meshkati N (eds) Human mental workload, advances in psychology, vol 52. Elsevier, North-Holland, pp 139–183

[34] Haydar M, Roussel D, Maïdi M, Otmane S, Mallem M. Virtual and augmented reality for cultural computing and heritage: A case study of virtual exploration of underwater archaeological sites. Virtual Reality, 2011,

[35] Hine B, Stoker C, Sims M, Rasmussen D, Hontalas P, Fong T et al (1994) The application of telepresence and virtual reality to subsea exploration. In: Proceedngs of ROV ’94 2nd Workshop on Mobile Robots for Subsea Environments, Monterey, pp 117–126. https://www.ri.cmu.edu/publications/the-application-of-telepresence-and-virtual-reality-to-subsea-exploration/

[36] Humphris GM, Morrison T, Lindsay S. The Modified Dental Anxiety Scale: validation and united kingdom norms. Community Dent Health, 1995, 12(3): 143-150

[37] Jain D, Sra M, Guo J, Marques R, Wu R, Chiu J et al (2016) Immersive scuba diving simulator using virtual reality. In: Proceedings of the 29th Annual Symposium on User Interface Software and Technology, Tokyo, pp 729–739. https://doi.org/10.1145/2984511.2984519

[38] Jalal A, Salman A, Mian A, Shortis M, Shafait F (2020) Fish detection and species classification in underwater environments using deep learning with temporal information. Ecol Inf 57:101088. https://doi.org/10.1016/j.ecoinf.2020.101088

[39] Jenkyns R, Cheng J, Kershtien S, Lavallee T, Qi D, Schouten R et al (2015) Ship to shore: ROV dive logging and data acquisition. In: OCEANS 2015–MTS/IEEE Washington, Washington, DC, pp 1–4. https://doi.org/10.23919/OCEANS.2015.7404608

[40] Jones RB, Baxter R, Varga M, Hagen O, Aly A, Bazazian D et a (2024) Intergenerational technology codesign in deprived coastal regions. In: ICDS 2024: The Eighteenth International Conference on Digital Society, Barcelona, pp 7

[41] Kalawsky RS. VRUSE-a computerised diagnostic tool: for usability evaluation of virtual/synthetic environment systems. Appl Ergon, 1999, 30(1): 11-25,

[42] Katsouri I, Tzanavari A, Herakleous K, Poullis C. Visualizing and assessing hypotheses for marine archaeology in a VR CAVE environment. J Comput Cult Herit, 2015, 8(2): 1-18,

[43] Kelleher SA, Fisher HM, Winger JG, Miller SN, Amaden GH, Somers TJ et al (2022) Virtual reality for improving pain and pain-related symptoms in patients with advanced stage colorectal cancer: a pilot trial to test feasibility and acceptability. Palliat Support Care 20(4):471–481. https://doi.org/10.1017/S1478951521002017

[44] Kim HK, Park J, Choi Y, Choe M. Virtual reality sickness questionnaire (VRSQ): Motion sickness measurement index in a virtual reality environment. Appl Ergon, 2018, 69: 66-73,

[45] Kouřil P, Liarokapis F (2018) Simulation of underwater excavation using dredging procedures. IEEE Comput Graph Appl 38(2):103–111. https://doi.org/10.1109/MCG.2018.021951637

[46] Liarokapis F, Kouřil P, Agrafiotis P, Demesticha S, Chmelík J, Skarlatos D. 3d modelling and mapping for virtual exploration of underwater archaeology assets. Int Arch Photogramm Remote Sens Spat Inf Sci, 2017,

[47] Liarokapis F, Vidová I, Rizvić S, Demesticha S, Skarlatos D (2020) Underwater search and discovery: from serious games to virtual reality. In: Lecture notes in Computer Science. Springer, Cham, pp 178–197. https://doi.org/10.1007/978-3-030-59990-4_15

[48] Liestøl G, Bendon M, Hadjidaki-Marder E (2021) Augmented reality storytelling submerged. Dry diving to a World War II wreck at ancient Phalasarna, Crete. Heritage 4(4):4647–4664. https://doi.org/10.3390/heritage4040256

[49] Magrini M, Moroni D, Pascali MA, Reggiannini M, Salvetti O, Tampucci M (2015a) Virtual environment as a tool to access the marine abysses. In: MTS/IEEE OCEANS 2015–Genova, Genova, pp 1–5. https://doi.org/10.1109/OCEANS-Genova.2015.7271472

[50] Magrini M, Moroni D, Pascali MA, Reggiannini M, Salvetti O, Tampucci M (2016) Marine virtual environment to access underwater cultural heritage. In: 2016 International Workshop on Computational Intelligence for Multimedia Understanding (IWCIM), Reggio Calabria, pp 1–5. https://doi.org/10.1109/IWCIM.2016.7801188

[51] Magrini M, Pascali MA, Reggiannini M, Salvetti O, Tampucci M (2015b) Virtual immersive environments for underwater archaeological exploration. In: Proceedings of the 5th International Workshop on Image Mining, Berlin, pp 53–57. https://doi.org/10.5220/0005461700530057

[52] Markland D, Hardy L. On the factorial and construct validity of the intrinsic motivation inventory: Conceptual and operational concerns. Res Q Exerc Sport, 1997, 68(1): 20-32,

[53] Markowitz DM, Laha R, Perone BP, Pea RD, Bailenson JN (2018) Immersive virtual reality field trips facilitate learning about climate change. Front Psychol 9:2364. https://doi.org/10.3389/fpsyg.2018.02364

[54] Mayer FS, Frantz CM. The connectedness to nature scale: A measure of individuals’ feeling in community with nature. J Environ Psychol, 2004, 24(4): 503-515,

[55] McCarthy J, Martin K (2019) Virtual reality for maritime archaeology in 2.5D: a virtual dive on a flute wreck of 1659 in Iceland. In: Proceedings–2019 23rd International Conference in Information Visualization, Adelaide, pp 104–109. https://doi.org/10.1109/IV-2.2019.00030

[56] McCauley ME, Sharkey TJ. Cybersickness: Perception of self-motion in virtual environments. Presence Teleoperators Virtual Environ, 1992, 1(3): 311-318,

[57] Milgram P, Kishino F (1994) A taxonomy of mixed reality visual displays. IEICE Trans Inf Syst E77-D(12):1321–1329

[58] Minsky M (1980) Telepresence: a manifesto. In: Omni Magazine, June, pp 44–52

[59] Mouy X, et al.. FishCam: A low-cost open source autonomous camera for aquatic research. HardwareX, 2020, 8: e00110,

[60] Munn Z, Peters MD, Stern C, Tufanaru C, McArthur A, Aromataris E. Systematic review or scoping review? guidance for authors when choosing between a systematic or scoping review approach. BMC Med Res Methodol, 2018, 18: 1-7,

[61] Nawaf M, Drap P, Ben-Ellefi M, Nocerino E, Chemisky B, Chassaing T et al (2021) Using virtual or augmented reality for the time-based study of complex underwater archaeological excavations. ISPRS Ann Photogramm Remote Sens Spatial Inf Sci VIII-M-1-2021:117–124. https://doi.org/10.5194/isprs-annals-VIII-M-1-2021-117-2021

[62] Nelson KM, Anggraini E, Schlüter A (2020) Virtual reality as a tool for environmental conservation and fundraising. PLoS One 15(4):e0223631

[63] Omerdic E, Toal D, Dooly G, Kaknjo A (2014) Remote presence: long endurance robotic systems for routine inspection of offshore subsea oil & gas installations and marine renewable energy devices. In: 2014 Oceans–St. John's, St. John's, pp 1–9. https://doi.org/10.1109/OCEANS.2014.7003054.

[64] Ougradar A, Ahmed B. Patients’ perceptions of the benefits of virtual reality during dental extractions. Br Dent J, 2019, 227(9): 813-816,

[65] Pallant A, McIntyre C, Stephens AL (2016) Transforming undergraduate research opportunities using telepresence. J Geosci Educ 64(2):138–146

[66] Pehlivanides G, Monastiridis K, Tourtas A, Karyati E, Ioannidis G, Bejelou K et a (2020) The VIRTUALDiver project. Making Greece’s underwater cultural heritage accessible to the public. Appl Sci 10(22):8172. https://doi.org/10.3390/app10228172

[67] Perkins WW, Brady LK. . CURV III (cable-controlled underwater recovery vehicle) characteristics and mission applications, 1984 Naval Ocean Systems Center, San Diego Tech. rep

[68] Plecher DA, Keil L, Kost G, Fiederling M, Eichhorn C, Klinker G (2022) Exploring underwater archaeology findings with a diving simulator in virtual reality. Front Virtual Real 3:901335. https://doi.org/10.3389/frvir.2022.901335

[69] Raza K, Song H (2020) Fast and accurate fish detection design with improved YOLO-v3 model and transfer learning. Int J Adv Comput Sci Appl 11(2):7–16. https://doi.org/10.14569/IJACSA.2020.0110202

[70] Rosenstiel AK, Keefe FJ. The use of coping strategies in chronic low back pain patients: relationship to patient characteristics and current adjustment. Pain, 1983, 17(1): 33-44,

[71] Ryan JJ, et al.. Virtual reality cues for binge drinking in college students. Cyberpsychol Behav Soc Netw, 2010, 13(2): 159-162,

[72] Savedra MC, Holzemer WL, Tesler MD, Wilkie DJ. Assessment of postoperation pain in children and adolescents using the adolescent pediatric pain tool. Nurs Res, 1993, 42(1): 5-9,

[73] Schebor FS (1995) Virtual environment for undersea telepresence. In: Salganicoff M (ed) Telemanipulator and telepresence technologies II, Philadelphia. https://doi.org/10.1117/12.227944

[74] Schubert T, Friedmann F, Regenbrecht H. The experience of presence: Factor analytic insights. Presence Teleoperators Virtual Environ, 2001, 10(3): 266-281,

[75] Schultz PW (2002) Inclusion with nature: the psychology of human-nature relations. In: Psychology of sustainable development. Springer, Boston, pp 61–78

[76] Secci M, Beltrame C, Manfio S, Guerra F (2019) Virtual reality in maritime archaeology legacy data for a virtual diving on the shipwreck of the Mercurio (1812). J Cult Herit 40:169–176. https://doi.org/10.1016/j.culher.2019.05.002

[77] Sensorama (1956) The first VR device, Sensorama, created by Morton Heilig. IEEE Communications Society. (Accessed 12 Jan 2024) https://www.comsoc.org/node/19151

[78] Sheridan T. Musings on telepresence and virtual presence. Presence, 1992, 1: 120-125,

[79] Silvia P. Psychology of Human Motivation. Exploring Psychol Interest, 2006,

[80] Singh N, Bhat A (2023) A systematic review of the methodologies for the processing and enhancement of the underwater images. Multimed Tools Appl 82(25):38371–38396

[81] Sivčev S, Coleman J, Omerdić E, Dooly G, Toal D. Underwater manipulators: A review. Ocean Engineering, 2018, 163: 431-450,

[82] Skamantzari M, Georgopoulos A, Palyvou C (2020) Sunken roman villa of ancient Epidaurus: documentation and enhancement using virtual reality. ISPRS Ann Photogramm, Remote Sens Spatial Inf Sci V-2-2020:981–988. https://doi.org/10.5194/isprs-annals-V-2-2020-981-2020

[83] Slater M. Place illusion and plausibility can lead to realistic behaviour in immersive virtual environments. Phil Trans R Soc B Biol Sci, 2009, 364(1535): 3549-3557,

[84] Slater M. Immersion and the illusion of presence in virtual reality. Br J Psychol, 2018, 109(3): 431-433,

[85] Slater M, Usoh M, Steed A. Depth of presence in virtual environments. Presence Teleoperators Virtual Environ, 1994, 3(2): 130-144,

[86] Steuer J. Defining Virtual Reality: Dimensions determining telepresence. J Commun, 1992, 42(4): 73-93,

[87] Theodoropoulos A, Antoniou A. VR games in cultural heritage: A systematic review of the emerging fields of virtual reality and culture games. Appl Sci, 2022, 12(17): 8476,

[88] Thompson S, Chalmers A, Rhee T (2019) Real-time mixed reality rendering for underwater 360 ∘ \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$^{\circ }$$\end{document} videos. In: 2019 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), Beijing, pp 74–82. https://doi.org/10.1109/ISMAR.2019.00-24

[89] Vercelloni J, et al.. Using virtual reality to estimate aesthetic values of coral reefs. R Soc Open Sci, 2018, 5(4): 172226,

[90] Verdes A, Navarro C, Álvarez Campos P (2021) Mobile learning applications to improve invertebrate zoology online teaching. Invertebr Biol 140(1):e12321. https://doi.org/10.1111/ivb.12321

[91] Witmer BG, Singer MJ. Measuring presence in virtual environments: A presence questionnaire. Presence, 1998, 7(3): 225-240,

[92] Xia P, McSweeney K, Wen F, Song Z, Krieg M, Li S et a (2022) Virtual telepresence for the future of ROV teleoperations: opportunities and challenges. In: SNAME 27th Offshore Symposium, Houston. https://doi.org/10.5957/TOS-2022-015

[93] Yeo NL, White MP, Alcock I, Garside R, Dean SG, Smalley AJ et al (2020) What is the best way of delivering virtual nature for improving mood? An experimental comparison of high definition TV, 360 ∘ \documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$^{\circ }$$\end{document} video, and computer generated virtual reality. J Environ Psychol 72:101500

[94] Zhang X, Yin Y, Wan F (2019) Underwater manipulation training simulation system for manned DeepSubmarine vehicle. In: 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR), Osaka, pp 1271–1272

[95] Zhu A, Martin LJ, Head A, Callison-Burch C (2023) CALYPSO: LLMs as dungeon masters’ assistants. Preprint at arXiv:2308.07540

Funding

Engineering and Physical Sciences Research Council(P/W024357/1)