Sonic immersion with interactive mobile web-application

Abstract

Over the past two decades, interactive technologies have gradually transformed audience engagement in live performance contexts. This paper explores audience interaction through a mobile web-app designed for immersive sonic experiences. The app transforms personal smartphones into a distributed speaker array and interactive controller. We examine two case studies: Freedom Collective, a contemporary opera where smartphones diffused pre-recorded sounds, creating a spatialized auditory field; and Exo Signals, an interactive sound experience at the Ubimus conference in Macau, where the audience actively shaped the soundscape by triggering audio samples and applying real-time DSP (band-pass filter) through touch interactions. We discuss the artistic and technological challenges in both cases, comparing the levels of immersion and audience engagement in passive versus active participation models.

Keywords

Mobile web-app / Sonic immersion / Audience participation / Interactive performance / Passive versus active interaction

References

[1] Liew K, Lindborg P, Rodrigues R, Styles SJ. Cross-modal perception of noise-in-music: Audiences generate spiky shapes in response to auditory roughness in a novel electroacoustic concert setting. In: Proceedings of the International Conference on Auditory Display; 2018.

[2] Nyckees J. Distributed Multi-Touch Interfaces for Musical Expression. Master’s Thesis. KTH Royal Institute of Technology; 2017.

[3] Wang G, Essl G, Penttinen H. Do Mobile Phones Dream of Electric Orchestras? In:Proceedings of the International Computer Music Conference (ICMC); 2008.

[4] Riionheimo J, Lokki T. Spatial sound, part 1: Exploring the impact of listening to room on perceptual immersion. J Audio Eng Soc. 2024; 72(12):913-929. doi: 10.17743/jaes.2022.0179

[5] Nosenko O. Immersive technologies in the formation of musical audio space. Notes Art Crit. 2024; 1(24):23-32.

[6] Levin G. Dialtones (A Telesymphony); 2001. Availablefrom: https://youtu.be/039_0Olb1WQ?si=oIEwrJp-GGOd60QV [Last accessed on 2025 Mar 01].

[7] Ciciliani M. SkylAR. Availablefrom: https://www.ciciliani.com/skylar.html [Last accessed on 2025 Mar 01].

[8] Taylor B. A history of the audience as a speaker array. In: Proceedings of the International Conference on New Interfaces for Musical Expression. Aalborg University Copenhagen; 2017. p. 481-486. doi: 10.5281/zenodo.1176324

[9] Forgette A, Manaris B, Gillikin M, Ramsden S. Speakers, More Speakers!! Developing Interactive, Distributed, Smartphone-Based, Immersive Experiences for Music and Art. In: Proceedings of the International Symposium on Electronic Art. Availablefrom: https://par.nsf.gov/biblio/10374099 [Last accessed on 2025 Mar 01].

[10] Marshall K, Loydell R. Thinking inside the box: Brian Eno, music, movement and light. J Vis Art Pract. 2017; 16(2):104-118. doi: 10.1080/14702029.2016.1195073

[11] Robaszkiewicz S, Schnell N. Soundworks: A Playground for Artists and Developers to Create Collaborative Mobile Web Performances. In: Proceedings of the Web Audio Conference (WAC); 2015.

[12] Essl G. The mobile phone ensemble as classroom. In: Proceedings of the International Computer Music Conference (ICMC); 2010.

[13] Hödl O, Kayali F, Fitzpatrick G. Poème Numérique: Large-Scale Audience Participation in Live Music Using Smartphones. In: Proceedings of the International Conference on New Interfaces for Musical Expression; 2020.

[14] Xambó A, Goudarzi V. The mobile audience as a digital musical persona in telematic performance. In: Proceedings of the International Conference on New Interfaces for Musical Expression; 2022.

[15] Koszolko MK, Studley T. From site-specific sampling to gamification: An exploration of performative engagement with the environment. Organ Sound. 2023; 28:338-351.

[16] Apple Inc. Human Interface Guidelines: Playing Audio. Availablefrom: https://developer.apple.com/design/human-interface-guidelines/playing-audio [Last accessed on 2025 Mar 01].

[17] Matuszewski B. A web-based framework for distributed music system research and creation. J Audio Eng Soc. 2020; 68:717-726.

[18] Shaw T, Piquemal S, Bowers J. Fields: An Exploration into the Use of Mobile Devices as a Medium for Sound Diffusion. In: Proceedings of the International Conference on New Interfaces for Musical Expression; 2015. p. 281-284.

[19] Chang T. An investigation of usability in immersive environments. In: Proceedings of the International Conference on Human-Computer Interaction; 2024.

[20] Vincze DB, Oeser M. Freedom Collective: Interactive Mobile Web-App for Immersive Audience Participation. In: Proceedings of the Ubiquitous Music Symposium (UBIMUS); 2024.

[21] Greenhalgh C, Hazzard A, Benford S, Cliffe L, Kelly E. Crafting trajectories of smartphone use at the opera. ACM Trans Comput Hum Interact. 2022; 29(6):59. doi: 10.1145/3531007

[22] Heim C. Audience as Performer: The Changing Role of Theatre Audiences in the Twenty-First Century. London: Routledge; 2015.

[23] Xambó A, Roma G. Performing Audiences: Composition Strategies for Network Music Using Mobile Phones. In: Proceedings of the International Conference on New Interfaces for Musical Expression; 2020.