Interactive audio toolkit: Creating sonic experiences and installations with low-cost, low-power microcontrollers

Abstract

Interactive sound installations and experiences are increasingly prevalent in museums, galleries, events, and various spaces worldwide. These installations are often powered by microcontroller board running programs that control interactivity and possibly even audio processing. Commonly used microcontrollers such as the ESP32 and Raspberry Pi Pico are generic, necessitating custom implementations for audio-specific functionalities. Existing audio libraries often have limited signal processing features and may not be designed with interactivity in mind. This paper presents the Interactive Audio Toolkit, which focuses on the seamless integration of sound generation with sensors to create interactive sound experiences with low-power, low-cost microcontrollers. This C++ toolkit is structured around sensor input and audio output classes, providing a simple interface to generate flexible audio responses from sensor interactions. This paper details the toolkit’s structure, components, and workflow, highlighting its ability to foster new forms of sonic and musical interactions for artists and audiences. A case study of an installation utilizing the toolkit is also presented.

Keywords

Sound installation / Interactive audio / Embedded audio / Interaction design / Ubimus toolkits

References

[1] Timoney J, Lazzarini V, Keller D. DIY electronics for ubiquitous music ecosystems. In: Lazzarini V, Keller D, Otero N, Turchet L,editors. Ubiquitous Music Ecologies. Milton Park: Routledge; 2020.

[2] Jagwani A. Creative possibilities and customizability of live performance systems with open source programming platforms. In: Yaseen A, Bridges B, Messina M, Keller D,eds. Proceedings of the 2nd International Symposium on Ubiquitous Music (UbiMus 2023). Northern Ireland:Ulster University; 2023:133-144. Available from: https://www.ulster.ac.uk/ conference/ubimus

[3] Espressif Systems. ESP32. Availablefrom: https://www.espressif.com/en/products/socs/esp32 [Last accessed on 2024 Apr 24].

[4] STM32. STM32 32-bit ARM Cortex MCUs. Availablefrom: https://www.st.com/en/microcontrollers-microprocessors/stm32-32-bit-arm-cortex-mcus.html [Last accessed on 2024 Apr 24].

[5] RaspberryPi. Availablefrom: https://www.raspberrypi.org [Last accessed on 2024 Apr 24].

[6] Arduino. Availablefrom: https://www.arduino.cc [Last accessed on 2024 Apr 28].

[7] PlatformIO. Availablefrom: https://platformio.org [Last accessed on 2024 Apr 28].

[8] Ribeira Netto A, Castheloge L, Oliosi A, Mateus A, Costalonga L, Coura D. Árvore das Memórias: Instalação Multimídia Interativa. In: Proceedings of the Brazilian Symposium of Computer Graphic and Image Processing; 2015.

[9] Keller D, Gomes C, Aliel L. The handy metaphor: Bimanual, touchless interaction for the internet of musical things. J New Music Res. 2019; 48:1-12. doi: 10.1080/09298215.2019.1654525

[10] Schatzmann P. Arduino-Audio-Tools. Availablefrom: https://github.com/pschatzmann/arduino-audio-tools [Last accessed on 2024 Apr 28].

[11] Puckette M. Pure Data: Another Integrated Computer Music Environment. In: Proceedings of the Second Intercollege Computer Music Concerts, Tachikawa, Japan; 1997.

[12] Lazzarini V, Yi S, Ffitch J, Heintz J, Brandtsegg Ø, McCurdy I. Csound: A Sound and Music Computing System. Cham, Switzerland: Springer International Publishing; 2016. doi: 10.1007/978-3-319-45370-5

[13] Lazzarini V, Jagwani A. CSOUND 7:Bare Metal and Co-Processing Hardwares. In:Proceedings of the 19th Linux Audio Conference. Lyon, France; 2025. Available from: https://hal.science/hal-05096036

[14] Electro-Smith. Daisy. Availablefrom: https://www.electro-smith.com/daisy [Last accessed on 2023 Jun 08].

[15] SparkFun. SparkFun VL53L1X Arduino Library. Availablefrom: https://github.com/sparkfun/sparkfun_vl53l1x_arduino_library [Last accessed on 2024 Apr 28].

[16] SparkFun. Load Cell Amplifier HX711 Breakout Hookup Guide. Availablefrom: https://learn.sparkfun.com/tutorials/load-cell-amplifier-hx711-breakout-hookup-guide/all [Last accessed on 2024 Apr 28].

[17] Arduino. Wire Library Documentation. Availablefrom: https://docs.arduino.cc/learn/communication/wire [Last accessed on 2024 Apr 28].

[18] Xilinx. PS UART; 2023. Availablefrom: https://xilinx-wiki.atlassian.net/wiki/spaces/a/pages/18842340/ps+uart [Last accessed on 2024 Feb 10].

[19] MaxBotix. Arduino Ultrasonic Sensors. Availablefrom: https://maxbotix.com/pages/arduino-ultrasonic-sensors [Last accessed on 2024 Apr 28].

[20] Lazzarini V. Spectral Music Design: A Computational Approach. Oxford: Oxford University Press; 2021.

[21] Adafruit. All About Stepper Motors: What is a Stepper Motor? Availablefrom: https://learn.adafruit.com/all-about-stepper-motors/what-is-a-stepper-motor [Last accessed on 2024 Apr 28].

[22] Arduino. AccelStepper Library. Availablefrom: https://www.arduino.cc/reference/en/libraries/accelstepper [Last accessed on 2024 Apr 28].

[23] Precision Microdrives. Vibration Motors. Availablefrom: https://www.precisionmicrodrives.com/motors/vibration-motors [Last accessed on 2024 Apr 28].

[24] ArduinoJson. ArduinoJson Library. Availablefrom: https://arduinojson.org [Last accessed on 2024 Apr 28].

[25] Waqif A. Chaal. Availablefrom: https://asimwaqif.com/chaal [Last accessed on 2025 Apr 20].

[26] DesignOwl. Chaal by Asim Waqif: Art Installation Using Bamboo. Availablefrom: https://www.youtube.com/watch?v=kmql_mw3m0i&ab_channel=designowl [Last accessed on 2024 Apr 28].

[27] MDN Web Docs. WebSockets API. Availablefrom: https://developer.mozilla.org/en-us/docs/web/api/websockets_api [Last accessed on 2024 Apr 28].

[28] React. React Documentation. Availablefrom: https://react.dev [Last accessed on 2024 Feb 16].

[29] Espressif. mDNS API Reference for ESP32. Availablefrom: https://docs.espressif.com/projects/esp-idf/en/stable/esp32/api-reference/protocols/mdns.html [Last accessed on 2024 Apr 28].

[30] Contributors F. Fastify: Fast and Low Overhead Web Framework for Node. Last accessed on 2024 Feb 16].

[31] Turchet L, Essl G, Fischione C. Ubiquitous music and the internet of musical things. In: Lazzarini V, Keller D, Otero N, Turchet L,editors. Ubiquitous Music Ecologies. London: Routledge; 2020.

[32] Wiener N. Cybernetics: Or Control and Communication in the Animal and the Machine. United States: MIT Press; 1948.

[33] Von Bertalanffy L. General System Theory: Foundations, Development, Applications. New York: George Braziller; 1968.

[34] Koszolko MK, Studley T. From site-specific sampling to gamification: An exploration of performative engagement with the environment. Organised Sound. 2023; 28(3):338-351. doi: 10.1017/S1355771823000547

[35] Bridges B, Lazzarini V, Keller D. Editorial: Ecologically grounded creative practices and ubiquitous music: Interaction and environment. Organised Sound. 2024; 28(3):321-327. doi: 10.1017/S1355771823000663

[36] TensorFlow. TensorFlow Lite for Microcontrollers. Availablefrom: https://www.tensorflow.org/lite/microcontrollers [Last accessed on 2024 Apr 28].