An End-to-End Musical Instrument System That Translates Electromyogram Biosignals to Synthesized Sound

Tanaka, Atau; Visi, Federico; Di Donato, Balandino; Klang, Martin; Zbyszyński, Michael

Abstract

This article presents a custom system combining hardware and sortware that sense physiological signals of the performer's body resulting from muscle contraction and translates them to computer-synthesized sound. Our goal was to build upon the history of research in the field to develop a complete, integrated system that could be used by nonspecialist musicians. We describe the Embodied AudioVisual Interaction Electromyogram, an end-to-end system, spanning wearable sensing on the musician's body, custom microcontroller-based biosignal acquisition hardware, machine learning– based gesture-to-sound mapping middleware, and software-based granular synthesis sound output. A novel hardware design digitizes the electromyogram signals from the muscle with minimal analog preprocessing and treats it in an audio signal-processing chain as a class-compliant audio and wireless MIDI interface. The mapping layer implements an interactive machine learning workflow in a reinforcement learning configuration and can map gesture features to auditory metadata in a multidimensional information space. The system adapts existing machine learning and synthesis modules adapted to work with the hardware, resulting in an integrated, end-to-end system. We explore its potential as a digital musical instrument through a series of public presentations and concert performance by a range of musical practitioners.

Citation

Tanaka, A., Visi, F., Di Donato, B., Klang, M., & Zbyszyński, M. (2024). An End-to-End Musical Instrument System That Translates Electromyogram Biosignals to Synthesized Sound. Computer Music Journal, 47(1), 64-84. https://doi.org/10.1162/comj_a_00672

• Journal Article Type: Article
• Acceptance Date: Sep 10, 2023
• Online Publication Date: Apr 10, 2024
• Publication Date: 2024-06
• Deposit Date: Apr 22, 2024
• Print ISSN: 0148-9267
• Electronic ISSN: 1531-5169
• Publisher: MIT Press
• Peer Reviewed: Peer Reviewed
• Volume: 47
• Issue: 1
• Pages: 64-84
• DOI: https://doi.org/10.1162/comj_a_00672
• Public URL: http://researchrepository.napier.ac.uk/Output/3597697