Interactive Movement-to-Audio with Pre-Trained Neural Networks

Systems to interactively generate audio from human movement are used by artists including dancers to support their performances and practice. However, current real-time movement-to-sound systems require specialized hardware or expertise, or map only very simple movement-to-audio relationships. We present a new technique and system implementation for interactive sonification of human movement through unsupervised machine learning. Our system maps between latent spaces, linking a pose estimator to a neural audio generator to enable sonification of human bodies. This may lower barriers to entry for artists to generate sound from their embodied movement through complex mappings. Our system requires no specialized hardware or niche AI expertise, minimal data to learn a user’s custom movements, and trains extremely fast. It represents a new method for mapping custom data to a latent space through unsupervised learning, and advances state-of-the-art interactive movement sonification through its increased accessibility and ease of use relative to its complexity.