Scaffolding Off-the-Shelf Text-to-Audio Models for Real-Time Generation in Live Performance

Contemporary text-to-audio generation models face two independent technical challenges when deployed in live performance: buffer-quantised input control and subreal-time generation speed. Live musicians and technologists often work with off-the-shelf models and cannot address these issues at their source, requiring external scaffolding solutions. We present a taxonomy of architectural strategies to address these constraints. We analyse tradeoffs in latency, continuity, and computational cost, discussing practical implications for system design. A case study with Meta’s MusicGen (Copet et al., 2023) text-to-audio model demonstrates these strategies in practice. We characterise each strategy along two independent axes: the time the user spends waiting for requests to take effect (stale time), and the proportion of audio that is unique rather than recycled (novelty). Pre-computation through crossfading reduces median input-to-playback latency from 5.34 s (looping baseline) to 0.03 s and eliminates stale time entirely, at the cost of low novelty; parameter-space interpolation achieves both responsiveness and novelty but requires the largest setup investment.

• Organisation