Gesture and Generative AI in NIME Design

Jason Smith; Bryan Pardo

Image credit: Jason Smith; Bryan Pardo

• Session: workshops-2
• Location: Hanna Neumann 3.41

Abstract:

Generative AI models have been used to great effect in creating large amounts of music and audio for creative applications. There are challenges in using generative AI in live performances, such as high latency and lack of agency in terms of expressive input for a performer to provide a generative model. We aim to build AI-based NIMEs that support musicians creatively, rather than replacing their efforts during a performance.

We propose a workshop on generative AI systems for live music, focusing on enabling a user’s musical expression. This workshop will discuss recognizing human motions and gestures through a camera and real-time audio generation. We will also discuss the use of adaptive machine learning to enable a system to evolve alongside its user. This demonstration will include a live tutorial on the development of a camera-based gesture recognition application using popular frameworks and libraries.