AR Matchmaking: The Compatibility of Musical Instruments with an AR Interface

Hyunkyung Shin; Henrik von Coler

Image credit: Hyunkyung Shin; Henrik von Coler

• Format: oral
• Session: papers-9
• Paper PDF link
• Presence: remote
• Duration: 15
• Type: long

Abstract:

Augmented Reality (AR) interfaces offer new possibilities for musical expression by extending the capabilities of acoustic, electronic, and electroacoustic instruments. This study investigates the usability of the ARCube, an AR-based spatial audio controller, with twelve distinct musical instruments played by experienced musicians. We identify usability challenges specific to certain instruments, particularly for two-handed playing, as well as issues related to gesture recognition and cube stability. Our analysis shows that interaction patterns, such as cube placement, sound effect usage, and gesture strategies, vary significantly between instruments. These differences are driven by the physical form of the instruments, the required playing techniques, and user expectations for control and responsiveness. Based on these insights, we suggest directions for developing adaptable AR interfaces that better accommodate diverse instruments and support broader integration of AR technologies into musical practice.