LLM-Powered Interactive Robotic Action Synthesis from Multimodal Speech, Gestures, and Music

The quest for intuitive and natural human-robot interaction (HRI) remains a significant challenge in robotics.

Traditional methods often rely on rigid, pre-programmed commands that limit the robot's expressiveness and adaptability.

This paper introduces a novel framework that leverages the reasoning capabilities of Large Language Models (LLMs) to synthesize complex robotic actions from a rich tapestry of multimodal human inputs: natural speech, hand gestures, and music/sound beats.

Our system architecture integrates a speech transcription model, a gesture recognition module, and a signal processing pipeline for beat detection.

These processed inputs are contextualized using prompt templates and fed into a LLM.

The LLM, informed by a predefined robot action space, reasons over the combined inputs to generate a coherent sequence of actions.

This sequence is dispatched to an action queue for execution on a quadruped robot over ROS.

The framework has ability to interpret and fuse semantic commands from speech, deictic information from gestures, and rhythmic cues from music.

This work represents a step towards creating robots that can interact with humans in a more fluid, creative, and context-aware manner.