A Modular Vision-Language-Action Robotics Framework for Indoor Environments

This paper presents an integrated system for the CMU Vision-Language-Action (VLA) Challenge, designed to enable an autonomous agent to perform complex tasks based on natural language instructions.

Our framework employs a modular architecture that orchestrates environment mapping, question processing, and navigation.

The system operates in two parallel streams: a perception pipeline that constructs a semantic voxel map from real-time camera feeds using OwlViT embeddings, and a language pipeline that classifies user commands with a Vision-Language Model.

The mapping is time-constrained; the system proceeds with a partial map if a 500-second exploration limit is reached.

The classified query is then grounded in the geometric and semantic context of the map to generate a detailed prompt for the VLM.

This yields an actionable output, demonstrating a capable solution for bridging the gap between human language and robotic action.