OpenGround: Planning-based Online Perception for Open-World 3D Visual Grounding

3D visual grounding aims to locate objects based on natural language descriptions in 3D scenes.

Existing supervised methods are limited by generalization and recent zero-shot methods typically rely on a predefined Object Lookup Table (OLT) to query Visual Language Models (VLMs) for reasoning about object locations via a single step grounding, which limits the applications in scenarios with undefined targets and complex queries.

To address these problems, we present OpenGround, a novel zero-shot framework for open-world 3D visual grounding that remains compatible with recent zero-shot methods.

OpenGround integrates Task-Chain Planning to decompose a query into a plan of context-to-target sub-goals for progressive grounding, and Context-Guided Perception to perceive novel objects online under context guidance from the task chain.

We also propose a new dataset named OpenTarget, which contains over 7000 object-description pairs to mimic open-world evaluation.

Extensive experiments demonstrate that OpenGround achieves competitive performance on Nr3D, state-of-the-art on ScanRefer, and delivers a substantial 17.6\% improvement on OpenTarget.

Project Page at this https URL.