Data Analysis in the Wild: Benchmarking Large Language Models Against Real-World Data Complexities

Current benchmarks for evaluating Large Language Models (LLMs) in data analysis often fail to reflect real-world settings.

They typically focus on fact retrieval from small tables and overlook the challenges of large multi-tabular datasets, external knowledge integration, and exploratory insight discovery.

We introduce DataGovBench, a benchmark derived from governmental open data designed to evaluate LLMs in practical scenarios.

The benchmark includes two tasks: Table QA that requires solving complex decomposable questions and producing textual answers or visualizations, and Table Insight that evaluates the ability of models to generate expert-level findings through exploratory data analysis.

Comprehensive experiments with state-of-the-art LLMs, both with and without agentic frameworks, reveal significant performance gaps across both tasks.

These results suggest that current LLM-based systems remain far from satisfying the demands of real-world data analytics.

DataGovBench provides a challenging benchmark for advancing research on LLMs capable of both answering analytical queries and discovering insights from data.

Code and sample data are available at this https URL.