Benchmarking LLM Agents: Exposing Statistical Blindness and Cost Implications

The author built a benchmark called RealDataAgentBench to evaluate LLM agents on key dimensions beyond just getting the right answer, including correctness, code quality, efficiency, and statistical validity. The results revealed that while models can perform well on simple tasks, they often struggle with real-world data science challenges like handling confounding variables and reporting proper uncertainty.

Details

The author got tired of seeing LLM agents ace toy benchmarks but struggle with real-world data science tasks. So they built RealDataAgentBench, a test track that grades agents on four key dimensions: correctness, code quality, efficiency, and statistical validity. The benchmark uses fully reproducible seeded datasets and automatically scores each run. After 163+ experiments across 10 models, the author found that while GPT-4o and Claude Sonnet performed similarly overall, GPT-4o was dramatically cheaper per task. The Groq Llama models were fast and cheap but sometimes skipped statistical rigor. The biggest failures were not in correctness, but in statistical validity and code quality. This is expensive for companies, as choosing the wrong model can waste thousands in API costs and produce analyses that look correct but are statistically flawed. The author hopes RealDataAgentBench can help small and medium companies quickly test and select the right LLM model for their data needs.