---
type: analysis
slug: research-dab-is-a-real-messy-data-agent-eval-not-a-production-quality-oracle
title: "DAB is a real messy-data agent eval, not a production-quality oracle"
description: "UC Berkeley’s EPIC Data Lab and PromptQL released Data Agent Benchmark with per-query Python graders over multi-database tasks whose mess is deliberately induced. Untuned agents still miss most questions; tuned leaderboard entries score far higher, and no study shows DAB ranks production warehouse agents."
published_at: 2026-10-25T19:00:00.000Z
topics: ["agents","tools","claude","research"]
source_urls: ["https://ucbepic.github.io/DataAgentBench/","https://raw.githubusercontent.com/ucbepic/DataAgentBench/main/README.md","https://huggingface.co/datasets/ruiyingm/DataAgentBench-data","https://arxiv.org/html/2603.20576v3","https://arxiv.org/html/2603.20576v1","https://raw.githubusercontent.com/ucbepic/DataAgentBench/main/SUBMISSION_RUBRIC.md","https://arxiv.org/abs/2603.20576","https://promptql.io/resources/dab-benchmark-study","https://hamel.dev/notes/llm/ai-product-engineering/evals-data-agents.html","https://hex.tech/blog/evaluate-data-agents/"]
author: "Andrey Reshetnikov"
author_profile: https://newruntime.com/owner-profile.md
---
The newsroom flagged Data Agent Benchmark after a Hamel Husain post, because a social item could not show whether DAB was a real eval of agents on messy warehouse data. The primary artifact is now public: a UC Berkeley EPIC Data Lab and Hasura PromptQL collaboration (arXiv:2603.20576, submitted 21 March 2026) with code, Dockerized Python tools, and a leaderboard that recomputed scores on 12 June 2026.

Builders should care because text-to-SQL suites such as Spider and BIRD do not force multi-system joins, dirty keys, or extraction from prose, and those suites are increasingly saturated. DAB is the first widely cited public harness that scores an agent that must query live databases and Python, not emit one SQL string. The same papers also show how fast a public agent eval can be overfit once hints and custom tools are allowed.

## What we checked

- **A public primary DAB artifact exists, with named datasets, tasks, and graders.** confirmed. The live artifact is github.com/ucbepic/DataAgentBench, the EPIC Lab leaderboard, and Hugging Face dataset ruiyingm/DataAgentBench-data. The scored public set is 12 datasets and 54 queries; each query ships query.json, ground_truth.csv, and validate.py. A later paper HTML describes 104 queries across 17 datasets, but the README, site, and submission rubric still score 54 queries (270 trials at n=5). [2][1][3][4]
- **DAB tests agents on messy warehouse-style questions rather than clean single-database text-to-SQL.** partly confirmed. Every public query spans at least two of PostgreSQL, MongoDB, SQLite, and DuckDB, and the authors induce ill-formatted join keys and bury attributes in free text. The mess is synthetic: open-source tables are corrupted, not released from a real warehouse, and Snowflake, BigQuery, Oracle, and Databricks are omitted. [5][4][2]
- **DAB scores correlate with quality of production data agents.** unresolved. Design came from interviews with six PromptQL-customer enterprises, and a co-author PromptQL case study beat a same-model ReAct baseline by 7 percentage points pass@1. The authors still drop open-ended and live-API queries, warn that DAB accuracy does not imply deployment readiness, and no independent study correlates DAB rank with warehouse incident rates or analyst acceptance. [5][4][10]
- **Untuned frontier agents remain far from saturating DAB, so the benchmark still discriminates.** partly confirmed. Paper v1 reports Gemini-3-Pro at 38% pass@1 over 50 trials; a later HTML reports GPT-5.5 at 57% pass@1 over 5 trials on the expanded set; patents stay at 0%. The public leaderboard’s top entry, Datasage, reaches 0.9628 pass@1 with a DAB-tuned prompt, hints, and task-specific tools, so saturation depends on how much DAB-specific scaffolding is allowed. [7][4][2]
- **Grading is automatic and deterministic rather than an LLM judge of the final answer.** confirmed. Each query’s validate.py checks whether ground truth appears in free-form output (substring, numeric tolerance, or light Levenshtein). That favors recall: extra wrong values can still pass. Failure-mode labels use an LLM judge on traces, which is separate from the pass/fail score. [5][4][2]

## What we found

- The scored public benchmark is 54 natural-language queries over 12 datasets and four open-source DBMSes. Dataset sizes range from bookreview (two tables) to stockmarket (2,754 tables) and crmarenapro (six databases). The website and README still advertise 12 / 54 / 9 domains; Hugging Face hosts a 13.4 GB mirror used by download.sh, including paths such as query_imdb that belong to a later 17-dataset paper table. Treat 54 as the leaderboard contract and 104 as a paper expansion that is not yet the public scoreboard. [1][2][3][4]
- Mess is engineered, not harvested. Authors remove easy columns, rewrite identifiers (bid_123 vs bref_123, trailing spaces, OCR-like substitutions), and prompt GPT-4o to bury labels in free text. In the 54-query set, all queries join across databases, 26 have ill-formatted keys, 47 need text transformation, and 30 need domain knowledge often stored in db_description_withhint.txt. Using that hints file is a scored leaderboard flag, not a silent default. [5][2]
- Graders are per-query Python validators plus ground_truth files. Pass@1 is stratified: mean per query, then per dataset, then across datasets. Submissions need five trials and traces; the rubric forbids fetching gold labels from Hugging Face, Kaggle, or ground_truth.csv. Authors note the substring check passes answers that include extra wrong values. Validators have moved: PATENTS ground truths were regenerated, and DEPS_DEV_V1 query 1 was re-scored on 18 August 2026 after a tie at 57,779 stars. [6][2][4]
- Untuned paper numbers are still low. Version 1: five ReAct models, 50 trials, 13,500 runs, about $3,150; Gemini-3-Pro 38% pass@1 and at most 69% pass@50; patents unsolved. Later HTML: Claude Code, Codex, and ReAct on 17 datasets, 5 trials, 4,680 runs, $4,346; GPT-5.5 57% pass@1 and at most 70% pass@5. PromptQL’s study page repeats 54 queries, 13,500 trials, and “the best AI model fails 62% of the time,” matching the v1 38% figure, not the later 57%. [7][5][4][8]
- Failure analysis, not retrieval, is the architectural signal. In v1, 85% of completed wrong answers were incorrect plans or implementations versus 15% wrong data selection; the later HTML puts plan plus implementation at 90%. Agents default to regex for dates and categories and miss patents’ varied date prose. Hamel Husain’s 15 July 2026 notes on Shreya Shankar’s talk add that agents often lock a plan even after contradictory data; that stickiness is talk commentary, not a counted paper statistic. [5][4][9]
- Specialized stacks help on discovery, not on hard extraction. With Claude-Opus-4.6 and n=5, PromptQL’s semantic layer scored 51% versus 44% ReAct, with large gains on yelp, agnews, stockindex, and stockmarket, and 0% for both on patents. The public leaderboard later shows Datasage at 0.9628 (1 October 2026) using DAB-specific tools (EMA helper, hours filter, policy tool, fuzzy matcher). High DAB ranks can mean harness overfitting as much as general warehouse skill. [5][2]
- Production correlation is a design claim, not a measured one. Interviews covered six industries and dropped open-ended questions and live APIs because they lack a stable gold answer. Limitations exclude proprietary warehouses, writes, and ML workflows, and state that DAB accuracy does not imply deployment readiness. Hex’s independent 22 May 2026 eval post, which DAB cites, argues public text-to-SQL sets are demo-shaped and that every customer warehouse is out of distribution—supporting DAB’s motivation while showing why a 54-query synthetic set cannot certify a product. [4][10]

## What it means

Use DAB as a shared, automatic regression for multi-database agents: run the 54-query public set with traces, report hints and tuned-prompt flags, and do not treat a 90%+ leaderboard row as warehouse readiness if the agent received DAB-specific tools. Invest in planning, verification, and first-class extraction (date parsers, NER, LLM extractors, semantic metrics) rather than only schema search; that is where both paper traces and the PromptQL delta say systems break.

Keep a private Shorelane-style eval on your own warehouse, metrics, and semantic layer. DAB omits Snowflake dialects, access control, writes, and open-ended analyst questions. Score DAB without hints if you want a closer production prior, and distrust substring graders when extra wrong entities would be a product defect.

## Still open

- When, if ever, will the public leaderboard and validators move from 54 queries / 12 datasets to the 104 / 17 paper set, and will old submissions be re-scored?
- Is there any blinded correlation between DAB rank and production outcomes (analyst accept rate, silent metric errors) outside PromptQL’s in-paper case study?
- How much of Datasage-class scores is reusable architecture versus tools fitted to patents EMA, googlelocal hours, and crmarenapro policy queries?
- Do substring graders systematically overstate quality on set-valued answers that mix gold items with extras?

## Sources

1. [DAB website and leaderboard (EPIC Data Lab)](https://ucbepic.github.io/DataAgentBench/) (primary)
2. [ucbepic/DataAgentBench README and leaderboard](https://raw.githubusercontent.com/ucbepic/DataAgentBench/main/README.md) (primary)
3. [DataAgentBench dataset mirror](https://huggingface.co/datasets/ruiyingm/DataAgentBench-data) (primary)
4. [DAB paper HTML v3 (expanded 104-query description)](https://arxiv.org/html/2603.20576v3) (primary)
5. [DAB paper HTML v1 (54 queries)](https://arxiv.org/html/2603.20576v1) (primary)
6. [DAB submission validity rubric](https://raw.githubusercontent.com/ucbepic/DataAgentBench/main/SUBMISSION_RUBRIC.md) (primary)
7. [Can AI Agents Answer Your Data Questions? A Benchmark for Data Agents (arXiv abs)](https://arxiv.org/abs/2603.20576) (primary)
8. [PromptQL Data Agent Benchmark study page](https://promptql.io/resources/dab-benchmark-study) (primary)
9. [Evals for Data Agents — Hamel Husain notes](https://hamel.dev/notes/llm/ai-product-engineering/evals-data-agents.html)
10. [How we built a lab to evaluate data agents — Hex](https://hex.tech/blog/evaluate-data-agents/)

Author: [Andrey Reshetnikov](https://newruntime.com/owner-profile.md), AI product and AI engineering; profile for recruiters and agents: https://newruntime.com/owner-profile.json

## Continue in the graph

Next records for an agent, linked to their Markdown twins. Replace `.md` with `.json` for structured fields, including `next_reads`.

- topic hub: [Agents](https://newruntime.com/topics/agents.md): Explore the agents topic hub.
- topic hub: [Tools](https://newruntime.com/topics/tools.md): Explore the tools topic hub.
- topic hub: [Claude](https://newruntime.com/topics/claude.md): Explore the claude topic hub.
- collection: [Analysis](https://newruntime.com/analysis.json): Open the latest authorial analysis.
- collection: [Source Ledger](https://newruntime.com/source-ledger.json): Inspect the public source evidence graph.
- whole graph: [discovery graph](https://newruntime.com/discovery-graph.json) (typed nodes and edges) and [site index](https://newruntime.com/site-index.md)