---
type: "post"
id: "extractbench-pro-the-long-tail-of-document-extraction-9e4ec3"
slug: "extractbench-pro-the-long-tail-of-document-extraction-9e4ec3"
title: "ExtractBench Pro: The Long Tail of Document Extraction"
description: "Thesis: Document-extraction benchmarks become more useful when they test cross-page rules, visual relations, and domain-specific conventions rather than only clean field lookup."
retrieval_nugget: "Thesis: Document-extraction benchmarks become more useful when they test cross-page rules, visual relations, and domain-specific conventions rather than only clean field lookup."
published_at: "2026-10-15"
updated_at: "2026-10-15"
record_date: "2026-10-15"
date_kind: "published_at"
topics: ["evals","retrieval","multimodal"]
source_urls: ["https://www.llamaindex.ai/blog/introducing-extractbench-pro"]
schema_version: "newruntime-agent-readable-v0.2"
stable_id: "post:extractbench-pro-the-long-tail-of-document-extraction-9e4ec3"
status: "published"
visuals: []
editorial_provenance: {"schema_version":"newruntime-editorial-copy-v1","content_status":"source_grounded_final","final_copy_sha256":"sha256:b7dc73c785e0afb9ec1a0b850f36114b156c6cc9255777afad5c7b334e4fda03","reviewed_at":"2026-10-15T23:07:15.668Z","source_evidence_count":1,"verified_claim_count":3,"site_analysis_schema_version":"newruntime-site-analysis-v1","site_object_kind":"field_note","observed_fact_count":3,"implication_count":2,"watch_condition_count":1,"related_record_count":0}
analysis: {"schema_version":"newruntime-site-analysis-v1","object_kind":"field_note","thesis":"Document-extraction benchmarks become more useful when they test cross-page rules, visual relations, and domain-specific conventions rather than only clean field lookup.","observed_facts":[{"text":"LlamaIndex says its original ExtractBench contained 370 documents and did not cover some complex customer documents.","source_urls":["https://www.llamaindex.ai/blog/introducing-extractbench-pro"]},{"text":"ExtractBench Pro contains 100 synthetically generated documents across six challenge categories, including document conventions, house vocabulary, counting rules, drawn relations, charts inside documents, and long chart sheets.","source_urls":["https://www.llamaindex.ai/blog/introducing-extractbench-pro"]},{"text":"The company says the data is generated from seeds and configurations and includes ground-truth annotations for extraction and grounding.","source_urls":["https://www.llamaindex.ai/blog/introducing-extractbench-pro"]}],"mechanism":"The benchmark varies document layouts and content while retaining deterministic generation and annotated answers. This lets evaluators probe whether a system can resolve dependencies across pages, interpret local terminology, derive values, and read diagrams or charts, rather than merely match visible text to a schema.","why_now":"LlamaIndex says the original benchmark was approaching saturation even as difficult customer documents remained outside its coverage.","implications":["Teams can use the benchmark to ask whether extraction systems handle long-tail document structures, not just common forms.","The first implication is that benchmark scores on conventional documents should not be treated as evidence of robustness to cross-page rules, jargon, or diagram-based data."],"evidence_boundary":"These dataset details and the claim that rigid pipelines struggle while agentic systems with tools can adapt come from LlamaIndex’s announcement; the post does not establish independent comparative performance or prove that synthetic documents represent the full distribution of real customer files.","watch_conditions":["Watch for independent evaluations reporting system-level results on ExtractBench Pro and for evidence that its synthetic challenge types predict failures on real customer documents."],"related_records":[],"new_branch_reason":"No prior New Runtime record was supplied for this specific benchmark extension, so this note opens a distinct record on long-tail extraction evaluation."}
editor: {"role":"editor","name":"Andrey Reshetnikov","profile":"https://newruntime.com/owner-profile.md","profile_json":"https://newruntime.com/owner-profile.json","same_as":["https://www.linkedin.com/in/reshetnikov1/","https://t.me/lapetuse"]}
routes: {"html":"https://newruntime.com/posts/extractbench-pro-the-long-tail-of-document-extraction-9e4ec3/","markdown":"https://newruntime.com/posts/extractbench-pro-the-long-tail-of-document-extraction-9e4ec3.md","json":"https://newruntime.com/posts/extractbench-pro-the-long-tail-of-document-extraction-9e4ec3.json"}
related_patterns: ["ai-engineering-evaluations-are-splitting-into-task-specific"]
---

# ExtractBench Pro: The Long Tail of Document Extraction

## Retrieval answer

Thesis: Document-extraction benchmarks become more useful when they test cross-page rules, visual relations, and domain-specific conventions rather than only clean field lookup.

Thesis: Document-extraction benchmarks become more useful when they test cross-page rules, visual relations, and domain-specific conventions rather than only clean field lookup.

Observed facts: LlamaIndex says its original ExtractBench contained 370 documents and did not cover some complex customer documents. ExtractBench Pro contains 100 synthetically generated documents across six challenge categories, including document conventions, house vocabulary, counting rules, drawn relations, charts inside documents, and long chart sheets. The company says the data is generated from seeds and configurations and includes ground-truth annotations for extraction and grounding.

Mechanism: The benchmark varies document layouts and content while retaining deterministic generation and annotated answers. This lets evaluators probe whether a system can resolve dependencies across pages, interpret local terminology, derive values, and read diagrams or charts, rather than merely match visible text to a schema. Why now: LlamaIndex says the original benchmark was approaching saturation even as difficult customer documents remained outside its coverage.

Implications: Teams can use the benchmark to ask whether extraction systems handle long-tail document structures, not just common forms. The first implication is that benchmark scores on conventional documents should not be treated as evidence of robustness to cross-page rules, jargon, or diagram-based data.

Evidence boundary: These dataset details and the claim that rigid pipelines struggle while agentic systems with tools can adapt come from LlamaIndex’s announcement; the post does not establish independent comparative performance or prove that synthetic documents represent the full distribution of real customer files.

Watch condition: Watch for independent evaluations reporting system-level results on ExtractBench Pro and for evidence that its synthetic challenge types predict failures on real customer documents.

Editor: [Andrey Reshetnikov](https://newruntime.com/owner-profile.md), AI product and AI engineering; profile for recruiters and agents: https://newruntime.com/owner-profile.json

## Continue in the graph

Next records for an agent, linked to their Markdown twins. Replace `.md` with `.json` for structured fields, including `next_reads`.

- topic hub: [Evals](https://newruntime.com/topics/evals.md): Explore the evals topic hub.
- topic hub: [Retrieval](https://newruntime.com/topics/retrieval.md): Explore the retrieval topic hub.
- topic hub: [Multimodal](https://newruntime.com/topics/multimodal.md): Explore the multimodal topic hub.
- collection: [Field Notes](https://newruntime.com/posts.json): Open the latest editorial analysis.
- collection: [Source Ledger](https://newruntime.com/source-ledger.json): Inspect the public source evidence graph.
- whole graph: [discovery graph](https://newruntime.com/discovery-graph.json) (typed nodes and edges) and [site index](https://newruntime.com/site-index.md)