Open-weight extractors hold on short pages, then lose the long lists

What we checked

• ExtractBench is a public LlamaIndex benchmark that scores self-hosted open-weight models on schema-guided document extraction. confirmed. The 31 July 2026 technical report and the 11 August 2026 LlamaIndex launch post both state that ExtractBench scored fourteen systems, including four self-hosted open-weight vision-language pipelines: Lift 9B, NuExtract3, Qwen3.6 35B-A3B, and Gemma4 26B. The public Hugging Face dataset and GitHub harness later add more open-weight runners, including Qwen3.8 Flash Next and Qwen3.8 27B. [1][4][2][6][5]
• Open-weight one-shot VLMs are competitive on short documents but fail completeness on long record lists, so they are not a drop-in substitute for iterative or specialized extractors on enterprise long-list workloads. confirmed. In the paper’s Table 2, Qwen3.6 35B-A3B reaches 93.1 percent value F1 on short documents and 26.8 percent on long ones; Gemma4 26B falls from 80.5 to 12.2 and NuExtract3 from 54.4 to 8.9. The current GitHub leaderboard.csv still shows every listed OSS row below 40 percent on the long split, with Qwen3.8 Flash Next at 37.74 percent long versus 94.82 percent short. [1][4][5]

What we found

• ExtractBench is a challenge-tagged, schema-guided corpus rather than a fixed-field KIE set. The Hugging Face card and paper agree on 370 documents and 4,869 pages, split as 252 short (≤10 pages, 615 pages), 98 medium (11–50 pages, 2,438 pages), and 20 long (>50 pages, 1,816 pages). Of those, 325 documents are real public records and 45 are synthetic long lists with values fixed before rendering; 134 have scanned page images and 55 have handwriting. Each document type shares one schema. Scoring is deterministic unified value F1 (order-insensitive record alignment, no LLM judge), plus word- and page-level grounding F1 where boxes are verified. [1][2][3]
• The original evaluation, run in June–July 2026, compared 14 systems in three families: commercial VLMs, coding agents, and specialized APIs, plus four self-hosted open-weight models named in Section 3.1: Lift 9B, NuExtract3, Qwen3.6 35B-A3B, and Gemma4 26B. Paper Table 2 overall value F1 is LlamaExtract Agentic Plus 95.6, Codex GPT-5.5 93.6, Qwen3.6 35B-A3B 87.3, Lift 9B 77.3, Gemma4 26B 66.2, and NuExtract3 47.9. The launch post repeats those figures and pairs them with Agentic Plus at 8.1 cents per page. Open-weight pipelines are omitted from cost plots because they have no vendor API price. [1][4]
• Length, not short-form accuracy, is where open and commercial one-shot VLMs separate. The paper reports that more than half of systems exceed 90 percent F1 on short documents, while commercial VLMs fall below 40 percent on long documents because recall collapses: returned values are often right, remaining records never come back. On the S4 “enormous table” slice (tables beyond a thousand rows), every VLM scores below 10 percent; Qwen3.6 35B-A3B is 1.3, Lift 9B is 1.2, NuExtract3 is 3.7, and Gemma4 26B is 0.0, against LlamaExtract Agentic Plus 95.9 and Reducto Deep Extract 95.3. Qwen is stronger than the other VLMs on scans and handwriting in the paper (above 92 percent on both P2 and P3), so perception is not the main OSS failure. [1][4]
• The live GitHub leaderboard is larger than the paper’s 14-system table and has moved some OSS numbers. In leaderboard.csv, Qwen3.8 Flash Next leads the OSS rows at 89.88 overall (94.82 short, 37.74 long), Qwen3.8 27B is 89.75 (38.45 long), Qwen3.6 35B is 88.11 (36.90 long), NuExtract3 is 81.02 (32.71 long), Lift Datalab 9B is 78.43 (24.78 long), and Gemma4 26B is 70.27 (28.88 long). NuExtract3’s overall score is far above the paper’s 47.9; the files opened here do not explain that jump. LlamaExtract Agentic Plus remains first at 96.17 overall and 7.50 cents per page in the README table. Dataset v1.1 (29 September 2026) tightened boxes and made minor value fixes without changing documents or schemas. [5][6][2]
• Open-weight and other one-shot VLMs score zero on grounding in both the paper and the current CSV, because the evaluated configurations do not return word- or page-level evidence. Even specialized APIs that do return boxes are far from solved: the paper’s best word-level grounding F1 is 46.4 for LlamaExtract Agentic Plus, versus 84.9 at page level. The current README grounding table is higher for LlamaExtract (word-level 83.25 for Agentic Plus) after v1.1 box updates, but OSS rows remain 0.00. Workflows that need an auditor to click the source span cannot stop at a self-hosted VLM as configured here. [1][5][6]
• The ranking is a vendor evaluation. LlamaIndex chose the corpus, wrote the schemas, built the grader, and ships LlamaExtract, which debuted at the top alongside a new Agentic Plus tier. On 13 August 2026, Datalab published an independent reproduction claiming the published 64.5 percent Datalab API score was mostly a harness bug: the adapter stripped *_citations and *_score but not *_meta confidence sidecars, so metadata keys were scored as extra wrong values. Datalab reports 91.3 percent after stripping _meta and 93.6 percent with format-agnostic matching, versus LlamaExtract’s 95.6. The current CSV lists Datalab Accurate + Balanced at 85.70, which is well above 64.5 and still below Datalab’s own corrected figure. Lift, the same vendor’s 9B open model without those sidecars, was 77.3 in the paper. [4][7][5]
• The work is reproducible enough to use as a regression suite, and the name collision is real. GitHub documents uv run extract-bench run <pipeline> against frozen schemas, Apache-2.0 code and data, and Harbor tasks that score value F1 without measuring grounding. LlamaIndex list pricing is $1.25 per 1,000 credits; default Agentic Plus extraction is 50 extract credits plus 10 Agentic parse credits per page (60 credits, 7.5 cents), matching the current README 7.50 cents figure, while the paper’s 8.1 cents reflects measured runs and a schema-size multiplier. Contextual AI’s separately named ExtractBench is a 35-document, five-schema PDF-to-JSON study and should not be mixed into these scores. [6][8][1][9]

What it means

Treat ExtractBench as a useful, public stress test for schema-guided extraction, not as an independent bake-off. If your documents are short forms, invoices, and scans, a self-hosted Qwen-class VLM can look close to managed APIs on value F1 and may beat small commercial VLMs; the paper even shows Qwen holding up on handwriting and scans. If your job is complete recovery of long holdings tables, creditor matrices, or thousand-row schedules, the same one-shot open models still drop most records. That is a pipeline problem (iteration, chunking, tools), not a leaderboard rounding error. Coding agents in the paper keep more recall on long documents at much higher token cost; specialized APIs that walk the document remain the systems that hold F1 on the long split.

Do not ship on the published ranking alone. Reproduce the harness on your schemas, pin dataset revision v1.0 versus v1.1, and inspect provider adapters for sidecar keys and surface-form mismatches of the kind Datalab documented. If reviewers must see a box, plan a grounding stage; OSS rows are still zero there. Price LlamaExtract from the credit table (about 7.5 cents per page at default Agentic Plus), and price open models as GPU time, which this benchmark does not report. Keep Contextual AI’s ExtractBench out of the same comparison.

Still open

• No independent lab outside the extraction-vendor set has published a full frozen-v1.0 reproduction of the open-weight scores.
• The files opened here do not explain NuExtract3’s move from 47.9 overall in the paper to 81.02 in leaderboard.csv, or Gemma4 26B’s move from 66.2 to 70.27.
• It is not settled whether LlamaIndex accepted Datalab’s remaining format-matching claims; the live CSV score of 85.70 sits between the original 64.5 and Datalab’s 93.6.
• GPU-hour and dollar cost of the self-hosted Qwen, Gemma, Lift, and NuExtract pipelines is not reported.