---
type: "post"
id: "parsebench-2-severity-aware-document-parsing-benchmark-fa4acb"
slug: "parsebench-2-severity-aware-document-parsing-benchmark-fa4acb"
title: "ParseBench-2: Severity-aware document parsing benchmark"
description: "LlamaIndex says ParseBench-2 expands its document-parsing benchmark and recalibrates deterministic metrics against the severity of specific errors."
retrieval_nugget: "LlamaIndex says ParseBench-2 expands its document-parsing benchmark and recalibrates deterministic metrics against the severity of specific errors. It adds forms as a sixth evaluation dimension and more complete chart checks. For teams building document workflows, the change is a reminder that a parser score is meaningful only if its penalties reflect the cost of errors in the downstream task."
published_at: "2026-11-12"
updated_at: "2026-11-12"
record_date: "2026-11-12"
date_kind: "published_at"
topics: ["evals"]
entities: ["LlamaIndex","ParseBench-2","OCRia-1.5","LlamaParse","ParseBench-1"]
related_patterns: ["ai-engineering-evaluations-are-splitting-into-task-specific"]
novelty_label: "new_fact"
what_changed: "LlamaIndex introduces ParseBench-2, a severity-aware document-parsing benchmark that expands its earlier benchmark work."
prior_refs: ["https://newruntime.com/posts/introducing-the-document-parsing-protocol-927f14/","https://newruntime.com/posts/llamaindex-publishes-boo-bench-a-179-page-benchmark-354b9c/"]
source_url: "https://www.llamaindex.ai/blog/introducing-parsebench-v2"
source_urls: ["https://www.llamaindex.ai/blog/introducing-parsebench-v2"]
schema_version: "newruntime-agent-readable-v0.2"
stable_id: "post:parsebench-2-severity-aware-document-parsing-benchmark-fa4acb"
status: "published"
visuals: []
editor: {"role":"editor","name":"Andrey Reshetnikov","profile":"https://newruntime.com/owner-profile.md","profile_json":"https://newruntime.com/owner-profile.json","same_as":["https://www.linkedin.com/in/reshetnikov1/","https://t.me/lapetuse"]}
routes: {"html":"https://newruntime.com/posts/parsebench-2-severity-aware-document-parsing-benchmark-fa4acb/","markdown":"https://newruntime.com/posts/parsebench-2-severity-aware-document-parsing-benchmark-fa4acb.md","json":"https://newruntime.com/posts/parsebench-2-severity-aware-document-parsing-benchmark-fa4acb.json"}
markdown_url: "https://newruntime.com/posts/parsebench-2-severity-aware-document-parsing-benchmark-fa4acb.md"
---

# ParseBench-2: Severity-aware document parsing benchmark

## Retrieval answer

LlamaIndex says ParseBench-2 expands its document-parsing benchmark and recalibrates deterministic metrics against the severity of specific errors. It adds forms as a sixth evaluation dimension and more complete chart checks. For teams building document workflows, the change is a reminder that a parser score is meaningful only if its penalties reflect the cost of errors in the downstream task.

## Summary
LlamaIndex says ParseBench-2 expands its document-parsing benchmark and recalibrates deterministic metrics against the severity of specific errors. It adds forms as a sixth evaluation dimension and more complete chart checks. For teams building document workflows, the change is a reminder that a parser score is meaningful only if its penalties reflect the cost of errors in the downstream task.

## What happened
In an announcement dated November 12, 2026, LlamaIndex introduced ParseBench-2, an update to its benchmark for document parsing. The company says ParseBench has more than 50,000 dataset downloads per month and is cited by teams including NVIDIA, Cohere and Perplexity, as well as open-source projects. ParseBench-2 expands the initial corpus from 2,100 to 4,300 cases, with more real-world enterprise documents containing tables, charts and forms. LlamaIndex also says it corrected errors and clarified ambiguous annotations across the five original dimensions.

The central change is severity-tuned scoring. LlamaIndex describes a catalog of more than 330 error types across text, tables, formatting, charts, forms and layout. Starting from correct parsing outputs, the team introduced perturbations such as harmless markup changes, incorrect values and misaligned table headers. Each error type was classified as critical, major, minor or harmless, and metrics were calibrated to produce score changes aligned with those judgments. The company says explicit severity caps ensure that a serious error still carries a substantial penalty when the rest of a page is correct. Users can change the severity assigned to scenarios and choose the metrics that best match their priorities.

LlamaIndex illustrates the difference with a 459-cell table. Under ParseBench-1, changing a number reduced the score by 0.13 points out of 100, while returning the correct table in Markdown rather than HTML brought the score to zero. In ParseBench-2, the wrong number costs 10 points and the Markdown presentation change costs none. The stated aim is to distinguish a changed fact from an equivalent representation, rather than letting formatting similarity dominate the result.

The expanded evaluation checks text transcription separately from reading order; recognizes equivalent Markdown and HTML formatting; and checks table values against headers and row labels, including multi-level headers and grouped rows. For charts, ParseBench-2 compares complete reference tables, covering 36,958 values across 1,025 charts. Its visual-grounding evaluation checks whether extracted blocks point to the corresponding locations on the page; LlamaIndex says the evaluation covers 538 pages and accepts either paragraph-sized or line-by-line blocks when their boxes locate the text correctly.

Forms are the sixth dimension. The benchmark checks whether values, checkbox states and signatures remain attached to the correct fields, and whether blank fields remain blank. LlamaIndex describes a core form dataset of 500 pages from 244 government form templates in 30 countries: 450 pages have synthetic typed values and 50 are blank. A separate set adds 100 mostly blank public forms. The announcement says the dataset, evaluation code and scientific paper are publicly available. It also introduces OCRia-1.5, a family of document-parsing models for LlamaParse's Cost Effective, Agentic and Agentic Plus modes; the announcement characterizes these as a capability step up from OCRia-1.

## What the archive adds
New Runtime's earlier coverage includes [a comparison of document parsers and benchmark design](https://newruntime.com/posts/figuring-out-which-document-parser-to-use-can-80bc24/) (October 24, 2026), [the Document Parsing Protocol](https://newruntime.com/posts/introducing-the-document-parsing-protocol-927f14/) (October 30, 2026), and [LlamaIndex's Boo Bench](https://newruntime.com/posts/llamaindex-publishes-boo-bench-a-179-page-benchmark-354b9c/) (October 31, 2026). ParseBench-2 is a new benchmark release on a known document-evaluation thread, with a specific change to severity-weighted scoring and coverage.

## Trend signal
This does not yet fit a pattern New Runtime tracks. The release concerns evaluation design for document parsers; the available pattern theses do not directly describe severity-calibrated benchmark metrics.

## Why now
LlamaIndex presents ParseBench-2 alongside OCRia-1.5, the company's new document-parsing model family for three LlamaParse modes. The announcement positions the benchmark as a way to evaluate the capability changes, but does not establish that the benchmark caused or independently verifies those changes.

## Why it matters
A single aggregate score can hide the failure that matters most to a workflow. A parser might preserve nearly all text while changing a value, associating it with the wrong table header, or losing whether a form field was blank. If the evaluation treats those failures like minor formatting differences, choosing the highest-scoring parser may expose a team to errors that are difficult to catch downstream.

ParseBench-2's scenario catalog offers teams a more explicit starting point: decide which errors are critical for the documents and decisions they handle, then inspect whether the scoring reflects those priorities. Its customizable severity policy also makes clear that no universal ranking substitutes for task-specific evaluation. The benchmark does not demonstrate that one parser is best for every workflow; it changes the questions a careful comparison can ask.

## Limits and uncertainty
The benchmark and its severity policy are published by LlamaIndex, which also develops LlamaParse and OCRia models. The announcement's examples illustrate metric behavior, but do not independently establish that its severity assignments match every team's production costs. The described dataset sizes and checks do not show how well the benchmark represents every document type, language or scanning condition. The OCRia-1.5 description is a company characterization; this announcement alone does not independently verify model performance or establish how the new models compare across teams' own workloads.

## Skill to build now
Build a small error-severity rubric from your own production documents before comparing parsers. Include examples such as a changed number, a value under the wrong header, lost checkbox state, missing text and harmless formatting variation. Score a representative sample, then check whether your chosen benchmark or evaluation script penalizes those cases in the order your workflow requires.

## Reader takeaway
ParseBench-2 makes a useful distinction explicit: document-parsing accuracy is not just how much output matches a reference, but how consequential each mismatch is. Use its expanded checks as a prompt to define your own error priorities, not as a substitute for testing on the documents and decisions your system actually handles.

Editor: [Andrey Reshetnikov](https://newruntime.com/owner-profile.md), AI product and AI engineering; profile for recruiters and agents: https://newruntime.com/owner-profile.json

## Continue in the graph

Next records for an agent, linked to their Markdown twins. Replace `.md` with `.json` for structured fields, including `next_reads`.

- topic hub: [Evals](https://newruntime.com/topics/evals.md): Explore the evals topic hub.
- collection: [Field Notes](https://newruntime.com/posts.json): Open the latest editorial analysis.
- collection: [Sources](https://newruntime.com/sources/): See the original sources this archive cites.
- whole graph: [discovery graph](https://newruntime.com/discovery-graph.json) (typed nodes and edges) and [site index](https://newruntime.com/site-index.md)