What happened
Boo Bench is presented by LlamaIndex as a “haunted house” of pages intended to make parsers repeat themselves or stop parsing properly. The source defines a “Boo” as a page element that scares a parser into a failure. The benchmark’s motivation is that some pages can trigger repetition loops or parsing failures even when specialized document OCR APIs handle them well. LlamaIndex frames the set as a simple litmus test for why document processing may require specialized models, tools, and agent harnesses rather than sending every document straight to a VLM.
The set contains 179 pages in four groups: 100 real pages featuring leaders, blanks, rows of 0.00, and repeated lines; 48 trap pages covering 16 tricks, including near-identical rows and confusion between O and 0; 16 synthetic “ghost” pages with grids and tallies of Halloween emoji; and 15 scans in scripts models rarely read. The public benchmark page also links to the dataset on Hugging Face under the name llamaindex/BooBench.
LlamaIndex says 37 document parsers and vision-language models were tested across all 179 pages. Its “ghost-free” score is the share of pages for which a system returned an answer without a Boo. A blank answer counts as a failure, as does still looping when the ten-minute clock runs out. The page distinguishes failure types, including repeated output, a silent or empty result, and a run that remains in progress when the time limit expires. This makes the score more than a conventional measure of whether a returned answer was correct: it captures whether the system completes at all under adversarial or awkward page conditions.
The benchmark board lists LlamaParse Agentic and Agentic Plus, Datalab Accurate and Fast, among others, at 100% (179/179). GPT-6 Luna and GPT-6.1 Terra each appear at 99.4% (178/179). The page reports individual failure details too: examples include repeated fill characters, repeated “1.0000” values, and runs of repeated script characters. The published material identifies the dataset and scoring rules, but does not provide a separate independent evaluation or establish that the scores generalize beyond this set of pages.
What the archive adds
This is a new benchmark in a continuing document-extraction thread. ExtractBench Pro: The Long Tail of Document Extraction argues that document benchmarks gain value by testing cross-page rules and visual relations, while OmniParseBench coverage describes a broad, auditable OCR test set. Boo Bench adds a targeted test for repetition and completion failures, not a restatement of either release.
Trend signal
Boo Bench supports verification bandwidth is the scarce engineering resource: a focused, inspectable test set reduces the ambiguity involved in deciding whether a document pipeline can be trusted. It does not establish a general industry trend, but it makes one operational dimension—finishing without runaway repeated output—observable and testable.
Why it matters
A parser can appear useful on ordinary pages and still fail expensively on a repeated row, blank-heavy form, or unusual script. A loop can inflate output, consume runtime, or leave a pipeline without a usable result. Boo Bench makes those cases visible and gives teams a shared set of examples to reproduce them against their chosen parser or VLM.
The practical implication is to evaluate document systems on completion and failure behavior as well as extraction quality. A team can use the dataset to identify whether its stack silently returns nothing, repeats content, or stalls, then decide where to add limits, fallback parsers, or review. The benchmark does not show that one component will solve every document task; it supplies a focused stress test for one class of failures that a production workflow should not discover only after deployment.
Limits and uncertainty
The supplied source describes the benchmark and its results, but does not establish that 179 pages represent the full diversity of production documents or that its rankings predict performance on a particular organization’s files. The “ghost-free” score combines several kinds of failure—empty output, repeated output, and timeout—so it is not a direct measure of semantic accuracy. The page does not specify comparable costs, latency outside the ten-minute cutoff, or the full evaluation conditions needed to reproduce every result. Nor does it show whether scores hold across versions, languages, layouts, or future model updates.
Skill to build now
Run the published set through the exact document path used in production: upload, parsing call, downstream cleanup, and any extraction step. Record whether each page completes, how much output it produces, and whether a timeout or empty response is visible to the calling system. Then define a bounded response for each failure type: terminate the run, retry through a fallback parser, or route the page for review. This is more useful than adding a generic prompt instruction not to repeat text, because the test targets the observable behavior of the whole pipeline.
Reader takeaway
Treat Boo Bench as a targeted robustness check, not a universal OCR leaderboard. Its contribution is to make repetition loops, empty results, and non-completion visible in a public test set. Teams processing scanned documents can add these cases to evaluation and verify that their pipeline finishes with bounded, usable output—not merely that it performs well on clean examples.