---
type: "post"
id: "the-inference-stack-behind-ocria-b166ae"
slug: "the-inference-stack-behind-ocria-b166ae"
title: "The Inference Stack Behind OCRia"
description: "LlamaIndex says it increased OCRia-1 token generation from about 285 to about 2,800 tokens per second on a single NVIDIA B200 GPU by combining quantization, speculative decoding and implementation-level optimizations."
retrieval_nugget: "LlamaIndex says it increased OCRia-1 token generation from about 285 to about 2,800 tokens per second on a single NVIDIA B200 GPU by combining quantization, speculative decoding and implementation-level optimizations."
published_at: "2026-11-04"
updated_at: "2026-11-04"
record_date: "2026-11-04"
date_kind: "published_at"
topics: ["inference"]
entities: ["LlamaIndex","OCRia-1","DFlash","DFlash2","vLLM","ReplaySSM","Daniel Bustamante Ospina","NVIDIA B200"]
related_patterns: []
novelty_label: "new_angle"
what_changed: "Explains the inference-stack engineering behind OCRia, shifting from OCRia-1’s release and benchmark results to how decoding latency is optimized."
prior_refs: ["https://newruntime.com/posts/llamaindex-has-introduced-ocria-1-a-4-billion-652dae/"]
source_url: "https://www.llamaindex.ai/blog/the-inference-stack-behind-ocria"
source_urls: ["https://www.llamaindex.ai/blog/the-inference-stack-behind-ocria"]
schema_version: "newruntime-agent-readable-v0.2"
stable_id: "post:the-inference-stack-behind-ocria-b166ae"
status: "published"
visuals: []
editor: {"role":"editor","name":"Andrey Reshetnikov","profile":"https://newruntime.com/owner-profile.md","profile_json":"https://newruntime.com/owner-profile.json","same_as":["https://www.linkedin.com/in/reshetnikov1/","https://t.me/lapetuse"]}
routes: {"html":"https://newruntime.com/posts/the-inference-stack-behind-ocria-b166ae/","markdown":"https://newruntime.com/posts/the-inference-stack-behind-ocria-b166ae.md","json":"https://newruntime.com/posts/the-inference-stack-behind-ocria-b166ae.json"}
markdown_url: "https://newruntime.com/posts/the-inference-stack-behind-ocria-b166ae.md"
---

# The Inference Stack Behind OCRia

## Retrieval answer

LlamaIndex says it increased OCRia-1 token generation from about 285 to about 2,800 tokens per second on a single NVIDIA B200 GPU by combining quantization, speculative decoding and implementation-level optimizations.

## Summary
LlamaIndex says it increased OCRia-1 token generation from about 285 to about 2,800 tokens per second on a single NVIDIA B200 GPU by combining quantization, speculative decoding and implementation-level optimizations. The account matters to teams serving document workloads because it shows how a model’s decoding bottlenecks can respond to workload-specific engineering, while leaving important questions about quality and production conditions open.

## What happened
In a November 4, 2026 engineering post, LlamaIndex’s Daniel Bustamante Ospina describes work to accelerate inference for OCRia-1, the company’s document-parsing model. The workload takes images with a text prompt and produces a structured textual representation of the document. LlamaIndex says customers had identified latency as a recurring concern, and that dense pages can require longer outputs. The team therefore focused on generation speed across concurrent requests on one B200 GPU. It reports nearly a tenfold increase, from approximately 285 to approximately 2,800 tokens per second. The post frames this as a combined result of many improvements, not a single technique.

The first part of the work targets memory traffic during decoding. The post distinguishes prefill, which processes input and builds state, from decode, which generates tokens one at a time. For LlamaIndex’s B200 workload and batch sizes, it says decoding is largely memory-bandwidth-bound: the GPU spends too much time moving weights and state relative to the computation. The team used mixed-precision quantization with BF16, FP8 and NVFP4 weights, keeping more sensitive layers at higher precision. Its activation-aware approach used calibration data drawn from the intended workload, and the team evaluated combinations on held-out examples for both speed and accuracy. It says it did not quantize the vision encoder, to preserve visual accuracy, and describes the quantization as effectively lossless on its target tasks. The post does not provide the evaluation set size or numerical accuracy results.

LlamaIndex also used speculative decoding: a smaller drafter proposes several future tokens, then the target model checks them in parallel. The company says this can preserve the target model’s output distribution when verification is done correctly. Its drafter is DFlash, a block-diffusion model conditioned on hidden-state features from the target. The team reports an acceptance rate of nine tokens with its original DFlash model and more than twelve with DFlash2, which adds local convolutions and a candidate selector. At batch sizes of 8 to 12, it says generation exceeded 1,400 tokens per second per user, compared with about 200 for the base model at that stage of the work.

The remaining improvements address overhead in the inference stack. OCRia uses Gated Delta Network layers with recurrent state. For speculative decoding, rejected tokens can require state rollback. LlamaIndex adapted ReplaySSM, a checkpoint-and-buffer method, to these layers; the post says its work remained a series of draft pull requests to vLLM. At 32 concurrent requests, a decode step took 6.07 milliseconds, with verification and flush kernels accounting for 2.10 milliseconds across 24 GDN layers. A custom CUDA kernel reduced per-layer time to 23–24 microseconds from 72–80 microseconds for the upstream Triton implementation, around 3.3 times faster in that benchmark. The team also describes fusing operations to reduce intermediate memory transfers. The supplied account gives no overall before-and-after latency breakdown for these final optimizations.

## What the archive adds
[Our earlier note on OCRia-1](https://newruntime.com/posts/llamaindex-has-introduced-ocria-1-a-4-billion-652dae/) covered the model’s introduction and parsing benchmark. This is a new fact on that known story: LlamaIndex now details how it optimized inference for OCRia-1, rather than announcing another model release.

## Trend signal
No supplied SITE_PATTERNS entry directly fits this account. It documents inference implementation and throughput improvements, but does not establish a broader shift in agent economics, interoperability, or another listed pattern.

## Why now
LlamaIndex says latency was a recurring customer concern for document-parsing workflows and describes the optimization work as occurring while it trained and deployed OCRia-1. The post’s timing is therefore tied to serving that model and addressing a workload constraint; it does not announce a general change in product access or availability.

## Why it matters
For teams operating document models, the account is a reminder that generated-output workloads can be limited by memory movement and state handling, not just by the model’s nominal compute. Quantization, speculative decoding and specialized kernels each address different parts of that problem, and their benefits depend on the workload, hardware, batch size and model architecture. LlamaIndex’s numbers offer concrete reference points for the scale of improvement it achieved on a B200, but they are not a promise that another model or service will see the same gains. Builders evaluating similar work should measure both throughput and task quality on representative documents, including dense pages, and account for concurrency. The post’s reported lossless quantization is specific to its target tasks; independent reproduction and full evaluation details are not supplied here.

## Limits and uncertainty
These results come from LlamaIndex’s own engineering account. The supplied material reports token-generation rates and selected kernel timings, but does not specify the exact OCRia-1 version, complete benchmark methodology, document mix, output-length distribution, precision-by-layer configuration or hardware/software stack beyond the single B200 GPU reference. It does not give costs, end-to-end document latency, service-level results, or independent verification. “Effectively lossless” refers to the team’s target tasks and evaluations, whose size and numerical quality scores are not stated. The post also identifies ReplaySSM work as draft pull requests to vLLM, so the text does not establish that this component is part of a generally available upstream release.

## Reader takeaway
LlamaIndex attributes a large OCRia-1 throughput gain to a stack of workload-specific optimizations, not a model change alone. The useful lesson for builders is to profile decoding and state movement on their own representative workload, then verify quality alongside speed; these figures are a case report, not a portable performance guarantee.

Editor: [Andrey Reshetnikov](https://newruntime.com/owner-profile.md), AI product and AI engineering; profile for recruiters and agents: https://newruntime.com/owner-profile.json

## Continue in the graph

Next records for an agent, linked to their Markdown twins. Replace `.md` with `.json` for structured fields, including `next_reads`.

- topic hub: [Inference](https://newruntime.com/topics/inference.md): Explore the inference topic hub.
- collection: [Field Notes](https://newruntime.com/posts.json): Open the latest editorial analysis.
- collection: [Sources](https://newruntime.com/sources/): See the original sources this archive cites.
- whole graph: [discovery graph](https://newruntime.com/discovery-graph.json) (typed nodes and edges) and [site index](https://newruntime.com/site-index.md)