What happened
Epoch AI published the Inference Performance Index on November 9, 2026. The report, by researcher Jason Li, says inference performance gains can substantially exceed what changes in raw compute throughput suggest. Its headline estimate is that an Nvidia GB300 provides about 25 times the inference capacity per chip of an Nvidia H100 on the agentic workloads Epoch models, despite having only 2.5 times the 8-bit FLOP/s. Epoch attributes the gap to memory and inter-chip communication, which can matter more than arithmetic throughput during inference.
IPI Version 1 measures how many requests a chip can serve concurrently across a range of required output speeds. Scores are expressed in GB300 equivalents (GB300e), with GB300 set to one. A score of two means the chip is estimated to serve twice as many users or agents as GB300 on average across the models and output-speed requirements in the index. The comparison covers 50 to 400 tokens per second per request, with equal weighting for each doubling of the speed requirement.
The index combines model-specific capacity curves for four representative architectures that Epoch says it can accurately model using Nvidia’s AISimulate: GLM-5.2, Kimi K3, Qwen3-Coder-480B and a proxy for MAI-Base-2. It models prefill and decode separately, includes speculative decoding, and uses a workload with 262,144 cached context tokens and 1,024 new input tokens per prefill. Epoch says agentic API keys consumed roughly seven times as many tokens as human API keys on OpenRouter in late September 2026; that observation informs its focus on agentic inference.
The reported scores include 2.18 for VR200, 0.55 for MI355X, 0.54 for B300 HGX, 0.12 for H200 HGX and 1 for GB300 NVL72. Epoch estimates GB300 NVL72 has about 1.9 times the modeled inference capacity per GPU of B300 HGX, although both use Blackwell Ultra GPUs; the report points to their different scale-up domains, 72 GPUs versus eight. Scores describe modeled potential. Epoch says future versions will update the workload and model selection as inference demand changes, and advises using scores from the same version for hardware comparisons.
What the archive adds
New Runtime’s earlier coverage of Gemini’s position near the frontier on October 24, 2026 also used Epoch’s index, but addressed model capability rather than chip capacity. This is a new index release and a new infrastructure comparison, not a restatement of that note. Earlier coverage of inference provider differences provides a related view of endpoint cost and speed; IPI instead models underlying hardware capacity.
Trend signal
This supports the shift toward comparing inference infrastructure by the work it can serve, rather than relying on a single peak-compute number. Epoch’s measure combines concurrency and output speed, though it remains a modeled capacity index rather than cost per verified task. This does not yet fit a pattern New Runtime tracks.
Why now
Epoch says Version 1 focuses on agentic inference because agentic API keys accounted for roughly seven times the token consumption of human API keys on OpenRouter in late September 2026. The index is framed as a changing reference: the organization plans to update its workload and model basket as inference demand evolves.
Why it matters
For teams planning AI workloads, a chip’s published FLOP/s is a weak proxy for how many agent requests it can serve at an acceptable response speed. Long contexts consume memory, and larger decode batches can increase total throughput while making each individual response slower. Hardware selection therefore involves a concurrency and latency tradeoff, as well as the shape of the model and serving setup.
IPI offers a common unit for comparing modeled capacity across chips and estimating a fleet in GB300 equivalents. It can help infrastructure teams ask more precise questions about the workloads they expect to run, but it cannot replace measurements on their own models, software stacks and service-level targets. The report’s results also suggest that scale-up topology can matter even where GPU generations are shared. Treat the index as a screening tool for what to benchmark, not a deployment guarantee.
Limits and uncertainty
The IPI estimates optimal performance under modeled hardware limits; it does not measure what every production system will achieve. Epoch explicitly says the scores do not account for software maturity, which affects realized performance. The model basket is four architectures, including a proxy for MAI-Base-2, and future versions may change scores when workloads or model selection change. The report also notes that its precision choices have not been established to preserve equivalent model quality, and that speculative decoding assumes an average of three accepted output tokens per decoding cycle. Realized capacity and rankings can therefore differ by workload, implementation and software optimization.
Reader takeaway
The useful question is not simply how many FLOP/s a chip offers, but how many requests it can serve at the output speed and context size a product needs. Epoch’s index makes that tradeoff more visible, while remaining a model to validate against real serving workloads.