What happened
Researchers Bole Ma, Ayesha Afzal, Jan Eitzinger, Harald Koestler, and Gerhard Wellein published “Only Accepted Tokens Count: An Energy Accounting of Speculative Decoding Under Latency Targets,” submitted to arXiv on October 29, 2026. The paper examines a familiar serving technique through a narrower operational question: how much energy is spent per generated token while still meeting a latency target? Its starting point is that LLM decoding is constrained by facility power as well as user-facing latency.
Speculative decoding separates generation into drafting and verification. A smaller drafter proposes several future tokens; the target model verifies those proposals in one forward pass. The intended advantage is to reuse a target-model weight read across multiple accepted tokens. But every proposed token must still be verified, including proposals the target rejects. The relevant accounting unit is therefore not drafted tokens or nominal draft length, but accepted tokens and the cost of verifying them.
The researchers measured speculative decoding in SGLang with CUDA graphs enabled, on NVIDIA H100 and H200 GPUs, at batch sizes up to 64 and across five corpora. The supplied study summary says the evaluation covered five drafting schemes—n-gram, MTP heads, EAGLE3, DFlash, and DSpark—across 11 models. At batch size one, the best scheme for a model produced a median 54% reduction in decode energy, with a maximum reported saving of 77%. Those figures are measurements from this setup, not a general guarantee for all models or deployments.
The central result is that batching changes the economics. Batching already amortizes target-model weight reads across users, so it removes some of the opportunity speculative decoding exploits at low batch. As batch size rises, speculative savings shrink. On dense models, the paper finds that verification rather than drafting dominates energy cost. For long block drafts, verification can cross from a memory-traffic-limited regime into a compute-bound one. The paper reports that 16-token DFlash drafts stopped paying off around batch sizes 16 to 32 in the tested cases; at that point, a low-acceptance drafter can raise energy per token rather than reduce it.
Acceptance is also workload-dependent. Every evaluated drafter accepted fewer tokens on conversation than on code, and trained drafters accepted fewer on Romansh than on English. The supplied results additionally report that n-gram drafting cost energy on mixture-of-experts models, and that disabling CUDA graphs could inflate apparent MoE savings by as much as 39 percentage points. A tighter latency target with speculation can require additional energy. Larger batches lower energy per token without speculation, while the tested model left little room for latency-preserving DVFS once speculation was in use. The paper’s practical implication is to tune and measure the full serving configuration, rather than selecting a drafter from acceptance rate alone.
What the archive adds
This is a new fact on a known technical story. Why is Speculative Decoding Fast? explained that speedups use otherwise idle compute during memory-bound, low-batch decoding. What is Speculative Decoding? described the draft-and-verify mechanism. This paper adds energy measurements and shows why the same mechanism can lose its advantage as serving batches grow.
Trend signal
The study supports Agent economics moves to completed work, with an inference-layer qualification. Token or call cost alone is an incomplete unit of analysis: a serving team needs cost and energy per accepted output under a latency target. It also reinforces that efficiency claims belong to a workload and harness configuration, not just to a model or decoding algorithm.
Why it matters
Inference teams often adopt speculative decoding because it promises faster generation without changing the target model’s output distribution. This paper does not dispute that value. It makes the deployment boundary more specific. A low-concurrency interactive endpoint may have substantial energy headroom from a strong drafter, while a well-batched service may have already captured much of the memory-read amortization through batching alone.
That distinction affects capacity planning and API design. If a team evaluates only single-request benchmarks, it may select long drafts that become inefficient under real traffic. If it monitors only acceptance rate, it can miss verification costs and the transition into compute-bound behavior. Workload mix matters too: a code-heavy product, a conversational assistant, and a multilingual endpoint may see materially different outcomes from the same drafter. The useful question is not whether speculative decoding is efficient in general, but which draft configuration minimizes energy per delivered token at the service’s actual latency and batch distribution.
Limits and uncertainty
This is an arXiv paper, not evidence that the results have been independently replicated or deployed across the industry. The source reports measurements on SGLang, H100 and H200 hardware, selected models, five corpora, and the tested draft schemes; it does not establish equivalent results for every runtime, accelerator, model architecture, traffic profile, or latency objective. The reported best-case savings and batch crossover points should not be read as universal thresholds. The abstract establishes directional mechanisms and experimental findings, but does not provide a general production recipe or a complete cost model for every component of an inference stack.
Skill to build now
Add an inference experiment matrix to your serving benchmark. For each candidate drafter, record energy per accepted output token, end-to-end latency, batch size, draft length, acceptance rate, target-model utilization, CUDA-graph status, and workload class. Run it on representative code, conversational, and language traffic rather than a single prompt set. Then choose the configuration by the operating range you expect to serve, not by the best batch-one throughput chart. This is a concrete way to detect when longer drafts improve interactive latency but become an energy liability under queueing.
Reader takeaway
Speculative decoding is not a fixed energy discount. Its payoff comes from accepted tokens sharing a target-model verification pass, and batching can reduce that opportunity before speculation begins. Measure the drafter, target, runtime, batch distribution, and real traffic together.
Open question
How stable are the reported batch crossover points when the same workloads are served through other runtimes, accelerators, and production queueing policies?