A small draft model proposes several tokens; the larger model verifies them in parallel in one forward pass. The article says single-token decoding is often memory-bound, leaving compute underused, while speculative verification can use that slack to advance one sequence faster. At higher throughput, speculation may waste compute, particularly when proposed tokens are rejected. Barrow notes that vLLM offers a batch-size setting to disable speculative decoding. The piece describes the mechanism and tradeoff; it does not provide a general performance guarantee or benchmark results. Source: https://jbarrow.ai/2026-10-08-why-is-speculative-decoding-fast
Raw SignalOct 10, 20261 min read1 source
Why is Speculative Decoding Fast? (4 minute read)
In one minute
Joe Barrow’s explainer argues that speculative decoding can accelerate low-batch inference not by reducing total computation, but by changing how available GPU compute is used.