Field note
Cohere published a hardware-aware Dynamic Speculative Decoding design for production inference. The premise is that a fixed speculation depth K is not reliably faster: the extra draft and verification work can reduce throughput when batch size or hardware contention changes.
The system builds offline profiles for acceptance length and inter-token-latency goodput, then looks up an appropriate K for the current model, hardware, and load. Dense models generally prefer a smaller K as batch size grows, while mixture-of-experts models can behave non-monotonically because expert loading changes the bottleneck.
On Command A, Cohere reports Dynamic Speculative Decoding about 23% faster than fixed-K at batch sizes 128 and 256. Against vanilla decoding, it reports 7.5% at batch 128 and 1.82% at batch 256, where fixed speculative decoding regressed. These are vendor benchmarks and depend on the tested hardware and workload.
The implementation integrates with vLLM's asynchronous scheduling and full CUDA graphs. The reusable idea is broader than this technique: inference optimizations should be selected by measured operating regime, not enabled as static model flags.