Jul 6–12, 2026

New Runtime synthesis · Oct 27

Speculative decoding becomes a runtime scheduling choice because a fixed depth can slow high-batch inference

Inference optimization this week is a scheduling problem, not a one-time decode setting. Speculative decoding only helps if speculation depth matches the hardware and the current load; Cohere chooses that depth from hardware and load profiles because one fixed K can regress at high batch sizes. The shift that matters is that a serving speedup can invert when batches get large. Treating K as a constant is the risk, not the default.

That creates a contradiction for anyone buying or shipping speculative decoding as a blanket win. The technique remains in the stack, but a single K for all traffic can slow the system when batch sizes are high. What got riskier is static tuning of decode policy; what got more necessary is runtime visibility into batch size and hardware profile. Teams that can change speculation depth with load keep the upside; teams that freeze K in a config take the regression. The crowded claim is having speculative decoding; the distinguishing one is whether depth is scheduled.

For a builder, the consequence is to stop validating speculative decoding at a single batch and a single K. Measure where the note says it fails: high batch sizes on the hardware you actually run. For a buyer, the question is no longer whether the vendor uses speculation, but whether the serving path can vary depth with load or will regress as traffic packs. Watch next whether inference configs still expose one speculation depth or treat it as a dynamic scheduler input.

Key stories

1. Cohere Makes Speculative Decoding A Hardware-Aware Scheduling DecisionShows that speculation depth is not a universal speedup: a fixed K can regress at high batch sizes, so decode policy has to follow hardware and load.