{"type":"post","stable_id":"post:cohere-hardware-aware-dynamic-speculative-decoding","slug":"cohere-hardware-aware-dynamic-speculative-decoding","title":"Cohere Makes Speculative Decoding A Hardware-Aware Scheduling Decision","description":"Cohere's Dynamic Speculative Decoding chooses speculation depth from hardware and load profiles because one fixed K can regress at high batch sizes.","retrieval_nugget":"The optimization is a scheduler decision informed by offline goodput profiles: dense and MoE models respond differently, and the optimal speculative depth changes with batch size and hardware contention.","published_at":"2026-07-10","updated_at":"2026-08-05","record_date":"2026-07-10","date_kind":"published_at","topics":["inference","speculative-decoding","hardware","vllm"],"entities":["Cohere","Command A","vLLM"],"source_urls":["https://cohere.com/blog/hardware-aware-dynamic-speculative-decoding"],"source_format":"article","editorial_timing":{"lane":"regular_hourly_analysis","scheduled_at":"2026-08-07T12:00:00+03:00","real_news_delta":"analytical follow-up"},"visual_decision":{"status":"included","reason":"the central mechanism is materially clearer as a diagram","reviewed_by":"codex"},"schema_version":"newruntime-agent-readable-v0.2","status":"published","visuals":[{"role":"hero","src":"/images/drip/cohere-hardware-aware-dynamic-speculative-decoding/cohere-hardware-aware-dynamic-speculative-decoding.webp","alt":"A New Runtime whiteboard diagram explaining cohere makes speculative decoding a hardware-aware scheduling decision.","caption":"New Runtime synthesis from Hardware-aware Dynamic Speculative Decoding."}],"routes":{"html":"https://newruntime.com/posts/cohere-hardware-aware-dynamic-speculative-decoding/","markdown":"https://newruntime.com/posts/cohere-hardware-aware-dynamic-speculative-decoding.md","json":"https://newruntime.com/posts/cohere-hardware-aware-dynamic-speculative-decoding.json"}}