{"id":"nr-open-model-serving-is-being-optimized-as-a-stack-not-just-a-checkpoint-a42f4e","slug":"open-model-serving-is-being-optimized-as-a-stack-not-just-a-checkpoint-a42f4e","type":"raw_signal","title":"Open model serving is being optimized as a stack, not just a checkpoint","description":"Serving work spans quantized kernels and weight layouts, optimized expert-parallel routing, distributed inference, and provider comparisons showing cost and speed variation for the same model.","observed_at":"2026-10-25","record_date":"2026-10-25","date_kind":"observed_at","novelty":"notable","source_platform":"newruntime.com","topics":["agent-economics","model-routing"],"entities":["humans&","NVIDIA Blackwell","NVFP4","BF16","FlashInfer","MegaMoE","GLM-5","NVIDIA B300","CuTe DSL","vLLM"],"related_patterns":["agent-economics-moves-to-completed-work","harness-architecture-outlives-model-choice"],"source_url":"https://humansand.ai/blog/faster-than-light-nvfp4-decode","source_urls":["https://humansand.ai/blog/faster-than-light-nvfp4-decode","https://github.com/flashinfer-ai/flashinfer/pull/5319","https://github.com/vllm-project/vllm/pull/62009","https://www.vals.ai/benchmarks/inference_provider","https://cloud.google.com/blog/topics/ai-infrastructure/llm-d-1-0-an-open-source-stack-for-distributed-inference"],"why_it_matters":"Serving work spans quantized kernels and weight layouts, optimized expert-parallel routing, distributed inference, and provider comparisons showing cost and speed variation for the same model. The practical implication is that model choice alone does not determine agent performance or cost; runtime, hardware, precision, and serving configuration matter too.","verification_level":"source-linked","signal_type":"synthesis","synthesis_of":[{"slug":"engineers-at-humans-accelerated-mixture-of-experts-decode-840662","title":"Engineers at humans& accelerated Mixture-of-Experts decode on NVIDIA Blackwell GPUs by keeping weights in 4-bit NVFP4 while retaining activations in 16-bit BF16.","url":"https://newruntime.com/posts/engineers-at-humans-accelerated-mixture-of-experts-decode-840662/"},{"slug":"disaggregated-deployments-can-assign-precision-by-serving-phase-01b3fb","title":"Disaggregated deployments can assign precision by serving phase because both paths consume the same prepared NVFP4 weight layout.","url":"https://newruntime.com/posts/disaggregated-deployments-can-assign-precision-by-serving-phase-01b3fb/"},{"slug":"speculative-decoding-uses-the-smaller-draft-model-to-48c074","title":"Speculative decoding uses the smaller draft model to propose tokens and the target model to verify them.","url":"https://newruntime.com/posts/speculative-decoding-uses-the-smaller-draft-model-to-48c074/"},{"slug":"vals-ai-launched-the-inference-provider-index-comparing-60fbd1","title":"Vals AI launched the Inference Provider Index comparing open-weight model endpoints on agentic tasks.","url":"https://newruntime.com/posts/vals-ai-launched-the-inference-provider-index-comparing-60fbd1/"},{"slug":"announcing-llm-d-1-0-open-distributed-inference-7539eb","title":"Announcing llm-d 1.0: Open distributed inference for agents on Google Cloud TPUs and GPUs","url":"https://newruntime.com/posts/announcing-llm-d-1-0-open-distributed-inference-7539eb/"}],"schema_version":"newruntime-agent-readable-v0.2","stable_id":"signal:open-model-serving-is-being-optimized-as-a-stack-not-just-a-checkpoint-a42f4e","retrieval_nugget":"Serving work spans quantized kernels and weight layouts, optimized expert-parallel routing, distributed inference, and provider comparisons showing cost and speed variation for the same model.","status":"published","visuals":[],"editor":{"role":"editor","name":"Andrey Reshetnikov","profile":"https://newruntime.com/owner-profile.md","profile_json":"https://newruntime.com/owner-profile.json","same_as":["https://www.linkedin.com/in/reshetnikov1/","https://t.me/lapetuse"]},"routes":{"html":"https://newruntime.com/signals/open-model-serving-is-being-optimized-as-a-stack-not-just-a-checkpoint-a42f4e/","markdown":"https://newruntime.com/signals/open-model-serving-is-being-optimized-as-a-stack-not-just-a-checkpoint-a42f4e.md","json":"https://newruntime.com/signals/open-model-serving-is-being-optimized-as-a-stack-not-just-a-checkpoint-a42f4e.json"},"next_reads":[{"type":"topic","path":"/topics/agent-economics/","url":"https://newruntime.com/topics/agent-economics/","title":"Agent Economics","reason":"Explore the agent-economics topic hub.","media_type":"text/html","markdown_url":"https://newruntime.com/topics/agent-economics.md","json_url":"https://newruntime.com/topics/agent-economics.json"},{"type":"topic","path":"/topics/model-routing/","url":"https://newruntime.com/topics/model-routing/","title":"Model Routing","reason":"Explore the model-routing topic hub.","media_type":"text/html","markdown_url":"https://newruntime.com/topics/model-routing.md","json_url":"https://newruntime.com/topics/model-routing.json"},{"type":"collection","path":"/signals/","url":"https://newruntime.com/signals/","title":"Raw Signals","reason":"Open every source-linked raw signal.","media_type":"text/html","json_url":"https://newruntime.com/signals.json"},{"type":"collection","path":"/source-ledger.json","url":"https://newruntime.com/source-ledger.json","title":"Source Ledger","reason":"Inspect the public source evidence graph.","media_type":"application/json"}]}