The add-on makes each test cheaper to run, but you still have to choose the topology.
Summary
AWS says SageMaker HyperPod Inference now runs NVIDIA Dynamo as a managed add-on, with multi-node workers, separate prefill and decode pools, and KV-aware routing. AWS's benchmark shows disaggregation doubling per-GPU throughput on one fixed 4,000-in/200-out workload. In the same week, Thinking Machines released the 594B-parameter Inkling 2 Small, Baseten described model-specific kernel work on DeepSeek V4.1 Flash, and Unsloth shipped GGUF quants of Ornith-1.5-122B. Our read: managed packaging removes the installation and operations work, but it does not tell you which serving topology suits your traffic.
What happened
According to AWS's post, the HyperPod Inference add-on installs the Dynamo custom resources plus four components. The HyperPod Inference Operator is driven by an InferenceEndpointConfig with a dynamoSpec block. The Dynamo operator manages a DynamoGraphDeployment. Grove coordinates pod groups, and the KAI Scheduler gang-schedules multi-node workers. HyperPod monitors node, GPU and EFA health and replaces failed nodes. Teams can bring their own DynamoGraphDeployment or let the operator generate one. On the operator path, it stages weights from S3 or FSx, handles TLS and endpoint registration, autoscales through KEDA, and supplies defaults for KV-aware routing and replica counts. vLLM is the default engine; SGLang and TensorRT-LLM are also supported. To move to a newer supported Dynamo release, you update the add-on.
AWS ran two benchmarks. The first used Qwen3-235B-A22B-FP8 with TensorRT-LLM on 16 H200s, 4,000-token inputs and 200-token outputs, with points kept only if they met a P90 TTFT under one second and at least 50 tokens per second per user. Disaggregated serving reached 207 output tokens per second per GPU at concurrency 80. Aggregated serving peaked at 101, at concurrency 32. The second test split Qwen3.5-397B across two nodes and kept 99.1% of single-node throughput. AWS says this model fits on one instance and was split only to get a controlled baseline.
The related releases show what teams now have to serve. The Inkling 2 Small model card lists 594B total and 17B active parameters under Apache 2.0, with at least 500GB of aggregated VRAM needed for the NVFP4 checkpoint. Baseten reports 991 output tokens per second for DeepSeek V4.1 Flash 1017 after kernel fusion, Programmatic Dependent Launch and host-loop changes. Unsloth's repository offers Ornith-1.5-122B from IQ1_M up to BF16, with instructions for llama.cpp, Ollama, vLLM and other runtimes.
What the archive adds
New Runtime's earlier coverage shows that disaggregated serving on AWS has been building for months. According to AWS's What's New feed, HyperPod added managed disaggregated prefill and decode in July 2026. AWS presented it as a fix for long-context requests stalling token generation, and said it routes short prompts around the split path to avoid KV transfer overhead. That routing choice already concedes that disaggregation does not help every request. An AWS post on bi-directional KV transfer in vLLM addressed multi-turn agentic TTFT by keeping decode-side KV blocks for a configurable TTL. vLLM's study of DeepSeek-V4.1-Flash at scale, covered by New Runtime on October 27, 2026, found that decode for that model is bound by weight reads and that Wide-EP was its main lever, which is a model-specific result. Dynamo also has competition. Google's llm-d 1.0 ships benchmark recipes and the Prism dashboard. The Serving Stack's November 10, 2026 comparison lists Dynamo 1.6.2 next to llm-d 1.0, AIBrix 0.8, Production Stack 0.1.14 and Buoyant's Kinference. NVIDIA's Dynamo 1.0 announcement named Baseten, ByteDance, CoreWeave, DigitalOcean and Together AI as production adopters.
Trend signal
The serving stack for open models is turning into a set of packaged layers. Hugging Face handles distribution. Dynamo, llm-d and AIBrix handle multi-node orchestration, and now AWS manages Dynamo inside HyperPod. Unsloth's quant ladders cover the low end. Each layer installs more easily than it did a year ago, but none of them comes with a topology choice that holds across workloads. AWS's split-pool win was measured on a short-output, prompt-heavy shape. On NVIDIA's own GPT-OSS-120B recipe page, which uses a different model, an agentic trace and 8 GPUs, aggregated vLLM serving shows higher per-GPU throughput than disaggregated on both B200 and H200, while disaggregated gives higher per-user speed on B200. That does not rebut AWS's result, because the model, engine and workload all differ. Read together, though, the two results say the winner depends on traffic shape and on which metric you optimize. Baseten's kernel work points the same way at a lower layer: its speedups come from one model's architecture, not from a general setting.
Why now
1) Sparse MoE open models need multi-GPU or multi-node capacity. AWS notes that GLM-5.3's FP8 weights exceed the 640GB on one ml.p5.48xlarge, and Inkling 2 Small's model card puts its NVFP4 floor at 500GB of aggregated VRAM. 2) Long-context and agentic traffic makes prefill interfere with decode. AWS gave this as the reason for its July 2026 managed disaggregation feature, and NVIDIA's Dynamo launch post gives the same reason. 3) Competing open serving stacks (llm-d 1.0, AIBrix, Production Stack, Kinference, per The Serving Stack's November 10 comparison) push each cloud to package one. 4) Specialist vendors compete on model-specific kernel work, as in Baseten's DeepSeek V4.1 Flash post, which raises the bar a default deployment is measured against.
Connections to other events
The constraint missing from the launch is evaluation: someone has to measure which topology fits a given workload. AWS's 2.05× figure depends on 4,000-token prompts, 200-token outputs, two SLO thresholds and a concurrency sweep in which the two topologies peaked at different points (80 against 32). Splitting prefill from decode is designed to help exactly that prompt-heavy shape. Both pools load the full weights, so for workloads that do not stall on prefill, disaggregation costs extra GPUs. AWS's best-practice list says as much: replay representative traffic, sweep concurrency, keep the points that meet your SLAs, and adjust topology before engine flags. The add-on cuts the cost of each of those experiments. It cuts setup time, TLS, weight staging and node recovery. It does not remove the experiments.
Inkling 2 Small shows where this bites. According to its model card, the NVFP4 checkpoint fits within a single 8-GPU H200 node (the card lists 4 H200s as one example configuration), so adopters do not need multi-node workers for the weights. They do still need to choose between aggregated and disaggregated serving, and set pool sizes, for their own prompt and output mix. The card names SGLang, vLLM, TokenSpeed, Unsloth and Hugging Face as deployment frameworks, which leaves the Dynamo configuration to the team. We have not measured how much of the cost gap between a default and a tuned deployment is due to topology. AWS's own numbers show a 2× swing on one shape, which suggests it is large enough to be worth measuring before committing capacity.
Why it matters
For builders: use managed Dynamo to cut the cost of each experiment, not as a reason to skip experiments. The asset worth building is a replay harness. Feed AIPerf or llm-d-benchmark your real prompt-length, output-length and prefix-reuse distributions, and have it produce SLA-filtered throughput-per-GPU curves for both topologies. For buyers: AWS's 207 and Baseten's 991 tokens per second describe those vendors' workloads. Ask for the input/output shape, concurrency and SLO behind any figure.
The strongest counter-argument has two parts. First, most teams may never self-host a 594B model. They buy per-token endpoints from Together, Baseten or similar vendors, so the tuning happens on the vendor side and practitioners can ignore it. Second, Dynamo recipes, AISimulate and the optimization agent skills are meant to automate sizing, and they could close the default-to-tuned gap for popular models within months. Both are plausible. Our answer: hosted APIs move the tuning to the vendor but do not remove it, and the per-token price then carries the vendor's tuning quality. Separately, AWS shipping managed disaggregation twice in five months suggests it expects self-hosting demand. Two observations would settle whose tuning matters. One is whether hosted per-token prices for new open models fall below what HyperPod customers report paying per token at matched SLOs. The other is whether Dynamo recipes for newly released models show up within weeks and match hand-tuned results.
What happens next
- AWS will move the HyperPod managed Dynamo add-on to a Dynamo release newer than 1.6.2 within three months. (3 months, 60%). Watch for: A new ai-dynamo/dynamo GitHub release followed by a HyperPod Inference add-on version bump in AWS documentation. Resolves on 2027-02-16, yes if: The AWS SageMaker HyperPod Dynamo deployment documentation page lists a supported Dynamo version higher than 1.6.2 by 2027-02-15.
- NVIDIA will not publish a Dynamo recipe for Inkling 2 Small within three months, leaving adopters to size topology themselves. (3 months, 60%). Watch for: Whether Inkling 2 Small appears in the Dynamo recipes browser. Resolves on 2027-02-16, yes if: The NVIDIA Dynamo documentation recipes page lists no recipe named for Inkling 2 Small on 2027-02-15.
- Baseten will publish at least part of its DeepSeek V4.1 Flash kernel optimizations as open source within three months. (3 months, 35%). Watch for: A Baseten GitHub repository or an upstream SGLang/DeepInfer pull request from Baseten containing the fused router or PDL changes. Resolves on 2027-02-16, yes if: Baseten's GitHub organization or engineering blog links a public repository or merged pull request containing DeepSeek V4.1 Flash kernel optimizations by 2027-02-15.
Limits and uncertainty
AWS published no Dynamo-specific pricing and no comparison with self-run Dynamo or hosted APIs, so total cost is unknown. Baseten's 991 tokens per second and its cost comparisons are vendor claims whose test conditions have not been independently checked. The NVIDIA GPT-OSS-120B numbers cover a different model, engine and workload from AWS's Qwen tests, so they show that results depend on the workload but do not directly contradict AWS. Independent benchmarks of Inkling 2 Small on any serving stack have not been published. We also cannot measure what share of real deployment cost comes from topology choice compared with kernels, utilization or instance pricing. If hosted endpoints win the market for very large open models, this take describes a vendor problem more than a practitioner one. New Runtime's archive has no comparable cases to calibrate against, so the probabilities here rest on judgment.
Competing readings
- Managed Dynamo removes setup work but establishes no universal topology winner; SLA-compliant throughput depends on traffic shape, so someone must still measure per workload (likely, about 57%). What would tell it apart: AWS's 2.05× result is tied to one 4,000-in/200-out shape and different peak concurrencies (aws.amazon.com); AWS itself instructs users to replay traffic and sweep concurrency against SLAs (aws.amazon.com); A different model and agentic workload on NVIDIA's recipe page favours aggregated serving on per-GPU throughput (docs.dynamo.nvidia.com)(aws.amazon.com).
- Defaults, recipes and simulators improve fast enough that managed disaggregated serving becomes near-automatic, and per-workload tuning stops mattering for most teams (unlikely, about 22%). What would tell it apart: The operator already ships routing and replica defaults (aws.amazon.com).
- Hardware floors push most teams to hosted per-token endpoints, so topology tuning becomes a vendor problem and practitioners can ignore it (unlikely, about 21%). What would tell it apart: Inkling 2 Small needs at least 500GB of aggregated VRAM to self-host (thinkingmachines.ai).
Base rate
The New Runtime archive could not be queried for comparable cases (canceling statement due to statement timeout), so it gives 0 cases: no base rate. New Runtime's archive returned 0 comparable cases and 0 matches. That is too thin to calibrate against, so the probabilities here are judgment calls, not base rates.
What would change this view
- By 2027-02-15, the Dynamo recipes page lists a recipe for Inkling 2 Small, and an independent benchmark shows it within 5% of a hand-tuned configuration's SLA-filtered throughput per GPU (model-listing signal): Recipes are catching up with releases fast enough that per-workload tuning is commoditizing.
- By 2027-02-15, AWS or NVIDIA publishes Dynamo-on-HyperPod benchmarks across at least three input/output shapes, including an output-heavy one, in which disaggregation beats aggregation on SLA-filtered throughput per GPU in every shape (signal): The topology choice depends less on workload than the current results suggest.
- By 2027-02-15, at least three hosted providers list Inkling 2 Small on their pricing pages at per-token prices that undercut a fully utilized HyperPod deployment at matched SLOs (pricing signal): Tuning is a vendor-side concern and most practitioners can skip it.
Skill to build now
Run a matched-SLA replay experiment. This week, export one day of production request logs into prompt-length, output-length and shared-prefix distributions and turn them into an AIPerf trace. Sweep concurrency for aggregated serving on one 8-GPU node and for a disaggregated split on the same GPU count. Keep only the points that meet one fixed P90 TTFT and per-user speed target, and record output tokens per GPU and cost per GPU-hour at each point. Then compare the two curves at the same SLA.
Reader takeaway
Before writing a dynamoSpec for any large open model, capture a week of real traffic as distributions of prompt length, output length and prefix reuse. Then run matched-SLA AIPerf sweeps for both aggregated and disaggregated topologies. Use the same P90 TTFT and per-user speed targets for both, keep only the points that meet them, and compare cost per GPU-hour of delivered output at those points. If your outputs are long relative to your prompts, do not assume AWS's 2.05× applies to you.
Open question
Will recipe generators like AISimulate and agent-driven tuning shrink the gap between default and tuned deployments to a few percent for each new MoE release? Or will each new architecture reopen a gap that only workload-specific measurement and kernel work can close?