---
type: "post"
id: "managed-nvidia-dynamo-on-amazon-sagemaker-hyperpod-inference-fb097d"
slug: "managed-nvidia-dynamo-on-amazon-sagemaker-hyperpod-inference-fb097d"
title: "Managed NVIDIA Dynamo on Amazon SageMaker HyperPod Inference"
description: "AWS says Amazon SageMaker HyperPod Inference now supports NVIDIA Dynamo for distributed model serving, including multi-node deployments, split prefill and decode pools, and KV-aware routing."
retrieval_nugget: "AWS says Amazon SageMaker HyperPod Inference now supports NVIDIA Dynamo for distributed model serving, including multi-node deployments, split prefill and decode pools, and KV-aware routing. The integration matters to teams serving large models because it packages orchestration and node recovery into a managed path, while AWS’s benchmarks illustrate the performance trade-offs of specific configurations."
published_at: "2026-11-18"
updated_at: "2026-11-18"
record_date: "2026-11-18"
date_kind: "published_at"
topics: ["inference"]
entities: ["Amazon Web Services","Amazon SageMaker HyperPod","NVIDIA Dynamo","vLLM","SGLang","TensorRT-LLM","GLM-5.3","Qwen3-235B"]
related_patterns: ["agent-economics-moves-to-completed-work"]
novelty_label: "new_fact"
what_changed: "AWS adds managed NVIDIA Dynamo support to SageMaker HyperPod Inference, providing a new managed deployment option for inference workloads."
prior_refs: []
source_url: "https://aws.amazon.com/blogs/machine-learning/managed-nvidia-dynamo-on-amazon-sagemaker-hyperpod-inference"
source_urls: ["https://aws.amazon.com/blogs/machine-learning/managed-nvidia-dynamo-on-amazon-sagemaker-hyperpod-inference"]
schema_version: "newruntime-agent-readable-v0.2"
stable_id: "post:managed-nvidia-dynamo-on-amazon-sagemaker-hyperpod-inference-fb097d"
status: "published"
visuals: []
editor: {"role":"editor","name":"Andrey Reshetnikov","profile":"https://newruntime.com/owner-profile.md","profile_json":"https://newruntime.com/owner-profile.json","same_as":["https://www.linkedin.com/in/reshetnikov1/","https://t.me/lapetuse"]}
routes: {"html":"https://newruntime.com/posts/managed-nvidia-dynamo-on-amazon-sagemaker-hyperpod-inference-fb097d/","markdown":"https://newruntime.com/posts/managed-nvidia-dynamo-on-amazon-sagemaker-hyperpod-inference-fb097d.md","json":"https://newruntime.com/posts/managed-nvidia-dynamo-on-amazon-sagemaker-hyperpod-inference-fb097d.json"}
markdown_url: "https://newruntime.com/posts/managed-nvidia-dynamo-on-amazon-sagemaker-hyperpod-inference-fb097d.md"
---

# Managed NVIDIA Dynamo on Amazon SageMaker HyperPod Inference

## Retrieval answer

AWS says Amazon SageMaker HyperPod Inference now supports NVIDIA Dynamo for distributed model serving, including multi-node deployments, split prefill and decode pools, and KV-aware routing. The integration matters to teams serving large models because it packages orchestration and node recovery into a managed path, while AWS’s benchmarks illustrate the performance trade-offs of specific configurations.

## Summary
AWS says Amazon SageMaker HyperPod Inference now supports NVIDIA Dynamo for distributed model serving, including multi-node deployments, split prefill and decode pools, and KV-aware routing. The integration matters to teams serving large models because it packages orchestration and node recovery into a managed path, while AWS’s benchmarks illustrate the performance trade-offs of specific configurations.

## What happened
AWS’s announcement describes support for NVIDIA Dynamo through the SageMaker HyperPod Inference add-on. Dynamo is an open-source distributed inference framework that supports multi-node deployment, disaggregated prefill and decode, and KV-aware routing with engines including vLLM, SGLang, and TensorRT-LLM. HyperPod contributes Elastic Fabric Adapter networking, coordinated GPU scheduling, and node health monitoring and recovery. AWS Deep Learning Containers provide the Dynamo runtime; updating the add-on moves a deployment to a supported Dynamo release.

The add-on installs serving and scheduling components. The HyperPod Inference Operator manages endpoints through an InferenceEndpointConfig (IEC), whose `dynamoSpec` supplies Dynamo configuration. The Dynamo operator manages a DynamoGraphDeployment, including the frontend, model workers, replica counts, and resource requirements. Grove coordinates related pod groups and scaling relationships, while the Kubernetes AI Scheduler places pods on GPUs and uses gang scheduling to place the pods required for a multi-node worker together.

AWS presents two configuration paths. Teams can bring their own DynamoGraphDeployment and retain direct control over the serving graph, using a supported container image and configuring storage, networking, and pod placement. Alternatively, teams can configure an IEC with `dynamoSpec`; the operator stages weights from S3 or Amazon FSx and manages endpoint, TLS, registration, and autoscaling through KEDA. The IEC path uses vLLM by default and offers SGLang and TensorRT-LLM as alternatives. Both approaches support single-instance and multi-node deployments, as well as aggregated or disaggregated serving.

In disaggregated mode, the frontend sends prompts to prefill workers and generation requests to decode workers. The pools can be scaled and tuned independently, while KV cache moves between them over EFA. AWS’s GLM-5.3 example configures one worker replica across two `ml.p5.48xlarge` instances, with two frontend replicas; its disaggregated example uses one two-instance prefill replica and two two-instance decode replicas, for six instances total. The post also gives a SageMaker Runtime API invocation example using an OpenAI-compatible chat-completions payload.

AWS reports two benchmarks. For Qwen3-235B-A22B-FP8 with TensorRT-LLM on two p5en.48xlarge instances (16 H200 GPUs), requests had 4,000 input and 200 output tokens. Among configurations meeting P90 time-to-first-token below one second and median generation speed of at least 50 tokens per second per user, disaggregated serving reached 207 output tokens per second per GPU versus 101 for aggregated serving. For Qwen3.5-397B-A17B-FP8 with vLLM, an eight-GPU, two-instance configuration reached 126.4 output tokens per second per GPU, which AWS says was 99.1% of the single-node throughput under its agentic workload with high prefix reuse at concurrency 32.

## What the archive adds
New Runtime’s earlier coverage described SageMaker HyperPod multi-region training, where data and compute locality were separated ([Multi-Region training with Amazon SageMaker HyperPod and Qumulo](https://newruntime.com/posts/multi-region-training-with-amazon-sagemaker-hyperpod-and-103b05/), 2026-09-26). This is a new inference-serving event on the same managed infrastructure, not a restatement of that training story. Earlier coverage also noted Multi-Instance GPU support for SageMaker Inference ([SageMaker Inference now supports Multi-Instance GPU to maximize GPU utilization for model endpoints](https://newruntime.com/posts/sagemaker-inference-now-supports-multi-instance-gpu-to-0c0db5/), 2026-11-07); Dynamo adds distributed serving topologies and their orchestration rather than GPU partitioning alone.

## Trend signal
This supports [agent economics moves to completed work](https://newruntime.com/patterns/agent-economics-moves-to-completed-work/) only indirectly: AWS evaluates throughput under latency and generation-speed constraints, rather than treating raw token rate as the sole measure. The announcement does not report cost per verified completed task, so it is not evidence that completed-work economics has arrived; it shows how serving configuration and workload shape the measurable capacity available to agentic systems.

## Why it matters
The deployment choice changes both infrastructure responsibilities and the shape of performance tuning. A team that needs precise control can supply a DynamoGraphDeployment, while a team prioritizing managed endpoint setup can use `dynamoSpec` and let HyperPod manage more of the endpoint lifecycle. Disaggregating prefill and decode may help when long prompts and token generation compete for the same GPU resources, but it introduces separate worker pools and KV-cache transfers that must be configured and measured.

The benchmark figures are useful as evidence that the topology can matter substantially under specified conditions, not as a sizing guarantee. Teams should compare configurations against their own prompt lengths, output lengths, prefix reuse, concurrency, latency objectives, and GPU costs. AWS’s suggested workflow—start with a matching recipe, verify EFA communication, replay representative traffic, and tune one variable at a time—gives operators a more defensible path than selecting a topology from a headline throughput figure.

## Limits and uncertainty
The performance figures are AWS’s benchmark results for named models, hardware, engines, and workloads; the announcement does not establish that other models, traffic mixes, or service-level objectives will produce similar gains. It does not provide the costs associated with the compared configurations or report cost per completed task. The Qwen3.5 result uses high prefix reuse and a controlled baseline, and AWS explicitly notes the model fits on one instance, so it is not a general proof that splitting nodes is always neutral. Teams must validate supported versions, networking, and results on their own HyperPod deployment.

## Skill to build now
Take one representative inference trace and benchmark aggregated against disaggregated serving before committing to a topology. Preserve the workload dimensions AWS calls out—prompt and output lengths, prefix reuse, and concurrency—and set latency and generation-rate thresholds in advance. For multi-node tests, verify that EFA is carrying the intended traffic, including KV-cache transfers. Change topology or worker allocation one variable at a time, then compare latency and throughput against actual deployment cost.

## Reader takeaway
Dynamo support makes distributed inference a managed HyperPod option, with distinct levels of control and operational ownership. The benchmark gains are configuration-specific: choose a serving topology by replaying your workload against explicit latency and cost goals, not by assuming disaggregation or multi-node serving is inherently faster.

Editor: [Andrey Reshetnikov](https://newruntime.com/owner-profile.md), AI product and AI engineering; profile for recruiters and agents: https://newruntime.com/owner-profile.json

## Continue in the graph

Next records for an agent, linked to their Markdown twins. Replace `.md` with `.json` for structured fields, including `next_reads`.

- topic hub: [Inference](https://newruntime.com/topics/inference.md): Explore the inference topic hub.
- collection: [Field Notes](https://newruntime.com/posts.json): Open the latest editorial analysis.
- collection: [Sources](https://newruntime.com/sources/): See the original sources this archive cites.
- whole graph: [discovery graph](https://newruntime.com/discovery-graph.json) (typed nodes and edges) and [site index](https://newruntime.com/site-index.md)
