---
type: "post"
id: "sagemaker-inference-now-supports-multi-instance-gpu-to-0c0db5"
slug: "sagemaker-inference-now-supports-multi-instance-gpu-to-0c0db5"
title: "SageMaker Inference now supports Multi-Instance GPU to maximize GPU utilization for model endpoints"
description: "AWS says SageMaker Inference now supports NVIDIA Multi-Instance GPU partitioning for managed endpoints that use inference components."
retrieval_nugget: "AWS says SageMaker Inference now supports NVIDIA Multi-Instance GPU partitioning for managed endpoints that use inference components. Teams can place separate models on isolated slices of one accelerator, and AWS reports a workload-specific H100-versus-A100 test where a slice improved throughput and cost at sufficiently high concurrency."
published_at: "2026-11-07"
updated_at: "2026-11-07"
record_date: "2026-11-07"
date_kind: "published_at"
topics: ["inference"]
entities: ["Amazon Web Services","Amazon SageMaker Inference","NVIDIA Multi-Instance GPU","NVIDIA A100","NVIDIA H100","Salesforce"]
related_patterns: ["agent-economics-moves-to-completed-work"]
novelty_label: "new_fact"
what_changed: "SageMaker Inference adds NVIDIA Multi-Instance GPU support, enabling model endpoints to share A100 and H100 GPUs for improved utilization."
prior_refs: []
source_url: "https://aws.amazon.com/blogs/machine-learning/sagemaker-inference-now-supports-multi-instance-gpu-to-maximize-gpu-utilization-for-model-endpoints"
source_urls: ["https://aws.amazon.com/blogs/machine-learning/sagemaker-inference-now-supports-multi-instance-gpu-to-maximize-gpu-utilization-for-model-endpoints"]
schema_version: "newruntime-agent-readable-v0.2"
stable_id: "post:sagemaker-inference-now-supports-multi-instance-gpu-to-0c0db5"
status: "published"
visuals: []
editor: {"role":"editor","name":"Andrey Reshetnikov","profile":"https://newruntime.com/owner-profile.md","profile_json":"https://newruntime.com/owner-profile.json","same_as":["https://www.linkedin.com/in/reshetnikov1/","https://t.me/lapetuse"]}
routes: {"html":"https://newruntime.com/posts/sagemaker-inference-now-supports-multi-instance-gpu-to-0c0db5/","markdown":"https://newruntime.com/posts/sagemaker-inference-now-supports-multi-instance-gpu-to-0c0db5.md","json":"https://newruntime.com/posts/sagemaker-inference-now-supports-multi-instance-gpu-to-0c0db5.json"}
markdown_url: "https://newruntime.com/posts/sagemaker-inference-now-supports-multi-instance-gpu-to-0c0db5.md"
---

# SageMaker Inference now supports Multi-Instance GPU to maximize GPU utilization for model endpoints

## Retrieval answer

AWS says SageMaker Inference now supports NVIDIA Multi-Instance GPU partitioning for managed endpoints that use inference components. Teams can place separate models on isolated slices of one accelerator, and AWS reports a workload-specific H100-versus-A100 test where a slice improved throughput and cost at sufficiently high concurrency.

## Summary
AWS says SageMaker Inference now supports NVIDIA Multi-Instance GPU partitioning for managed endpoints that use inference components. Teams can place separate models on isolated slices of one accelerator, and AWS reports a workload-specific H100-versus-A100 test where a slice improved throughput and cost at sufficiently high concurrency.

## What happened
AWS’s announcement describes GPU partitioning as a way to divide one physical accelerator into independently allocated MIG partitions. Each partition receives fixed compute and memory resources, with hardware-enforced isolation. The intended change is both operational and economic: host more models than there are physical GPUs, or select a fraction of a newer accelerator rather than committing a workload to a whole device.

The feature is built on SageMaker inference components, which let operators deploy multiple models to one endpoint and assign compute resources to each. AWS says partitioning is unavailable for single-model endpoints, serverless endpoints, and heterogeneous instance pools. Two optional fields extend existing API calls: `AcceleratorPartitionLayouts` on a production variant defines how each accelerator is divided, while `AcceleratorPartitionType` on a component selects the partition type for the model. No new API operations, resource types, or IAM actions are required; omitting both fields leaves an endpoint behaving as before.

AWS lists support for P4d instances with NVIDIA A100 40 GB GPUs, P5 with H100 80 GB, G7 with RTX PRO 4500 Blackwell 32 GB, and G7e with RTX PRO 6000 Blackwell Server Edition 96 GB. A100 and H100 support up to seven partitions per GPU, G7 up to two, and G7e up to four. Profiles combine compute and memory allocations, and valid layouts must fit the accelerator’s available resources. The layout is chosen when the endpoint is created; changing it requires endpoint recreation. Models requiring multiple GPUs still need whole devices.

For a performance comparison, AWS tested one H100 `mig-3g.40gb` partition against one unpartitioned A100-40GB, each serving Llama-3.1-8B-Instruct through a production SageMaker real-time endpoint. Using genai-bench, the team tested five input/output traffic shapes and concurrency from 2 to 512. AWS reports 53–89% more output tokens per second at each scenario’s peak cost benefit, 8.9–42.1% lower inter-token latency, and 5.6–47.5% lower time to first token. Above the cost crossover, the partition was 18–34% cheaper per million output tokens. The crossover was concurrency 64 for 100-token prompts and 4 for 8,000-token prompts; below it, the full A100 was cheaper. AWS also reports that at the highest concurrency on 6,000- and 8,000-token workloads, the A100 missed a third to half of requests against a serving deadline while the H100 partition missed none.

## What the archive adds
New Runtime’s earlier coverage on October 22 described SageMaker AI model-customization sessions that do not require GPU management ([earlier coverage](https://newruntime.com/posts/announcing-new-ways-to-customize-models-with-sagemaker-030e9c/)). That is related platform context, not the same capability: this is a new inference deployment feature.

## Trend signal
This supports the idea that inference economics depend on workload-matched execution choices, though AWS’s test is narrower than a general cost-per-completed-task measure. It is adjacent to [agent economics moving to completed work](https://newruntime.com/patterns/agent-economics-moves-to-completed-work), but does not itself measure agent task completion.

## Why now
AWS frames the release around a practical gap: customers needing GPU partitioning had used self-managed Kubernetes and taken on driver management, device plugins, scheduling, and fleet consistency. The managed endpoint option removes that platform work while retaining MIG partitioning.

## Why it matters
For teams serving portfolios of small or mid-sized models, GPU capacity often goes unused because the allocation unit is a whole accelerator. SageMaker’s partitions offer another packing and sizing option, with each model retaining a fixed share of compute and memory rather than competing for an undivided GPU. That can matter for per-tenant models, fine-tuned variants, classifiers, rerankers, or stages of a pipeline.

The measurements are useful as a hypothesis to test, not as a procurement conclusion. A slice of a newer GPU may beat a whole older GPU for long prompts and high concurrency, while the older full GPU can remain cheaper at low load. Teams should compare the available partition profiles against their own prompt lengths, output lengths, concurrency, memory needs, and serving deadlines. AWS also cautions that MIG isolates GPU resources, not host CPU, memory, networking, storage, or endpoint routing; it is not a complete tenant boundary.

## Limits and uncertainty
The reported benchmark is AWS’s own test of one H100 partition against one full A100, using a single model and five traffic shapes. It does not establish results for other models, containers, regions, traffic mixes, or production portfolios. AWS presents the cost comparison using on-demand instance prices current at its analysis time and derives a partition unit cost by dividing whole-instance cost across GPU units. The Salesforce quotation says it is preparing to validate the solution with representative production workloads, so independent production evidence is not established by the announcement. Hardware partitioning also does not isolate shared host resources.

## Skill to build now
Build a small workload matrix before choosing a partition: record prompt length, output length, peak and typical concurrency, model-weight size, KV-cache needs, and latency target. Then benchmark the candidate profile against a whole-GPU baseline using the same model, container, and client. Include low-load cases as well as peak concurrency; AWS’s own comparison shows that the cheaper choice can reverse at the crossover.

## Reader takeaway
MIG partitioning makes SageMaker endpoints more granular, not automatically cheaper. The right question is whether a specific partition fits your model’s compute and memory profile and beats a whole-GPU option under your real concurrency and prompt mix.

Editor: [Andrey Reshetnikov](https://newruntime.com/owner-profile.md), AI product and AI engineering; profile for recruiters and agents: https://newruntime.com/owner-profile.json

## Continue in the graph

Next records for an agent, linked to their Markdown twins. Replace `.md` with `.json` for structured fields, including `next_reads`.

- topic hub: [Inference](https://newruntime.com/topics/inference.md): Explore the inference topic hub.
- collection: [Field Notes](https://newruntime.com/posts.json): Open the latest editorial analysis.
- collection: [Sources](https://newruntime.com/sources/): See the original sources this archive cites.
- whole graph: [discovery graph](https://newruntime.com/discovery-graph.json) (typed nodes and edges) and [site index](https://newruntime.com/site-index.md)