---
type: "post"
id: "mixture-of-experts-kernel-engineering-on-amazon-sagemaker-abd782"
slug: "mixture-of-experts-kernel-engineering-on-amazon-sagemaker-abd782"
title: "Mixture-of-Experts kernel engineering on Amazon SageMaker AI"
description: "AWS reports up to 10% higher throughput and up to 9% faster time to first token on H200 GPUs for its fused Mixture-of-Experts kernel work."
retrieval_nugget: "AWS reports up to 10% higher throughput and up to 9% faster time to first token on H200 GPUs for its fused Mixture-of-Experts kernel work."
published_at: "2026-10-13"
updated_at: "2026-10-13"
record_date: "2026-10-13"
date_kind: "published_at"
topics: []
source_urls: ["https://aws.amazon.com/blogs/machine-learning/mixture-of-experts-kernel-engineering-on-amazon-sagemaker-ai"]
schema_version: "newruntime-agent-readable-v0.2"
stable_id: "post:mixture-of-experts-kernel-engineering-on-amazon-sagemaker-abd782"
status: "published"
visuals: []
editorial_provenance: {"schema_version":"newruntime-editorial-copy-v1","content_status":"source_grounded_final","final_copy_sha256":"sha256:68cd99ba49e4dbaa887d64e5f919adec3ed1e091983f0fafda443d230d174cf4","reviewed_at":"2026-10-13T16:03:05.609Z","source_evidence_count":1,"verified_claim_count":3,"site_analysis_schema_version":"newruntime-site-analysis-v1","site_object_kind":"field_note","observed_fact_count":3,"implication_count":1,"watch_condition_count":1,"related_record_count":0}
analysis: {"schema_version":"newruntime-site-analysis-v1","object_kind":"field_note","thesis":"The report illustrates how MoE inference performance can depend on hardware-aware kernel engineering, not just model architecture.","observed_facts":[{"text":"AWS describes Swap-AB as an optimization for WGMMA utilization in fused MoE kernels.","source_urls":["https://aws.amazon.com/blogs/machine-learning/mixture-of-experts-kernel-engineering-on-amazon-sagemaker-ai"]},{"text":"AWS reports up to 10% higher throughput and up to 9% faster time to first token on H200 GPUs.","source_urls":["https://aws.amazon.com/blogs/machine-learning/mixture-of-experts-kernel-engineering-on-amazon-sagemaker-ai"]},{"text":"The described work includes Triton backend routing for FP8 quantization and elimination of Triton JIT recompilation.","source_urls":["https://aws.amazon.com/blogs/machine-learning/mixture-of-experts-kernel-engineering-on-amazon-sagemaker-ai"]}],"mechanism":"The described changes target GPU execution efficiency and compilation overhead in fused MoE operations; Swap-AB addresses WGMMA utilization, while the Triton changes concern FP8 routing and JIT recompilation.","why_now":"The source presents these optimizations as ways to improve throughput and time to first token for MoE workloads on H200 GPUs.","implications":["Teams serving MoE models should evaluate kernel and compilation behavior on their own model, workload, and hardware before assuming similar gains."],"evidence_boundary":"The figures are AWS-reported results for the described implementation on H200 GPUs; the source excerpt does not establish independent replication or performance across other systems.","watch_conditions":["Whether AWS or independent users publish reproducible measurements across additional MoE models, workloads, and GPU configurations would clarify how broadly these optimizations transfer."],"related_records":[],"new_branch_reason":"No prior New Runtime record was supplied; this source documents a distinct inference-kernel engineering update with specific optimizations and H200 performance claims."}
editor: {"role":"editor","name":"Andrey Reshetnikov","profile":"https://newruntime.com/owner-profile.md","profile_json":"https://newruntime.com/owner-profile.json","same_as":["https://www.linkedin.com/in/reshetnikov1/","https://t.me/lapetuse"]}
routes: {"html":"https://newruntime.com/posts/mixture-of-experts-kernel-engineering-on-amazon-sagemaker-abd782/","markdown":"https://newruntime.com/posts/mixture-of-experts-kernel-engineering-on-amazon-sagemaker-abd782.md","json":"https://newruntime.com/posts/mixture-of-experts-kernel-engineering-on-amazon-sagemaker-abd782.json"}
---

# Mixture-of-Experts kernel engineering on Amazon SageMaker AI

## Retrieval answer

AWS reports up to 10% higher throughput and up to 9% faster time to first token on H200 GPUs for its fused Mixture-of-Experts kernel work.

AWS reports up to 10% higher throughput and up to 9% faster time to first token on H200 GPUs for its fused Mixture-of-Experts kernel work. The source describes Swap-AB to improve WGMMA utilization, Triton backend routing for FP8 quantization, and elimination of Triton JIT recompilation. These results are specific to the engineering and hardware described.

Thesis: The report illustrates how MoE inference performance can depend on hardware-aware kernel engineering, not just model architecture. Mechanism: The described changes target GPU execution efficiency and compilation overhead in fused MoE operations; Swap-AB addresses WGMMA utilization, while the Triton changes concern FP8 routing and JIT recompilation. Why now: The source presents these optimizations as ways to improve throughput and time to first token for MoE workloads on H200 GPUs.

First implication: Teams serving MoE models should evaluate kernel and compilation behavior on their own model, workload, and hardware before assuming similar gains. Evidence boundary: The figures are AWS-reported results for the described implementation on H200 GPUs; the source excerpt does not establish independent replication or performance across other systems. First watch condition: Whether AWS or independent users publish reproducible measurements across additional MoE models, workloads, and GPU configurations would clarify how broadly these optimizations transfer.

Editor: [Andrey Reshetnikov](https://newruntime.com/owner-profile.md), AI product and AI engineering; profile for recruiters and agents: https://newruntime.com/owner-profile.json

## Continue in the graph

Next records for an agent, linked to their Markdown twins. Replace `.md` with `.json` for structured fields, including `next_reads`.

- collection: [Field Notes](https://newruntime.com/posts.json): Open the latest editorial analysis.
- collection: [Source Ledger](https://newruntime.com/source-ledger.json): Inspect the public source evidence graph.
- whole graph: [discovery graph](https://newruntime.com/discovery-graph.json) (typed nodes and edges) and [site index](https://newruntime.com/site-index.md)