{"type":"post","id":"mixture-of-experts-kernel-engineering-on-amazon-sagemaker-abd782","slug":"mixture-of-experts-kernel-engineering-on-amazon-sagemaker-abd782","title":"Mixture-of-Experts kernel engineering on Amazon SageMaker AI","description":"AWS reports up to 10% higher throughput and up to 9% faster time to first token on H200 GPUs for its fused Mixture-of-Experts kernel work.","retrieval_nugget":"AWS reports up to 10% higher throughput and up to 9% faster time to first token on H200 GPUs for its fused Mixture-of-Experts kernel work.","published_at":"2026-10-13","updated_at":"2026-10-13","record_date":"2026-10-13","date_kind":"published_at","topics":[],"source_urls":["https://aws.amazon.com/blogs/machine-learning/mixture-of-experts-kernel-engineering-on-amazon-sagemaker-ai"],"schema_version":"newruntime-agent-readable-v0.2","stable_id":"post:mixture-of-experts-kernel-engineering-on-amazon-sagemaker-abd782","status":"published","visuals":[],"editorial_provenance":{"schema_version":"newruntime-editorial-copy-v1","content_status":"source_grounded_final","final_copy_sha256":"sha256:68cd99ba49e4dbaa887d64e5f919adec3ed1e091983f0fafda443d230d174cf4","reviewed_at":"2026-10-13T16:03:05.609Z","source_evidence_count":1,"verified_claim_count":3,"site_analysis_schema_version":"newruntime-site-analysis-v1","site_object_kind":"field_note","observed_fact_count":3,"implication_count":1,"watch_condition_count":1,"related_record_count":0},"analysis":{"schema_version":"newruntime-site-analysis-v1","object_kind":"field_note","thesis":"The report illustrates how MoE inference performance can depend on hardware-aware kernel engineering, not just model architecture.","observed_facts":[{"text":"AWS describes Swap-AB as an optimization for WGMMA utilization in fused MoE kernels.","source_urls":["https://aws.amazon.com/blogs/machine-learning/mixture-of-experts-kernel-engineering-on-amazon-sagemaker-ai"]},{"text":"AWS reports up to 10% higher throughput and up to 9% faster time to first token on H200 GPUs.","source_urls":["https://aws.amazon.com/blogs/machine-learning/mixture-of-experts-kernel-engineering-on-amazon-sagemaker-ai"]},{"text":"The described work includes Triton backend routing for FP8 quantization and elimination of Triton JIT recompilation.","source_urls":["https://aws.amazon.com/blogs/machine-learning/mixture-of-experts-kernel-engineering-on-amazon-sagemaker-ai"]}],"mechanism":"The described changes target GPU execution efficiency and compilation overhead in fused MoE operations; Swap-AB addresses WGMMA utilization, while the Triton changes concern FP8 routing and JIT recompilation.","why_now":"The source presents these optimizations as ways to improve throughput and time to first token for MoE workloads on H200 GPUs.","implications":["Teams serving MoE models should evaluate kernel and compilation behavior on their own model, workload, and hardware before assuming similar gains."],"evidence_boundary":"The figures are AWS-reported results for the described implementation on H200 GPUs; the source excerpt does not establish independent replication or performance across other systems.","watch_conditions":["Whether AWS or independent users publish reproducible measurements across additional MoE models, workloads, and GPU configurations would clarify how broadly these optimizations transfer."],"related_records":[],"new_branch_reason":"No prior New Runtime record was supplied; this source documents a distinct inference-kernel engineering update with specific optimizations and H200 performance claims."},"editor":{"role":"editor","name":"Andrey Reshetnikov","profile":"https://newruntime.com/owner-profile.md","profile_json":"https://newruntime.com/owner-profile.json","same_as":["https://www.linkedin.com/in/reshetnikov1/","https://t.me/lapetuse"]},"routes":{"html":"https://newruntime.com/posts/mixture-of-experts-kernel-engineering-on-amazon-sagemaker-abd782/","markdown":"https://newruntime.com/posts/mixture-of-experts-kernel-engineering-on-amazon-sagemaker-abd782.md","json":"https://newruntime.com/posts/mixture-of-experts-kernel-engineering-on-amazon-sagemaker-abd782.json"},"next_reads":[{"type":"collection","path":"/posts/","url":"https://newruntime.com/posts/","title":"Field Notes","reason":"Open the latest editorial analysis.","media_type":"text/html","json_url":"https://newruntime.com/posts.json"},{"type":"collection","path":"/source-ledger.json","url":"https://newruntime.com/source-ledger.json","title":"Source Ledger","reason":"Inspect the public source evidence graph.","media_type":"application/json"}]}