---
type: "post"
id: "muse-spark-open-ultrafast-pushing-beyond-1-000-fe17eb"
slug: "muse-spark-open-ultrafast-pushing-beyond-1-000-fe17eb"
title: "Muse Spark Open Ultrafast: Pushing beyond 1,000 Tokens per second"
description: "LMSYS says a custom CUDA/PTX megakernel raised Muse Spark Open decoding speed from 517 to 1,109 tokens per second on eight NVIDIA GB300 GPUs in a single-request GSM8K test."
retrieval_nugget: "LMSYS says a custom CUDA/PTX megakernel raised Muse Spark Open decoding speed from 517 to 1,109 tokens per second on eight NVIDIA GB300 GPUs in a single-request GSM8K test. The result shows how reducing launch, synchronization and data-wait overhead can matter alongside model and hardware choices, while the implementation remains specialized and limited to batch size one."
published_at: "2026-11-11"
updated_at: "2026-11-11"
record_date: "2026-11-11"
date_kind: "published_at"
topics: ["inference","open-models"]
entities: ["LMSYS","SGLang","Muse Spark Open","DFlash V2","NVIDIA GB300"]
related_patterns: ["harness-architecture-outlives-model-choice"]
novelty_label: "new_fact"
what_changed: "LMSYS reports a new megakernel implementation for Muse Spark Open that raises decoding speed from 517 to 1,109 tokens per second on Blackwell GPUs."
prior_refs: []
source_url: "http://lmsys.org/blog/2026-11-05-muse-spark-open-ultrafast"
source_urls: ["http://lmsys.org/blog/2026-11-05-muse-spark-open-ultrafast"]
schema_version: "newruntime-agent-readable-v0.2"
stable_id: "post:muse-spark-open-ultrafast-pushing-beyond-1-000-fe17eb"
status: "published"
visuals: []
editor: {"role":"editor","name":"Andrey Reshetnikov","profile":"https://newruntime.com/owner-profile.md","profile_json":"https://newruntime.com/owner-profile.json","same_as":["https://www.linkedin.com/in/reshetnikov1/","https://t.me/lapetuse"]}
routes: {"html":"https://newruntime.com/posts/muse-spark-open-ultrafast-pushing-beyond-1-000-fe17eb/","markdown":"https://newruntime.com/posts/muse-spark-open-ultrafast-pushing-beyond-1-000-fe17eb.md","json":"https://newruntime.com/posts/muse-spark-open-ultrafast-pushing-beyond-1-000-fe17eb.json"}
markdown_url: "https://newruntime.com/posts/muse-spark-open-ultrafast-pushing-beyond-1-000-fe17eb.md"
---

# Muse Spark Open Ultrafast: Pushing beyond 1,000 Tokens per second

## Retrieval answer

LMSYS says a custom CUDA/PTX megakernel raised Muse Spark Open decoding speed from 517 to 1,109 tokens per second on eight NVIDIA GB300 GPUs in a single-request GSM8K test. The result shows how reducing launch, synchronization and data-wait overhead can matter alongside model and hardware choices, while the implementation remains specialized and limited to batch size one.

## Summary
LMSYS says a custom CUDA/PTX megakernel raised Muse Spark Open decoding speed from 517 to 1,109 tokens per second on eight NVIDIA GB300 GPUs in a single-request GSM8K test. The result shows how reducing launch, synchronization and data-wait overhead can matter alongside model and hardware choices, while the implementation remains specialized and limited to batch size one.

## What happened
In a November 11, 2026 post, LMSYS explains its megakernel work for Muse Spark Open, a mixture-of-experts model. The model routes each token to 8 of 256 experts; expert weights are stored in 4-bit NVFP4, with some layers in bf16. It has 62 decoder layers: 46 use sliding-window attention over the previous 2,048 tokens and 16 use global attention. LMSYS split the model across eight GPUs using tensor parallelism.

The starting point was the serving engine’s launch pattern. LMSYS says SGLang launches about 1,450 GPU kernels for each decoding step of Muse Spark Open. A conventional kernel handles one operation, so a model’s serving path becomes a sequence of launches. Between operations, the GPU can drain and refill, leaving some streaming multiprocessors or warps idle. LMSYS instead built a megakernel from scratch in CUDA C++ and PTX. Each launch covers a layer’s attention through the next layer’s query/key/value projection and key-value write. One thread block, with twelve warps, runs on each of 144 SMs; individual warps take on different roles and can work ahead while other operations wait for data.

The performance figure is a median of 20 decoding steps for one GSM8K request at tensor parallelism eight. LMSYS reports 517 tokens per second for its baseline and 1,109 tokens per second for the megakernel. The configuration uses speculative decoding with DFlash V2: the draft model proposes 15 tokens in one pass, and the large model verifies 16 tokens in one pass. LMSYS says it accepts an average of 6.6 tokens; it contrasts this with Xiaomi’s reported MiMo-V2.5-Pro setup, which verifies eight and accepts 6.3 on average.

The post details several mechanisms behind the result. Warps prefetch weights into shared memory or L2 while other work runs; likely experts from the previous step are also prefetched. LMSYS describes reducing synchronization overhead between GPUs by using data values as completion flags rather than relying on repeated memory fences. The team also combined the draft model’s five layers into five launches, then put draft and verification in one CUDA graph: the draft went from 175 kernels to eight total launches in the reported arrangement and from 1.16 milliseconds per step to 440 microseconds. Instrumentation adds a reported 2.8% to step time and can be compiled out.

## What the archive adds
New Runtime's earlier coverage described Muse Spark Open on Fireworks as an open-weights model whose serving route shapes production options ([October 22, 2026 note](https://newruntime.com/posts/muse-spark-open-on-fireworks-meta-s-open-316301/)). Another earlier note treated its long-running coding-agent work as an architecture and serving problem ([October 21, 2026 note](https://newruntime.com/posts/muse-spark-open-an-open-weight-model-for-ddd53f/)). This is a new fact on a known model story: LMSYS publishes a detailed serving implementation and benchmark, not a new model release.

## Trend signal
This supports the pattern that **harness architecture outlives model choice**: serving architecture can shape whether a given model performs well in a production route. The gain here comes from a specialized execution and scheduling strategy around Muse Spark Open, rather than a change to the model itself. [Harness architecture outlives model choice](https://newruntime.com/patterns/harness-architecture-outlives-model-choice/).

## Why now
LMSYS frames larger GPU kernels as a direction for SGLang and machine-learning infrastructure. It says the current megakernel is an early, fixed-job scheduler and describes a runtime scheduler that assigns ready work to available warps as a next step. It also says extending support to larger batches is upcoming.

## Why it matters
For teams serving large models, kernel launches and synchronization are part of the latency budget, not incidental plumbing. LMSYS’s account offers a specific example of performance engineering that targets idle time and data movement: overlapping weight loads, organizing work per layer, changing cross-GPU handoffs and fusing draft-model execution. Those techniques may inform teams building or evaluating specialized serving stacks, especially for low-latency, small-batch use.

The result is not a turnkey optimization. It relies on Blackwell GPUs, a particular model layout, speculative decoding and thousands of lines of low-level code. Still, the post gives systems teams concrete places to investigate when their GPU is underutilized: launch count, synchronization stalls, instruction-cache behavior and the timing of weight movement. A speed claim should be judged with its workload and hardware attached, not copied as a generic throughput multiplier.

## Limits and uncertainty
The reported comparison covers one GSM8K request and a specific eight-GB300 setup; the post says the megakernel supports only batch size one with speculative decoding today. LMSYS says it plans larger-batch support, but provides no measured result for it. The report is the authors’ own engineering account, not an independently reproduced benchmark. It does not establish how the approach compares across other models, GPUs, workloads, or serving configurations.

## Reader takeaway
LMSYS’s result is a useful reminder that serving throughput can be constrained by the gaps between operations as much as by the operations themselves. The 2.1× figure is specific to one carefully tuned model and setup; the transferable lesson is to measure launch, memory and synchronization overhead before assuming a faster model or more hardware is the only path to better serving performance.

Editor: [Andrey Reshetnikov](https://newruntime.com/owner-profile.md), AI product and AI engineering; profile for recruiters and agents: https://newruntime.com/owner-profile.json

## Continue in the graph

Next records for an agent, linked to their Markdown twins. Replace `.md` with `.json` for structured fields, including `next_reads`.

- topic hub: [Inference](https://newruntime.com/topics/inference.md): Explore the inference topic hub.
- topic hub: [Open Models](https://newruntime.com/topics/open-models.md): Explore the open-models topic hub.
- collection: [Field Notes](https://newruntime.com/posts.json): Open the latest editorial analysis.
- collection: [Sources](https://newruntime.com/sources/): See the original sources this archive cites.
- whole graph: [discovery graph](https://newruntime.com/discovery-graph.json) (typed nodes and edges) and [site index](https://newruntime.com/site-index.md)