Field noteNov 17, 20266 min read1 source

MegaKernel part 1: vLLM and llama.cpp leave over half of memory roofline unused on SmolLM2-135M at batch 1

In one minute

A public MegaKernel repository benchmarks batch-one decode on an RTX 4060 Laptop GPU and traces a context-scaling problem to attention dispatch. Its split-K Triton attention path reduces the measured attention time at 2,048 tokens from 2.99 ms to 0.50 ms. For inference builders, the result is a useful reminder that kernel launches and attention behavior can dominate small-model decoding.

What changed: MegaKernel reports that vLLM and llama.cpp leave more than half of the memory roofline unused when decoding SmolLM2-135M at batch 1.

What happened

The MegaKernel repository describes Part 1 of an attempt to understand and build megakernels. It includes a measurement harness, baselines, a context-scaling fix, and figures; the persistent megakernel and Triton fusion are not implemented. The headline tests use batch-one greedy decode and report median latency over 50 steps. Tests ran on an RTX 4060 Laptop GPU with 8 GB of memory, under an overclock disclosed in the repository.

For SmolLM2-135M, the repository reports 0.269 GB of weights and a 1.02 ms weight-read floor. Across contexts of 128, 512, and 2,048 tokens, vLLM with CUDA graphs measured 2.30, 2.18, and 2.41 ms per token; llama.cpp F16 with flash attention measured 2.32, 2.37, and 2.54 ms. These correspond to roughly 42–47% and 40–44% of the 263.5 GB/s sustained-read roofline used in the repository. The CUDA-graph path built for MegaKernel measured 3.39, 3.43, and 3.72 ms. The repository concludes that the two established runtimes are close on this model, while its own path remains slower.

The comparison changes with model size. On Qwen2.5-1.5B, whose weights are reported as 3.087 GB, vLLM measured 12.49, 12.34, and 12.54 ms per token at the same context lengths, about 93–95% of the stated roofline. The repository did not run llama.cpp on this model. It therefore treats the larger model as a contrast already near the measured memory limit, not as evidence that all decode workloads have the same optimization headroom.

The main implemented change addresses attention in the small-model path. The first CUDA-graph version took 4.80, 4.30, and 6.69 ms at 128, 512, and 2,048 tokens. Profiling attributed the context growth to PyTorch SDPA selecting a memory-efficient CUTLASS kernel for a static cache with a four-dimensional causal mask. For query length one, MegaKernel patches LlamaAttention to call a split-K Triton kernel: it uses 16 splits per head, online softmax, then a reduction. Reading cache length from device-side cumulative length keeps shapes fixed and the kernel CUDA-graph-safe. The fix cut attention at 2,048 tokens from 2.99 to 0.50 ms and achieved exact token parity against eager in the reported test.

The repository's 128-token breakdown for the fixed graph is 3.36 ms per token: 1.59 ms for weight streaming, 1.33 ms for elementwise and other work, 0.26 ms of host gap, 0.17 ms for attention, and 0.011 ms for sampling. The stated next target is 1.8–2.1 ms per token through fusing elementwise work into GEMM epilogues and pipelining weight loads. That target is not a result: persistent fusion is still unimplemented. The repository includes reproduction scripts and a parity test, and says the exact environment uses WSL2, CUDA, and pinned software versions.

What the archive adds

New Runtime's earlier coverage includes a specialized vLLM decode path for Kimi K3 (2026-10-22), where the note emphasized designing validation alongside inference optimization. This is a new measurement thread: MegaKernel isolates launch and attention costs on a laptop GPU rather than describing a deployed model-specific path.

Trend signal

This supports agent economics moves to completed work only weakly: the repository makes a measured task-level latency profile more useful than a generic claim about kernel efficiency, but it does not report cost per verified completed task. More directly, it reinforces the practical need to measure the full execution path before drawing conclusions from memory bandwidth alone.

Why it matters

Small-model local inference can be limited by overhead that a bandwidth-only estimate misses. This benchmark finds that vLLM and llama.cpp leave substantial measured roofline headroom on SmolLM2-135M, while vLLM is already near the measured limit on Qwen2.5-1.5B. The difference means teams should avoid assuming that a kernel optimization transfers uniformly across model sizes or runtimes.

The repository also makes the optimization work inspectable: it identifies a specific attention path, reports the before-and-after latency across context lengths, and includes a token-parity gate. That is a more useful engineering artifact than a bare speedup claim. Builders can use the breakdown as a profiling checklist, then validate any replacement kernel against both latency and output behavior. The prospective megakernel target is not yet evidence of achieved performance, so it should be treated as a hypothesis to test, not a procurement or deployment promise.

What happens next

  • A stock-clock rerun will test whether the reported runtime ordering survives without the disclosed overclock. (<3 months, 65%). Watch for: updated roofline and latency tables.
  • Persistent fusion may be added as the repository's next performance experiment. (<6 months, 45%). Watch for: a reproducible fused-kernel benchmark with parity results.

Limits and uncertainty

The results are specific to one RTX 4060 Laptop GPU, one software stack, batch one, greedy decode, and an overclock of +1,000 MHz memory and +158 MHz core. The repository says a stock-clock rerun is pending and expects roughly 230–240 GB/s sustained bandwidth there, but that is an estimate, not a measurement. Absolute roofline percentages depend on the 263.5 GB/s sustained-read figure; the repository advises adjusting comparisons against the theoretical peak.

The comparisons are not fully symmetric: llama.cpp was not run on Qwen2.5-1.5B, and its F16 GGUF baseline is compared with bf16 PyTorch. The repository reports no token-parity check for llama.cpp. Its persistent megakernel, fusion, and energy-per-token measurements are unfinished; power telemetry is partial under WSL. These results are author measurements and are not independently reproduced here.

Skill to build now

If you are optimizing single-token local decode, reproduce a latency sweep at short and long contexts before changing kernels. Keep a parity check against a simple eager path, then use profiler categories to distinguish attention growth from weight streaming, elementwise work, and host gaps. This repository's 128-to-2,048-token comparison shows why both the context sweep and output check matter.

Reader takeaway

The useful result is not that a new megakernel has beaten established runtimes; it has not been implemented. It is that a carefully measured attention fix can flatten context-related latency in a small-model decode path, while a larger model may already sit near the measured bandwidth roofline. Profile the bottleneck your workload actually has.

Evidence / sources

  1. 1github.com/crankyastrologer/megakernel
All Field Notes

Next reads

Read next

Open the graph
  1. 01topicMemoryExplore the memory topic hub.
  2. 02archiveField NotesOpen the latest editorial analysis.
  3. 03sourcesSourcesSee the original sources this archive cites.
Who read this page?Machine requests, hidden until opened

Loading the privacy-safe route aggregate...

Open the JSON contract