---
type: analysis
slug: research-looping-the-middle-half-of-an-moe-model-twice-saves-7-18-of-training-compute
title: "Looping the middle half of an MoE model twice saves 7–18% of training compute when budgets are matched, in one proprietary-stack study"
description: "SMELT, from Tsinghua, ByteDance Seed, M-A-P and TokenWave.AI, matches per-token FLOPs, parameters and KV cache between looped and plain MoE models. Looping still comes out ahead, but the paper measures no wall-clock cost and releases no code, so for now builders can only treat it as a design hypothesis."
published_at: 2026-10-23T04:00:00.000Z
topics: ["moe","architecture","design","research"]
source_urls: ["https://arxiv.org/html/2609.01343","https://arxiv.org/abs/2609.01343","https://ouro-llm.github.io/","https://arxiv.org/abs/2607.16051"]
author: "Andrey Reshetnikov"
author_profile: https://newruntime.com/owner-profile.md
---
Looped (or recurrent-depth) Transformers run some of their layers more than once, getting more effective depth without adding unique weights. Earlier headline results such as Huginn and ByteDance's Ouro compared looped models against same-size models without controlling FLOPs. That mixes up an architectural advantage with simply spending more compute per token. SMELT (arXiv 2609.01343, v1 on September 1, 2026, revised to v3 on September 11) comes from Tsinghua, ByteDance Seed, M-A-P and TokenWave.AI. It asks the cleaner question: does looping still help a Mixture-of-Experts model once per-token FLOPs, total non-embedding parameters and KV cache are all held roughly equal?

This matters to builders because MoE is now the default architecture for large models, and serving cost is driven mostly by FLOPs per token and KV-cache memory. If looping really improves loss and downstream quality at a fixed budget, it becomes a new design knob alongside width, depth and expert count. If the gain vanishes once compute is matched, it is just a way of spending more compute.

## What we checked

- **Does the cluster show a broader market or architecture transition that one release note can't capture?** partly confirmed. SMELT belongs to a 2025–2026 line of looped-model work: ByteDance's Ouro (open weights), Loop the Loopies! (looped MoE models at 20B-A2B and 6B-A0.6B), and the many prior papers SMELT cites. That suggests looping is turning into an active research direction for MoE pretraining. But every result so far is a lab paper, mostly without independent replication. Nothing shows production adoption, and vLLM even removed native Ouro support in July 2026. [2][1][4][3]
- **(Added) Are SMELT's gains real under compute matching, or do they come from extra FLOPs, as in earlier looped-model comparisons?** partly confirmed. The paper matches per-token FLOPs, non-embedding parameters and KV cache. The residual mismatch is usually around 3% and reaches at most 3.9% in FLOPs, and each run enters the fit at its measured FLOPs. Each configuration was trained only once, the design ablations ran only at the 200M-active scale, and wall-clock cost is not matched. [1]
- **(Added) Can builders reproduce or deploy SMELT directly?** refuted. The authors say their proprietary Transformer family's "complete architecture and training stack cannot be released." The training data is an internal corpus, and the abstract page links no official code. [2][1]

## What we found

- The recipe ('Sparse MoE Transformer, middle layers Loop Twice') loops the middle half of the layers twice, scaling each looped update by 1/r. It pays for the extra passes by narrowing the hidden dimension and buys back parameters with more experts. In the 200M-active example, the plain baseline has 12 layers, H=1280 and 192 experts. The SMELT model runs 18 layers, with H=1056 and 288 experts. Head size and GQA ratio are adjusted so the KV cache stays within about 4%, leaving residual mismatches of +2.9% FLOPs and +0.4% parameters. [1]
- The authors trained four active sizes (100M, 200M, 600M, 1.6B) across four sparsity levels, up to 54B total non-embedding parameters and about 215B tokens. They then fit a modified Chinchilla law for each architecture. SMELT's frontier exponent is 0.250 versus 0.237 for the baseline, which works out to 6.8–10.0% compute savings at 10^20 FLOPs and 14.7–18.0% at 10^21. The authors flag the 19.6–23.5% figure at 10^22 as an extrapolation beyond the fitted range. [2][1]
- Downstream gains exceed what validation loss alone would predict. SMELT wins 96 of 96 matched pairs on DCLM Completion, 83 of 96 on DCLM Core, and 29 of 30 above-chance MMLU pairs. By domain, the gain is largest on code (20.4% compute-equivalent at 10^21, S≈95%). The advantage also grows with the number of in-context examples. [1]
- Ablations produce three rules: loop the middle half rather than the full stack, use a higher effective depth-to-width ratio, and loop twice rather than three or four times. That runs against Ouro's whole-stack R=4 design. On mechanism, the second pass nearly removes the attention sink in a Dyck case study: BOS attention mass falls from 0.60 to 0.02 while demonstration-answer mass rises from 0.24 to 0.85. The authors call this descriptive, not causal. [1][3]
- The paper reports no measurements of latency, throughput or wall-clock time. The authors acknowledge that serial block re-execution and sparse routing 'might introduce hardware-efficiency gaps that require systems-level optimization to close.' Each configuration was trained once, the design ablations are limited to the 200M scale, and the architecture, data and training stack are proprietary and will not be released. [1]
- Related work points the same way, but none of it is independent replication. Loop the Loopies! (July 2026) reports looped MoE models at 20B-A2B and 6B-A0.6B beating vanilla baselines, including a 30B-A3B, at matched compute, though its abstract gives no numbers. Ouro released open looped models at 1.4B and 2.6B trained on 7.7T tokens, yet vLLM dropped native support in July 2026. That says something about how much friction looped architectures meet in serving stacks. [4][3]

## What it means

For teams pretraining MoE models, SMELT is the most carefully budget-matched evidence yet that reusing depth is a real architectural gain and not just a hidden compute increase. It also comes with a concrete starting recipe: loop the middle 50% of layers twice, narrow the width, add experts to restore parameter count, scale looped residuals by 1/2, and retune GQA to hold the KV cache. The predicted savings of about 7–18% of training FLOPs are big enough to justify a small-scale ablation in your own stack, especially if code or long-context in-context learning is a target. Treat the numbers as a hypothesis to test, not a constant to plan budgets around. They come from one lab's proprietary stack and internal data, with single training runs, and the largest figure is extrapolated.

On inference, FLOP and KV-cache parity is not cost parity. Running 18 executed layers instead of 12 at a narrower width means more serial steps per token. That is likely to hurt decode latency and kernel efficiency, and the paper doesn't measure it. Before adopting the recipe, benchmark tokens per second and time-to-first-token on your own hardware and serving engine, and check that the engine supports weight-shared repeated blocks. Ouro losing vLLM support is a warning about maintenance burden. Anyone writing about this should describe it as a growing research direction with one strong compute-matched data point. It is not yet an industry shift.

## Still open

- What does SMELT cost in wall-clock decode latency and throughput compared with the baseline on standard GPU serving stacks?
- Does the gain hold up under independent replication on open data and open architectures, with multiple training seeds?
- Does the advantage persist or grow past 10^21 FLOPs, where the 19.6–23.5% figure is only extrapolated?
- Is the attention-sink reduction causally responsible for the in-context learning gains?

## Sources

1. [SMELT full text (arXiv HTML)](https://arxiv.org/html/2609.01343) (primary)
2. [SMELT: Scaling Laws for Compute-Matched MoE Looped Transformers (arXiv abstract)](https://arxiv.org/abs/2609.01343) (primary)
3. [Ouro: Scaling Latent Reasoning via Looped Language Models](https://ouro-llm.github.io/)
4. [Loop the Loopies! (arXiv)](https://arxiv.org/abs/2607.16051)

Author: [Andrey Reshetnikov](https://newruntime.com/owner-profile.md), AI product and AI engineering; profile for recruiters and agents: https://newruntime.com/owner-profile.json

## Continue in the graph

Next records for an agent, linked to their Markdown twins. Replace `.md` with `.json` for structured fields, including `next_reads`.

- topic hub: [Moe](https://newruntime.com/topics/moe.md): Explore the moe topic hub.
- topic hub: [Architecture](https://newruntime.com/topics/architecture.md): Explore the architecture topic hub.
- topic hub: [Design](https://newruntime.com/topics/design.md): Explore the design topic hub.
- collection: [Analysis](https://newruntime.com/analysis.json): Open the latest authorial analysis.
- collection: [Source Ledger](https://newruntime.com/source-ledger.json): Inspect the public source evidence graph.
- whole graph: [discovery graph](https://newruntime.com/discovery-graph.json) (typed nodes and edges) and [site index](https://newruntime.com/site-index.md)