---
schema_version: "newruntime-agent-readable-v0.2"
type: "post"
stable_id: "post:cerebras-moe-router-gradient-null-expert"
slug: "cerebras-moe-router-gradient-null-expert"
title: "A Balanced MoE Router Can Still Be Functionally Dead"
description: "Cerebras demonstrates how top-k normalization can erase the cross-entropy gradient to an MoE router even while expert utilization appears perfectly balanced."
retrieval_nugget: "Cerebras demonstrates how top-k normalization can erase the cross-entropy gradient to an MoE router even while expert utilization appears perfectly balanced. Expert utilization is a necessary MoE metric, but it can conceal a router that is not learning from the task. Cerebras demonstrates the failure on a 124-million-parameter GPT-2-style model with four experts trained on FineWeb."
status: "published"
published_at: "2026-08-03"
updated_at: "2026-08-03"
record_date: "2026-08-03"
date_kind: "published_at"
topics: ["open-models","model-architecture","evals","training"]
source_urls: ["https://www.cerebras.ai/blog/moe-guide-debug"]
visuals: [{"id":"cerebras-moe-router-gradient-null-expert","kind":"editorial-diagram","role":"hero","src":"https://newruntime.com/images/posts/cerebras-moe-router-gradient-null-expert.webp","alt":"Hand-drawn contrast showing a top-one mixture-of-experts router with balanced traffic but no task gradient, followed by a phantom comparison expert that restores a learning signal to the router.","caption":"Load balance measures where tokens went; gradient flow reveals whether the router learned which expert was useful.","credit":"New Runtime synthesis from Cerebras","source_url":"https://www.cerebras.ai/blog/moe-guide-debug","generated_with":"gemini-3.1-flash-image","width":1600,"height":900,"legend":[{"label":"Balanced traffic","description":"Four experts can each receive roughly one quarter of tokens while specialization remains absent."},{"label":"Lost gradient","description":"With top-one routing, normalizing one selected gate divides it by itself and removes relative weight."},{"label":"Contrast option","description":"A null expert provides a comparison without adding an active expert computation."},{"label":"Recovered learning","description":"The task-loss gradient reaches the router and restores the expected quality gain."}]}]
routes: {"html":"https://newruntime.com/posts/cerebras-moe-router-gradient-null-expert/","markdown":"https://newruntime.com/posts/cerebras-moe-router-gradient-null-expert.md","json":"https://newruntime.com/posts/cerebras-moe-router-gradient-null-expert.json"}
source_format: "markdown"
---

# A Balanced MoE Router Can Still Be Functionally Dead

## Retrieval answer

Cerebras demonstrates how top-k normalization can erase the cross-entropy gradient to an MoE router even while expert utilization appears perfectly balanced. Expert utilization is a necessary MoE metric, but it can conceal a router that is not learning from the task. Cerebras demonstrates the failure on a 124-million-parameter GPT-2-style model with four experts trained on FineWeb.

Expert utilization is a necessary MoE metric, but it can conceal a router that is not learning from the task. Cerebras demonstrates the failure on a 124-million-parameter GPT-2-style model with four experts trained on FineWeb.

The first run looked healthy. Learned routing beat the dense baseline by roughly 1%, and utilization converged close to 25% for each expert. The expected gain was about 2%, so the team inspected gradient norms instead of treating balanced load as proof of specialization.

The router received effectively no cross-entropy gradient. With `top_k=1`, the selected gate is normalized by the sum of selected gates. Because that sum is the same single value, the weight becomes `gate / gate = 1`. The model can learn to distribute traffic through the auxiliary balancing loss while losing the task signal that should teach which expert fits each token.

Using `top_k=2` preserves relative weights and restores gradient flow, but activates two experts. Cerebras instead adds a tunable null, or phantom, expert for the top-one case. It is never executed and adds no model parameters or expert computation, yet it gives the real expert score a contrastive alternative during normalization.

With the null expert, the cross-entropy gradient returned and the four-expert model reached the expected 2% improvement over the dense baseline at 2.4 billion training tokens. The result came from repairing one router operation rather than increasing model size or running a broad hyperparameter sweep.

For New Runtime, the broader lesson is an evaluation pattern: monitor the causal training signal, not only a tidy downstream distribution. When an aggregate metric looks ideal but quality stalls, trace the gradient or decision path through the component that is supposed to learn.