---
type: analysis
slug: cerebras-s-5-shows-that-where-each-inference-stage-runs-now-sets-capacity-but
title: "Cerebras's 5× shows that where each inference stage runs now sets capacity, but it doesn't show which stage will cost more"
description: "Cerebras moved prompt processing to AMD GPUs and says it now gets five times more throughput per megawatt. The ratio is believable, but it rests on one endpoint, one day and a power boundary Cerebras hasn't disclosed. Until the missing numbers are published, treat it as a reason to measure your own traffic, not as a price signal."
published_at: 2026-11-18T17:00:00.000Z
topics: ["trends"]
source_urls: ["https://www.cerebras.ai/blog/cerebras-launches-disaggregated-inference","https://aws.amazon.com/blogs/machine-learning/deploy-frontier-ai-models-on-aws-trainium-with-sglang-neuron","https://aws.amazon.com/blogs/machine-learning/secure-oauth-patterns-for-snowflake-cortex-ai-through-amazon-bedrock-agentcore","https://aws.amazon.com/blogs/machine-learning/reducing-token-costs-and-latency-in-amazon-bedrock-agentcore-json-workloads"]
author: "Andrey Reshetnikov"
author_profile: https://newruntime.com/owner-profile.md
---
# Cerebras's 5× shows that where each inference stage runs now sets capacity, but it doesn't show which stage will cost more

Cerebras moved prompt processing to AMD GPUs and says it now gets five times more throughput per megawatt. The ratio is believable, but it rests on one endpoint, one day and a power boundary Cerebras hasn't disclosed. Until the missing numbers are published, treat it as a reason to measure your own traffic, not as a price signal.

## Summary

On November 17, 2026, Cerebras put disaggregated inference into production. AMD Instinct GPUs handle prefill (prompt processing) and CS-3 systems handle decode (token generation). Cerebras says a shared GPT-OSS-120B endpoint got 5× the throughput per megawatt of a CS-3-only setup with no loss in generation speed. Our read: the result shows that placing each stage on the right hardware is now a real capacity lever. It does not show how big the gain is outside this one endpoint, or which stage will command a price premium.

## What happened

Cerebras's announcement describes a split setup on Cerebras Inference Cloud. AMD Instinct GPUs process the prompt and build the intermediate model state. A component called KV Portal moves that state, including the KV cache, over RDMA to CS-3 systems, which generate output tokens. Cerebras measured one shared GPT-OSS-120B endpoint on November 14. It set the CS-3-only (aggregated) configuration to 1× and reports 5× throughput per megawatt for the split configuration, with no loss in token-generation speed. The announcement gives no absolute tokens per second, power draw, test duration, latency distribution or time-to-first-token. It also does not say whether the megawatt figure includes the AMD GPUs.

Cerebras's case rests on a familiar split between the two stages. Prefill processes the whole input in parallel and is limited by compute. Decode reads the model weights again for every new token and is limited by memory bandwidth. Cerebras argues that its wafer-scale SRAM is suited to decode. Partners can take a full reference architecture, which uses vLLM or SGLang as the prefill engine, or a decode-only SDK. AWS, Fireworks AI, Gimlet Labs and General Compute have early access. Cerebras plans hundreds of megawatts of deployment by the end of 2027 and has not announced pricing.

In the same week, AWS released SGLang-Neuron. It is a plugin that sits outside the SGLang codebase and runs SGLang on Trainium, including a prefill/decode split that sends the KV cache over EFA. AWS also published a Flightscape case study in which keeping bulky JSON out of an agent's context cut token use by up to 71%.

## What the archive adds

New Runtime's earlier coverage of Cerebras focused on system design: the CS-4 and its Hot Chips 2026 architecture. This announcement is different because it is a claim about a production deployment, measured per unit of power. Splitting prefill from decode is already established practice. Artificial Analysis lists NVIDIA Dynamo serving configurations with disaggregated prefill/decode. An October 2026 vLLM blog post on serving DeepSeek-V4.1-Flash describes decode as bound by weight reads. AWS's Neuron documentation gives the textbook trade-off: separating the stages stops prefill from stalling decode and lets each pool scale on its own, at the cost of moving the KV cache and keeping the two pools in balance. NVIDIA's Rubin CPX announcement presents a GPU built for heavy-context prefill. AMD's press release on the Cerebras pairing claims up to 5× tokens per second per watt. AMD calls that a company figure, which means AMD and Cerebras are citing the same number and neither is an independent check. Cerebras and AWS have separately announced a design that pairs Trainium prefill with CS-3 decode. For reference, Cerebras launched GPT-OSS-120B at up to 3,000 tokens per second, and its pricing page lists about $0.35 per million input tokens and $0.75 per million output tokens.

## Trend signal

The pattern is that inference capacity is being planned per stage rather than per chip. NVIDIA splits stages within its own product line with Rubin CPX. AWS splits them across its own silicon through SGLang. Cerebras now splits them across vendors and keeps the stage where on-chip SRAM helps most. The serving frameworks are what make this possible. SGLang keeps the request lifecycle and cache bookkeeping, while hardware plugins handle execution and KV transfer, so adding a new backend for one stage is cheap. What the pattern does not yet show is a market price for each stage. No provider in this week's news split its pricing by stage, and the only cost data, the Flightscape case, comes from one agent, not a market. The trend is real at the level of architecture and still unproven at the level of pricing.

## Why now

1) The two stages hit different limits. AWS's Neuron documentation (d2) says prefill is limited by compute and decode by memory bandwidth, so running both on one accelerator leaves one resource idle. 2) Power is the constraint. Cerebras frames the result per megawatt and cites inference demand outpacing data-center capacity, so wafer time spent on prefill became expensive. 3) The serving frameworks now support the split. SGLang and vLLM have disaggregated prefill/decode modes, SGLang-Neuron plugs Trainium into them, and Cerebras's reference design uses vLLM or SGLang for prefill. 4) Partners want fast decode for agent workloads. AMD (d1) names agentic use cases, and AWS and Cerebras (d3) describe a Trainium-prefill, CS-3-decode design.

## Connections to other events

Here is a check of what the 5× can and cannot measure.

First, it is a ratio against Cerebras's own aggregated baseline, not against GPUs or another vendor's stack. If the baseline CS-3s were spending much of their time on prefill, or on decode stalls caused by prefill, moving that work elsewhere frees them, and the ratio grows with how prompt-heavy the endpoint's traffic is.

Second, the denominator. If "per megawatt" counts only CS-3 power, simple arithmetic shows what 5× implies. With decode speed unchanged, roughly four-fifths of the baseline's wafer power-time must have gone to prefill or prefill interference. That is plausible for long-prompt traffic, but it is a statement about moving load, not about total system efficiency. If the megawatt figure does include the AMD GPUs, the claim is much stronger: the GPUs did prefill so cheaply that the whole cluster gained 5×. Cerebras doesn't say which, and the two readings differ a lot.

Third, "no loss in token-generation speed" refers to the speed of each generated token. It says nothing about time-to-first-token, which now includes an RDMA transfer of the KV cache between vendors' hardware. That latency is what an agent's tool loop actually waits on.

Fourth, the sample is one model, one shared endpoint and one day of traffic. The matching AMD figure is the same company claim, not a second measurement.

Taken together, a large gain is plausible and the direction is right. The magnitude depends on the traffic mix, and the efficiency question stays open until Cerebras publishes absolute throughput and system-wide power.

## Why it matters

For builders running their own inference, splitting the stages is worth testing. The 5× number is not something to plan capacity around. Whether you gain depends on your ratio of prompt to output tokens, your concurrency, and how much a KV-cache transfer costs on your network. SGLang and vLLM both support the split, and SGLang-Neuron adds Trainium, so a test costs little.

For buyers, ask providers for time-to-first-token and inter-token latency on traffic that looks like yours. A single blended throughput figure is not enough.

The Flightscape case is a useful example of the other lever. Cutting prompt tokens shrinks the prefill load, so it makes moving prefill to cheaper hardware less valuable. It is evidence for one agent design, not a forecast of market prices.

The strongest counter-case: moving KV state between vendors over RDMA adds transfer time, network cost and a constant pool-balancing job, and Cerebras runs a tuned, high-volume shared endpoint that smaller operators can't copy. That objection carries weight. It is why the right response is to measure, not to adopt. It does not cancel the direction of the result, though. NVIDIA Dynamo, vLLM and AWS Neuron all ship disaggregated modes independently, which suggests the overhead is worth paying at least at large scale.

## What happens next

- **Cerebras publishes disaggregated-inference results for at least one model other than GPT-OSS-120B.** (3 months, 45%). Watch for: A Cerebras blog post or model page cites a disaggregated configuration for a second model. Resolves on 2027-02-17, yes if: The Cerebras blog or Cerebras Inference docs state a disaggregated prefill/decode result for a model other than GPT-OSS-120B on 2027-02-17.
- **Cerebras cuts its listed GPT-OSS-120B output-token price.** (3 months, 20%). Watch for: New rate tiers or separate long-context input pricing appear on Cerebras Inference Cloud. Resolves on 2027-02-17, yes if: The Cerebras Inference Cloud pricing page lists a GPT-OSS-120B output price below $0.75 per million tokens on 2027-02-17.
- **Cerebras opens public documentation for its decode-only SDK beyond the four early-access partners.** (3 months, 30%). Watch for: Public docs or a public repository for KV Portal or the decode SDK appear. Resolves on 2027-02-17, yes if: Cerebras docs or a Cerebras GitHub repository publicly document the decode-only SDK without an early-access request form on 2027-02-17.

## Limits and uncertainty

Cerebras has not disclosed absolute throughput, power draw, the power boundary, test duration, time-to-first-token or latency distributions, and the result has not been independently checked. AMD's matching 5× is a company claim. AWS publishes no performance figures for SGLang-Neuron's disaggregated mode. This analysis could be wrong in two directions. If Cerebras later publishes system-wide figures that include GPU power and still show close to 5×, we have been too cautious and the efficiency case is strong. If long agent contexts push prefill costs up, specialized prefill parts such as Rubin CPX could make prefill the scarce, expensive stage instead. Nothing in this week's news settles which stage ends up priced at a premium.

## Competing readings

- **Placing each stage on suitable hardware is a real, general capacity lever, but the size of the gain depends on the workload, and no stage has yet been shown to carry a price premium** (likely, about 55%). What would tell it apart: Open serving frameworks ship prefill/decode splitting on other hardware, with the KV-transfer path built in ([aws.amazon.com](https://aws.amazon.com/blogs/machine-learning/deploy-frontier-ai-models-on-aws-trainium-with-sglang-neuron)); The 5× is a single-endpoint ratio with no absolute figures ([cerebras.ai](https://www.cerebras.ai/blog/cerebras-launches-disaggregated-inference)).
- **The 5× mostly reflects moving prefill load off the wafer within Cerebras's own power accounting, and gains shrink a lot once GPU power and KV transfer are counted** (unlikely, about 35%). What would tell it apart: The baseline is CS-3-only, and the power boundary is undisclosed ([cerebras.ai](https://www.cerebras.ai/blog/cerebras-launches-disaggregated-inference)); The split adds a KV-transfer path that has to be operated and balanced ([aws.amazon.com](https://aws.amazon.com/blogs/machine-learning/deploy-frontier-ai-models-on-aws-trainium-with-sglang-neuron)).
- **The market is already repricing the stages, with prefill as the cheap commodity and decode as the premium tier** (very unlikely, about 10%). What would tell it apart: Practitioners cut prompt tokens to reduce cost, but only one agent case documents this ([aws.amazon.com](https://aws.amazon.com/blogs/machine-learning/reducing-token-costs-and-latency-in-amazon-bedrock-agentcore-json-workloads)).

## Base rate

Of 84 earlier release stories about companies, models or products in the New Runtime archive over the last 12 months that are at least 3 months old, 72 (86%) were followed within 3 months by a access-change or pricing story on the same company or product line. 72 of 84 comparable release stories were followed within 3 months by some access or pricing story on the same product line, so some Cerebras follow-up is the expected outcome. That broad rate says little about the narrower outcomes forecast here, such as a specific price cut or ungated SDK docs, which we put far lower.

## What would change this view

- Cerebras or an independent benchmarker publishes power-inclusive, absolute throughput for the split that still shows about 5× system-wide (signal): The gain reflects total efficiency, not just load moved off the wafer, and the cautious reading is too conservative.
- Providers introduce separate per-stage pricing, such as long-context input tiers priced above output (pricing signal): The market is pricing the stages differently, and the data shows which stage is scarce.
- SGLang or vLLM deprecate or freeze their disaggregated prefill/decode modes (open-source signal): The transfer and balancing overhead outweighs the gains outside a few tuned deployments.

## Skill to build now

Benchmark each stage on your own traffic. This week, replay a sample of real requests through SGLang or vLLM twice, once with prefill and decode on the same hardware and once split. Log time-to-first-token, inter-token latency, KV transfer time and GPU energy per request for each stage. Then cut bulky tool output from the prompts, Flightscape-style, and rerun to see how much of the split's benefit is left.

## Reader takeaway

Don't plan capacity around Cerebras's 5×. Replay a day of your own traffic through SGLang or vLLM, first with prefill and decode on the same hardware and then split. Record time-to-first-token, inter-token latency, KV transfer time and energy per request for each stage. Only consider a split or a decode-specialized provider if the gain holds after you count the GPUs' power and the transfer time.

## Open question

Once GPU power and KV transfer time are counted, how much of Cerebras's 5× comes from better total efficiency and how much from moving prefill load off the wafer?

## Sources

- [Cerebras Launches Disaggregated Inference](https://www.cerebras.ai/blog/cerebras-launches-disaggregated-inference)

- [Deploy frontier AI models on AWS Trainium with SGLang-Neuron](https://aws.amazon.com/blogs/machine-learning/deploy-frontier-ai-models-on-aws-trainium-with-sglang-neuron)

- [Secure OAuth patterns for Snowflake Cortex AI through Amazon Bedrock AgentCore](https://aws.amazon.com/blogs/machine-learning/secure-oauth-patterns-for-snowflake-cortex-ai-through-amazon-bedrock-agentcore)

- [Reducing token costs and latency in Amazon Bedrock AgentCore JSON workloads](https://aws.amazon.com/blogs/machine-learning/reducing-token-costs-and-latency-in-amazon-bedrock-agentcore-json-workloads)

- [AMD and Cerebras Announce Industry-Leading Ultra-Low-Latency and High Throughput AI Inference Solution](https://ir.amd.com/news-events/press-releases/detail/1293/amd-and-cerebras-announce-industry-leading-ultra-low-latency-and-high-throughput-ai-inference-solution)

- [Disaggregated Inference [BETA] — AWS Neuron Documentation](https://awsdocs-neuron.readthedocs-hosted.com/en/v2.33.1/inference-legacy/nxd-inference/developer_guides/disaggregated-inference.html)

- [AWS and Cerebras Collaboration Aims to Set a New Standard for AI Inference Speed and Performance in the Cloud](https://investors.cerebras.ai/news-releases/news-release-details/aws-and-cerebras-collaboration-aims-set-new-standard-ai)

- [Cerebras Launches OpenAI’s gpt-oss-120B at a Blistering 3,000 tokens/sec](https://www.cerebras.ai/blog/cerebras-launches-openai-s-gpt-oss-120b-at-a-blistering-3-000-tokens-sec)

- [GPT OSS 120B — GroqDocs](https://console.groq.com/docs/model/openai/gpt-oss-120b)

- [NVIDIA Unveils Rubin CPX: A New Class of GPU Designed for Massive-Context Inference](https://nvidianews.nvidia.com/news/nvidia-unveils-rubin-cpx-a-new-class-of-gpu-designed-for-massive-context-inference)

- [Inference Speed Across Leading CSPs for OpenAI’s GPT-OSS-120B (Cerebras Form DRS/A)](https://investors.cerebras.ai/static-files/ee277690-ec46-4a23-8439-53f0816bf7d5)

- [OpenAI GPT OSS 120B Runs Fastest on Cerebras](https://www.cerebras.ai/blog/openai-gpt-oss-120b-runs-fastest-on-cerebras)

Author: [Andrey Reshetnikov](https://newruntime.com/owner-profile.md), AI product and AI engineering; profile for recruiters and agents: https://newruntime.com/owner-profile.json

## Continue in the graph

Next records for an agent, linked to their Markdown twins. Replace `.md` with `.json` for structured fields, including `next_reads`.

- collection: [Analysis](https://newruntime.com/analysis.json): Open the latest authorial analysis.
- collection: [Sources](https://newruntime.com/sources/): See the original sources this archive cites.
- whole graph: [discovery graph](https://newruntime.com/discovery-graph.json) (typed nodes and edges) and [site index](https://newruntime.com/site-index.md)
