What happened
LMSYS announced Unified Memory for SGLang on November 11, 2026. It targets hybrid models whose layers combine different kinds of state: full-attention KV grows with context length, sliding-window KV retains a bounded context suffix, and recurrent state consumes a fixed-size allocation per running request. In SGLang’s default design, each type uses a separate sub-pool sized at startup. Operators use --swa-full-tokens-ratio and --mamba-full-memory-ratio to choose the split, but the right split depends on traffic. A workload can exhaust one pool while another has idle capacity.
With --enable-unified-memory, those state types share one GPU memory arena. Full-attention KV and bounded state grow inward from opposite ends, with compaction moving live allocations so free bytes can be reused by whichever type needs them. Their allocation geometries remain distinct: KV uses pages, recurrent state uses slots, and operators can select a KV --page-size. SGLang also changes the KV layout so each token’s rows across layers are packed into one entry. That allows allocation, eviction and movement to operate on the token’s state together, including compatible speculative-draft rows with different shapes or dtypes.
A common allocator handles capacity checks for request admission, decode headroom, cache load-back, prefill-decode preallocation and eviction. It estimates a request’s combined demand across full KV, window KV and recurrent-state slots, including page rounding and decode allowance. A scheduler-aware page-use ledger accounts for GPU work still in flight: a page can count as logically freed immediately, but its physical bytes are not overwritten until older operations that may use them have finished. The feature works with several serving capabilities, including overlap scheduling, Radix cache, HiCache, speculative decoding and pipeline parallelism, subject to restrictions.
LMSYS reports tests against SGLang’s separate pools at their default ratios, averaging two runs on one H200 per server, using SGLang commit daa171846f. On Qwen3.5-4B with 1,024 input tokens, 256 output tokens and concurrency 64, Unified Memory sustained 5,830 output tokens per second versus 5,880 for separate pools; both scored 0.875 on GSM8K’s 200 questions. On GPT-OSS-20B with 512 input and 1,536 output tokens at concurrency 1,024, Unified Memory ran about 1,000 requests at once versus 618, with 14% higher throughput and time to first token of 6 seconds rather than 43. On Qwen3.5-4B with 16K input and 4,096 output tokens at concurrency 64, it ran 32 requests versus 22 and was 26% faster.
What the archive adds
New Runtime covered SGLang’s v0.5.23 release on November 3, 2026, including serving-stack capabilities beyond its earlier TPU engine announcement (earlier coverage). Unified Memory is a new feature event in that ongoing SGLang thread, not a restatement of the release. The change addresses GPU-memory allocation across hybrid-model state rather than adding another serving backend.
Trend signal
This supports the idea that production agent and inference systems depend on a durable serving harness, not only model choice. SGLang’s shared allocator integrates admission, caching, scheduling and in-flight page safety behind one memory system, a concrete instance of harness architecture outliving model choice. The evidence is specific to an experimental feature and selected workloads, so it does not establish a general performance advantage.
Why now
LMSYS frames the feature around hybrid architectures whose memory demands diverge: long contexts stress full-attention KV, while many short concurrent requests can stress bounded state. Fixed startup ratios can strand capacity as workload mix changes. Unified Memory is intended to remove that runtime partitioning constraint; LMSYS says it is still experimental and aims to make it the default after closing remaining model, backend and lifecycle gaps.
Why it matters
For teams serving supported hybrid models, memory allocation is part of effective capacity. A GPU can have free bytes that are unusable to the pool under pressure, so tuning startup ratios becomes an operational dependency and a source of throughput loss. A shared arena can let requests draw from capacity according to their actual mix of context and concurrency, potentially raising concurrency without changing hardware.
The reported gains are not a blanket promise: the benchmarks compare against default pool ratios on one H200, and badly chosen ratios perform worse. Teams should evaluate the feature against their own prompts, concurrency, cache behavior and latency goals, while checking the supported-model and backend matrix. The unified allocator also introduces relocation and lifecycle complexity; its value depends on safe coordination with ongoing GPU work and integrations such as caching or disaggregated serving.
Limits and uncertainty
LMSYS labels Unified Memory experimental and says it currently targets hybrid models. Full-attention-only models, MLA with sliding-window layers and DeepSeek-V4 are unsupported; additional backend and feature restrictions apply. Priority preemption is disabled, and request slots remain fixed by --max-running-requests. The company also describes a case where window rows can temporarily exceed the amount priced at admission if recurrent checkpoints cannot advance. The published results cover selected models and configurations, average two runs and use one H200 per server; they do not establish behavior across hardware, model families or production traffic. Independent replication and broader measurements are not reported.
Reader takeaway
Unified Memory tackles a real serving inefficiency: memory reserved for one kind of hybrid-model state can sit idle while another kind runs short. The reported tests show meaningful gains in selected workloads, but this is an experimental, compatibility-limited feature. Benchmark it with your traffic and verify the restriction matrix before treating the results as capacity you can count on.