What happened
In a November 2, 2026 post, LMSYS describes how HiCache extends radix-tree prefix caching from GPU memory into host memory and external storage, and what changes when serving is distributed across multiple ranks. The central problem is consistency: a prefix hit must identify the same logical cached pages across ranks even when those pages are physically stored on different GPUs. The article focuses on preserving that relationship as systems add parallel computation, asynchronous transfers, draft models, or separate prefill and decode workers.
For context parallelism (CP), the post describes page-based key-value (KV) sharding. Each rank maintains the same logical radix tree and page identifiers, while the KV data for each page belongs to one rank. A deterministic mapping derives the owner rank and local address from the logical page identifier, rather than requiring a separate per-page lookup table. During attention, ranks gather their owned pages for one layer into a temporary buffer; page-table mapping preserves their logical order. Two scratch buffers can alternate across layers, overlapping the next layer’s gather with current-layer computation. The scheduler accounts for that buffer memory when deciding whether a batch fits. The post notes that on supported configurations the gather can use GPU copy engines rather than NCCL SM kernels, reducing competition with model computation. Gathering still has a cost: a cache hit avoids recomputing the prefix but requires assembling distributed KV.
For pipeline parallelism (PP), HiCache must keep cache decisions aligned across stages even when asynchronous host transfers finish at different times. When restoring sharded pages, each rank must reload its own pages to preserve ownership before the subsequent gather. The article also covers speculative decoding: the required draft-model state has to be restored alongside the target model’s cache. A unified KV layout can allow reuse across compatible parallel configurations. In prefill-decode (P/D) disaggregation, a decode worker may already have a reusable prefix; transferring only the missing KV avoids sending the whole prefix again. The post describes an external-storage approach in which workers agree on the cached prefix before reuse.
The practical example is the iFlytek Astron MaaS team serving GLM-5.1/5.2 CodingPlan workloads with pipeline parallelism, P/D disaggregation, and multi-token prediction (MTP). Requests typically contain 30,000 to 200,000 input tokens, and some reach 1 million. The team reports that before adding HiCache its cache-hit rate was around 30%; with HiCache and external storage it reached 95%, while median time to first token (TTFT) fell below three seconds. LMSYS also introduces rank_consensus_checker for locating inconsistent cache decisions across ranks. The supplied account gives no additional benchmark methodology, hardware configuration, or absolute cost figures.
What the archive adds
This is a new technical report, not a restatement of our earlier serving coverage. vLLM’s report on serving DeepSeek-V4.1-Flash at scale also addressed production inference serving, but the present post adds specific HiCache mechanisms for distributed prefix reuse and an iFlytek-reported hit-rate and TTFT result.
Trend signal
No supplied SITE_PATTERNS thesis directly covers distributed KV-cache placement and restoration. The post is adjacent to the broader shift toward serving architectures that coordinate more than a model call, but treating it as evidence for a specific listed pattern would overstate what this source establishes.
Why now
The post frames the problem around agentic and coding workloads that repeatedly submit long, mostly unchanged histories. As deployments add parallelism, speculative decoding, and separate prefill and decode workers, cache reuse must survive across more boundaries. The source presents these serving arrangements as the setting for HiCache’s design, not as a newly announced access or pricing change.
Why it matters
For teams serving long-context workloads, prefix caching is not simply a matter of keeping reusable KV in one GPU pool. Sharding can reduce duplicate storage, but it makes page ownership, cache decisions, and restoration order part of the serving contract. A mismatch between ranks can undermine a hit; a successful hit can still incur gather and transfer costs. The article gives systems engineers concrete design concerns to evaluate when extending a cache beyond one device or topology.
The iFlytek figures make the potential operational value legible: in that deployment, a higher cache-hit rate coincided with median TTFT below three seconds on requests often tens or hundreds of thousands of tokens long. They are a team-reported result, not a guarantee for another workload. Builders should therefore treat the result as a reason to measure their own prefix reuse, transfer cost, and latency—not as a prediction that HiCache will deliver the same improvement in another deployment.
Limits and uncertainty
The source describes implementation choices and reports one deployment result, but the supplied material does not establish an independently reproduced benchmark. It does not provide hardware details, workload mix, traffic volume, the precise measurement window, the baseline latency, or a breakdown of the external-storage costs. The reported cache-hit and TTFT changes cannot isolate the effect of HiCache from other deployment choices, nor show how results vary across models or configurations. Compatibility is also qualified: unified KV reuse is described for compatible parallel configurations, and the zero-SM gather applies only to supported configurations. The post does not establish general availability, licensing, pricing, or a broadly applicable performance guarantee.
Reader takeaway
HiCache’s engineering challenge is to preserve a coherent logical prefix while the underlying KV pages move across ranks, memory tiers, and serving stages. The iFlytek case suggests that solving this can matter for long coding and agent histories, but each team still needs to validate cache hits, transfer overhead, and TTFT in its own topology.
Open question
How much of the reported latency improvement comes from cache reuse itself versus the deployment’s parallelism and external-storage configuration?