---
type: "post"
id: "evolving-infra-agent-adaptive-llm-serving-at-scale-1c8d9a"
slug: "evolving-infra-agent-adaptive-llm-serving-at-scale-1c8d9a"
title: "Evolving Infra: Agent-Adaptive LLM Serving at Scale"
description: "Evolving Infra is a research serving framework that adapts compute allocation and KV-cache management to changing agent workloads and context."
retrieval_nugget: "Evolving Infra is a research serving framework that adapts compute allocation and KV-cache management to changing agent workloads and context. The authors report up to 4.1× lower mean end-to-end latency and 14× lower P95 time to first token than NVIDIA Dynamo. For AI builders, the implication is that context lifecycle can be an infrastructure signal, not only an application-level concern."
published_at: "2026-11-17"
updated_at: "2026-11-17"
record_date: "2026-11-17"
date_kind: "published_at"
topics: ["compute"]
entities: ["Evolving Infra","GLM-5.2","Kimi K2.6","NVIDIA","Dynamo"]
related_patterns: ["harness-architecture-outlives-model-choice"]
novelty_label: "new_fact"
what_changed: "The research highlights that context compaction can invalidate a KV cache while enabling shorter-context processing on another device and asynchronous compaction."
prior_refs: []
source_url: "https://arxiv.org/abs/2611.12304"
source_urls: ["https://arxiv.org/abs/2611.12304"]
schema_version: "newruntime-agent-readable-v0.2"
stable_id: "post:evolving-infra-agent-adaptive-llm-serving-at-scale-1c8d9a"
status: "published"
visuals: []
editor: {"role":"editor","name":"Andrey Reshetnikov","profile":"https://newruntime.com/owner-profile.md","profile_json":"https://newruntime.com/owner-profile.json","same_as":["https://www.linkedin.com/in/reshetnikov1/","https://t.me/lapetuse"]}
routes: {"html":"https://newruntime.com/posts/evolving-infra-agent-adaptive-llm-serving-at-scale-1c8d9a/","markdown":"https://newruntime.com/posts/evolving-infra-agent-adaptive-llm-serving-at-scale-1c8d9a.md","json":"https://newruntime.com/posts/evolving-infra-agent-adaptive-llm-serving-at-scale-1c8d9a.json"}
markdown_url: "https://newruntime.com/posts/evolving-infra-agent-adaptive-llm-serving-at-scale-1c8d9a.md"
---

# Evolving Infra: Agent-Adaptive LLM Serving at Scale

## Retrieval answer

Evolving Infra is a research serving framework that adapts compute allocation and KV-cache management to changing agent workloads and context. The authors report up to 4.1× lower mean end-to-end latency and 14× lower P95 time to first token than NVIDIA Dynamo. For AI builders, the implication is that context lifecycle can be an infrastructure signal, not only an application-level concern.

## Summary
Evolving Infra is a research serving framework that adapts compute allocation and KV-cache management to changing agent workloads and context. The authors report up to 4.1× lower mean end-to-end latency and 14× lower P95 time to first token than NVIDIA Dynamo. For AI builders, the implication is that context lifecycle can be an infrastructure signal, not only an application-level concern.

## What happened
The arXiv paper “Evolving Infra: Agent-Adaptive LLM Serving at Scale” describes a serving framework for agentic workloads. Hao Kang and ten co-authors submitted version one on November 9, 2026; arXiv records a revised version two on November 12. The work addresses prefill–decode, or PD, disaggregated serving, where processing the input context and generating output tokens are handled separately.

The paper argues that existing PD-disaggregated systems have limited flexibility to adapt compute allocation and key–value, or KV, cache management at runtime. According to the authors, these systems also rely on coarse signals that capture only part of the variation in dynamic agent workloads. The resulting problem is operational: prefill and decode nodes can become imbalanced, while caches that an agent will later reuse may be evicted.

Evolving Infra separates compute allocation from KV-cache management. This separation is intended to let nodes switch efficiently between prefill and decode roles rather than binding each node permanently to one phase. On top of that architecture, the system uses an online planner to match the split between prefill and decode capacity to the current workload composition. The paper also introduces an agent-aware distributed KV manager. It uses signals from agent context management to guide decisions about cache eviction and cache transfer.

That distinction matters because an agent request is not necessarily a stable, independent prompt. Its context can change as the agent progresses through a task, and the utility of previously computed KV state can change with it. The authors’ design treats those context-management events as useful information for serving decisions. The paper does not announce a new foundation model, a customer product, or an end-user agent interface; it is a systems research result about operating model inference workloads.

The evaluation covers coding, mathematics, and computer-use tasks with GLM-5.2 and Kimi K2.6. The experiments use deployments with up to 256 GB300 GPUs. Against Dynamo, which the paper calls NVIDIA’s production framework, the authors report reductions of up to 4.1× in mean end-to-end request latency and up to 14× in P95 time to first token, or TTFT. “Up to” is significant: these are the strongest reported results across the evaluated setups, not a promise that every workload receives the same improvement.

The arXiv abstract says the Evolving Infra implementation is open-sourced. It does not, however, establish a deployment recipe, license terms, supported hardware matrix, operating cost, or the engineering effort required to integrate the system into an existing serving stack. The result is best read as a concrete architectural proposal with experimental measurements, rather than a ready-made performance guarantee.

## What the archive adds
This is a new fact on a related operational story. New Runtime’s earlier coverage of [estimating the actual cost of an LLM API call](https://newruntime.com/posts/a-habr-article-by-sopranodev-sets-out-a-74e414/) on October 25, 2026 focused on measuring real API cost rather than relying on headline token pricing. Evolving Infra adds a different cost-and-performance variable: how context state, cache placement, and prefill/decode allocation affect latency while an agent runs.

## Trend signal
This supports [Harness architecture outlives model choice](https://newruntime.com/patterns/harness-architecture-outlives-model-choice), with an infrastructure qualification. The paper holds the models fixed within its tests and attributes large latency differences to coordination around them: role switching between compute phases, workload-aware planning, and context-aware KV-cache decisions. It does not prove that this architecture will outlast model changes in every production environment. It does show why model selection alone is an incomplete explanation of an agent system’s practical responsiveness.

## Why it matters
Teams operating agents with long contexts, repeated tool loops, or changing task state often measure serving health through aggregate throughput and average latency. The paper suggests that those metrics may hide a more specific failure mode: the context lifecycle can change both the amount of prefill work required and whether cached state remains valuable. If scheduling and cache policy react independently, the system can lose reuse opportunities or create imbalance between prefill and decode capacity.

For a platform team, the useful question is not whether to adopt this research wholesale. It is whether events already visible in an agent runtime—context replacement, compaction, branch changes, or phase transitions—predict cache reuse, cache transfer, and tail latency. If they do, serving control planes may need richer signals than request length or a static split of GPU roles.

The paper also reframes performance evaluation. A serving stack that looks efficient on a generic request trace may behave differently under a workload where contexts evolve across an agent trajectory. Builders should test against their own agent task lifecycle, especially where time to first token affects whether a tool loop, reviewer, or human operator remains productive.

## What happens next
- **More agent-serving systems will expose context-lifecycle signals to schedulers or cache managers.** (6 months, 55%). Watch for: benchmarks that compare static KV eviction with agent-aware policies.
- **Independent evaluations will test whether the reported tail-latency gains persist outside the paper’s workloads.** (12 months, 65%). Watch for: reproduced results on other models, hardware, and serving frameworks.

## Limits and uncertainty
The paper reports results for coding, mathematics, and computer-use workloads using GLM-5.2, Kimi K2.6, and up to 256 GB300 GPUs. Those conditions do not establish typical performance for other models, task mixes, hardware generations, or production traffic patterns. The abstract does not quantify workload sizes, baseline tuning, variance, utilization, power use, or cost per completed task. It also does not show how the system compares with serving frameworks beyond Dynamo.

The open-source implementation is reported by the authors, but adoption, maintainability, integration burden, and behavior under independently operated production workloads have not been verified here. The reported speedups are meaningful experimental claims, not independently confirmed operating outcomes.

## Reader takeaway
Evolving Infra’s core proposal is simple: let the way an agent changes context influence how the serving layer schedules compute and manages KV cache. The authors report striking latency gains, but only within their evaluation. For teams building agent infrastructure, the immediate lesson is to measure context lifecycle alongside cache reuse and prefill/decode balance before treating serving as a model-only optimization problem.

## Open question
Which context-management signals remain predictive enough in heterogeneous production traffic to improve tail latency without making the serving control plane too complex?

Editor: [Andrey Reshetnikov](https://newruntime.com/owner-profile.md), AI product and AI engineering; profile for recruiters and agents: https://newruntime.com/owner-profile.json

## Continue in the graph

Next records for an agent, linked to their Markdown twins. Replace `.md` with `.json` for structured fields, including `next_reads`.

- topic hub: [Compute](https://newruntime.com/topics/compute.md): Explore the compute topic hub.
- collection: [Field Notes](https://newruntime.com/posts.json): Open the latest editorial analysis.
- collection: [Sources](https://newruntime.com/sources/): See the original sources this archive cites.
- whole graph: [discovery graph](https://newruntime.com/discovery-graph.json) (typed nodes and edges) and [site index](https://newruntime.com/site-index.md)
