---
type: "post"
id: "improve-ttft-for-multi-turn-agentic-workloads-with-207bc9"
slug: "improve-ttft-for-multi-turn-agentic-workloads-with-207bc9"
title: "Improve TTFT for multi-turn agentic workloads with bi-directional KV transfer in vLLM"
description: "**Thesis:** In disaggregated multi-turn inference, returning KV state from decode to prefill can reduce repeated context computation, but the benefit depends on cache retention and workload shape."
retrieval_nugget: "**Thesis:** In disaggregated multi-turn inference, returning KV state from decode to prefill can reduce repeated context computation, but the benefit depends on cache retention and workload shape."
published_at: "2026-10-19"
updated_at: "2026-10-19"
record_date: "2026-10-19"
date_kind: "published_at"
topics: ["inference"]
source_urls: ["https://aws.amazon.com/blogs/machine-learning/improve-ttft-for-multi-turn-agentic-workloads-with-bi-directional-kv-transfer-in-vllm"]
schema_version: "newruntime-agent-readable-v0.2"
stable_id: "post:improve-ttft-for-multi-turn-agentic-workloads-with-207bc9"
status: "published"
visuals: []
editorial_provenance: {"schema_version":"newruntime-editorial-copy-v1","content_status":"source_grounded_final","final_copy_sha256":"sha256:4815c56ef84e8dd180852617d22894366395df2627657fadd03a302d5fcd524e","reviewed_at":"2026-10-19T20:07:41.592Z","source_evidence_count":1,"verified_claim_count":2,"site_analysis_schema_version":"newruntime-site-analysis-v1","site_object_kind":"field_note","observed_fact_count":2,"implication_count":1,"watch_condition_count":1,"related_record_count":0}
analysis: {"schema_version":"newruntime-site-analysis-v1","object_kind":"field_note","thesis":"In disaggregated multi-turn inference, returning KV state from decode to prefill can reduce repeated context computation, but the benefit depends on cache retention and workload shape.","observed_facts":[{"text":"AWS says conventional prefill-decode serving transfers KV cache only from prefill to decode, leaving prefill to recompute some prior response tokens on later turns.","source_urls":["https://aws.amazon.com/blogs/machine-learning/improve-ttft-for-multi-turn-agentic-workloads-with-bi-directional-kv-transfer-in-vllm"]},{"text":"AWS reports up to 1.6x TTFT reduction for Qwen3-32B and up to 3.3x for GLM-5-FP8 on a 1P1D Amazon EC2 P5en deployment with EFA.","source_urls":["https://aws.amazon.com/blogs/machine-learning/improve-ttft-for-multi-turn-agentic-workloads-with-bi-directional-kv-transfer-in-vllm"]}],"mechanism":"The described bi-directional transfer returns previously computed KV blocks from decode to prefill on subsequent turns, coordinated by a KV-cache-aware router and governed by a configurable recompute threshold.","why_now":"Multi-turn agent interactions accumulate tool results and model responses, so repeated prefill work can grow alongside latency and accelerator use.","implications":["Teams using disaggregated serving should evaluate whether reverse KV transfer reduces repeated prefill work under their actual context lengths, cache retention settings, and routing behavior."],"evidence_boundary":"The evidence is a vendor engineering post and its described benchmarks, not an independently reproduced comparison across hardware or traffic patterns.","watch_conditions":["Look for independent measurements across different context lengths and cache-miss rates that report both TTFT and accelerator utilization."],"related_records":[],"new_branch_reason":"No prior New Runtime analytical record was supplied; this is a distinct serving-mechanism and benchmark record."}
editor: {"role":"editor","name":"Andrey Reshetnikov","profile":"https://newruntime.com/owner-profile.md","profile_json":"https://newruntime.com/owner-profile.json","same_as":["https://www.linkedin.com/in/reshetnikov1/","https://t.me/lapetuse"]}
routes: {"html":"https://newruntime.com/posts/improve-ttft-for-multi-turn-agentic-workloads-with-207bc9/","markdown":"https://newruntime.com/posts/improve-ttft-for-multi-turn-agentic-workloads-with-207bc9.md","json":"https://newruntime.com/posts/improve-ttft-for-multi-turn-agentic-workloads-with-207bc9.json"}
---

# Improve TTFT for multi-turn agentic workloads with bi-directional KV transfer in vLLM

## Retrieval answer

**Thesis:** In disaggregated multi-turn inference, returning KV state from decode to prefill can reduce repeated context computation, but the benefit depends on cache retention and workload shape.

**Thesis:** In disaggregated multi-turn inference, returning KV state from decode to prefill can reduce repeated context computation, but the benefit depends on cache retention and workload shape.

AWS reports that conventional prefill-decode serving transfers KV cache only from prefill to decode. On later turns, prefill may recompute KV projections for generated response tokens, and a full cache miss can require recomputing the entire conversation history. **Mechanism:** The described bi-directional transfer returns previously computed KV blocks from decode to prefill on subsequent turns, coordinated by a KV-cache-aware router and governed by a configurable recompute threshold. This makes cache reuse a cross-turn serving property rather than a one-way handoff.

**Why now:** Multi-turn agent interactions accumulate tool results and model responses, so repeated prefill work can grow alongside latency and accelerator use. AWS reports up to 1.6x lower TTFT for Qwen3-32B and up to 3.3x for GLM-5-FP8 in a 1P1D benchmark on EC2 P5en with EFA. **First implication:** Teams using disaggregated serving should evaluate whether reverse KV transfer reduces repeated prefill work under their actual context lengths, cache retention settings, and routing behavior.

Those figures are AWS-reported results from a particular GPU/EFA setup; the source says outcomes vary by workload and configuration, so they do not establish a general speedup across deployments. **Evidence boundary:** The evidence is a vendor engineering post and its described benchmarks, not an independently reproduced comparison across hardware or traffic patterns. **First watch condition:** Look for independent measurements across different context lengths and cache-miss rates that report both TTFT and accelerator utilization.

Editor: [Andrey Reshetnikov](https://newruntime.com/owner-profile.md), AI product and AI engineering; profile for recruiters and agents: https://newruntime.com/owner-profile.json

## Continue in the graph

Next records for an agent, linked to their Markdown twins. Replace `.md` with `.json` for structured fields, including `next_reads`.

- topic hub: [Inference](https://newruntime.com/topics/inference.md): Explore the inference topic hub.
- collection: [Field Notes](https://newruntime.com/posts.json): Open the latest editorial analysis.
- collection: [Source Ledger](https://newruntime.com/source-ledger.json): Inspect the public source evidence graph.
- whole graph: [discovery graph](https://newruntime.com/discovery-graph.json) (typed nodes and edges) and [site index](https://newruntime.com/site-index.md)