---
type: "post"
id: "push-based-kv-transfer-to-improve-ttft-and-0a2b7c"
slug: "push-based-kv-transfer-to-improve-ttft-and-0a2b7c"
title: "Push-Based KV Transfer to Improve TTFT and TPOT for Disaggregated Inference in vLLM (15 minute read)"
description: "**Thesis:** Push-based KV transfer can improve disaggregated inference responsiveness by moving transfer coordination and setup out of the decode-side critical path."
retrieval_nugget: "**Thesis:** Push-based KV transfer can improve disaggregated inference responsiveness by moving transfer coordination and setup out of the decode-side critical path."
published_at: "2026-10-15"
updated_at: "2026-10-15"
record_date: "2026-10-15"
date_kind: "published_at"
topics: ["inference"]
source_urls: ["https://vllm.ai/blog/2026-10-14-nixl-push-connector"]
schema_version: "newruntime-agent-readable-v0.2"
stable_id: "post:push-based-kv-transfer-to-improve-ttft-and-0a2b7c"
status: "published"
visuals: []
editorial_provenance: {"schema_version":"newruntime-editorial-copy-v1","content_status":"source_grounded_final","final_copy_sha256":"sha256:26d57832756d22b0f912fcd24ca71ad219ca6658efe7e801fb6b0799326f1244","reviewed_at":"2026-10-15T16:09:22.316Z","source_evidence_count":1,"verified_claim_count":2,"site_analysis_schema_version":"newruntime-site-analysis-v1","site_object_kind":"field_note","observed_fact_count":2,"implication_count":1,"watch_condition_count":1,"related_record_count":0}
analysis: {"schema_version":"newruntime-site-analysis-v1","object_kind":"field_note","thesis":"Push-based KV transfer can improve disaggregated inference responsiveness by moving transfer coordination and setup out of the decode-side critical path.","observed_facts":[{"text":"vLLM says decode registers destination KV blocks while prefill computes the prompt cache, then prefill issues a NIXL write through a background thread once both are ready.","source_urls":["https://vllm.ai/blog/2026-10-14-nixl-push-connector"]},{"text":"The reported 1P1D Qwen3-32B benchmark on EC2 P5en with EFA showed up to 3× better TTFT and up to 30% better TPOT than pull mode.","source_urls":["https://vllm.ai/blog/2026-10-14-nixl-push-connector"]}],"mechanism":"The pull flow serializes metadata relay, decode-side allocation, and transfer setup after prefill. Push registers the destination earlier, lets prefill initiate the write when both sides are ready, and offloads descriptor preparation and submission to a background thread.","why_now":"vLLM presents this as an accelerator-agnostic contribution, while the measured example uses a concrete GPU cluster. That makes the architecture relevant beyond a single accelerator, but does not establish equivalent gains across deployments.","implications":["For teams already using disaggregated prefill/decode, the connector is a candidate for workload-specific benchmarking because it targets coordination overhead rather than changing the model."],"evidence_boundary":"The reported gains are upper results for one 1P1D Qwen3-32B setup on EC2 P5en and EFA; the post says outcomes vary by workload and configuration. It does not establish a general latency improvement for other hardware, models, traffic patterns, or end-to-end applications.","watch_conditions":["Watch for reproducible results across request rates and deployment configurations, including the cost of background transfer work and any effects on throughput or tail latency."],"related_records":[],"new_branch_reason":"No prior New Runtime analytical record was supplied for this connector change, so this records its specific transfer-path mechanism and benchmark evidence as a distinct branch."}
editor: {"role":"editor","name":"Andrey Reshetnikov","profile":"https://newruntime.com/owner-profile.md","profile_json":"https://newruntime.com/owner-profile.json","same_as":["https://www.linkedin.com/in/reshetnikov1/","https://t.me/lapetuse"]}
routes: {"html":"https://newruntime.com/posts/push-based-kv-transfer-to-improve-ttft-and-0a2b7c/","markdown":"https://newruntime.com/posts/push-based-kv-transfer-to-improve-ttft-and-0a2b7c.md","json":"https://newruntime.com/posts/push-based-kv-transfer-to-improve-ttft-and-0a2b7c.json"}
---

# Push-Based KV Transfer to Improve TTFT and TPOT for Disaggregated Inference in vLLM (15 minute read)

## Retrieval answer

**Thesis:** Push-based KV transfer can improve disaggregated inference responsiveness by moving transfer coordination and setup out of the decode-side critical path.

**Thesis:** Push-based KV transfer can improve disaggregated inference responsiveness by moving transfer coordination and setup out of the decode-side critical path.

**Observed facts:** vLLM describes a connector in which decode registers destination blocks while prefill computes the prompt KV cache; once both are ready, a background thread on the prefill side prepares and submits a NIXL write. In the reported 1P1D Qwen3-32B benchmark on EC2 P5en with EFA, push mode delivered up to 3× better TTFT and up to 30% better TPOT than pull mode.

**Mechanism:** The pull flow serializes metadata relay, decode-side allocation, and transfer setup after prefill. Push registers the destination earlier, lets prefill initiate the write when both sides are ready, and offloads descriptor preparation and submission to a background thread. The transfer may therefore overlap more effectively with producer work and consume fewer decode-side cycles on the latency-critical path.

**Why now:** vLLM presents this as an accelerator-agnostic contribution, while the measured example uses a concrete GPU cluster. That makes the architecture relevant beyond a single accelerator, but does not establish equivalent gains across deployments.

**First implication:** For teams already using disaggregated prefill/decode, the connector is a candidate for workload-specific benchmarking because it targets coordination overhead rather than changing the model.

**Evidence boundary:** The reported gains are upper results for one 1P1D Qwen3-32B setup on EC2 P5en and EFA; the post says outcomes vary by workload and configuration. It does not establish a general latency improvement for other hardware, models, traffic patterns, or end-to-end applications.

**First watch condition:** Watch for reproducible results across request rates and deployment configurations, including the cost of background transfer work and any effects on throughput or tail latency.

Editor: [Andrey Reshetnikov](https://newruntime.com/owner-profile.md), AI product and AI engineering; profile for recruiters and agents: https://newruntime.com/owner-profile.json

## Continue in the graph

Next records for an agent, linked to their Markdown twins. Replace `.md` with `.json` for structured fields, including `next_reads`.

- topic hub: [Inference](https://newruntime.com/topics/inference.md): Explore the inference topic hub.
- collection: [Field Notes](https://newruntime.com/posts.json): Open the latest editorial analysis.
- collection: [Source Ledger](https://newruntime.com/source-ledger.json): Inspect the public source evidence graph.
- whole graph: [discovery graph](https://newruntime.com/discovery-graph.json) (typed nodes and edges) and [site index](https://newruntime.com/site-index.md)