---
type: "post"
id: "buoyant-publishes-kinference-vs-plain-kubernetes-distributed-benchmarks-de2cef"
slug: "buoyant-publishes-kinference-vs-plain-kubernetes-distributed-benchmarks-de2cef"
title: "Buoyant publishes Kinference vs plain Kubernetes distributed benchmarks"
description: "Buoyant reports that Kinference, its Kubernetes inference control plane, cut median time-to-first-token by 95% on a shared-prefix workload and increased goodput substantially in tests against a plain Kubernetes Service."
retrieval_nugget: "Buoyant reports that Kinference, its Kubernetes inference control plane, cut median time-to-first-token by 95% on a shared-prefix workload and increased goodput substantially in tests against a plain Kubernetes Service. For teams serving open-weight models, the result highlights how routing decisions that account for cached prompt context can matter as much as adding replicas."
published_at: "2026-11-07"
updated_at: "2026-11-07"
record_date: "2026-11-07"
date_kind: "published_at"
topics: ["inference"]
entities: ["Buoyant","Kinference","vLLM","Qwen-3.8-27B-FP8","Kubernetes","GuideLLM"]
related_patterns: ["harness-architecture-outlives-model-choice"]
novelty_label: "new_fact"
what_changed: "Buoyant published distributed-inference benchmarks comparing Kinference with plain Kubernetes."
prior_refs: ["https://newruntime.com/posts/buoyant-open-sources-kinference-an-inference-control-plane-aa8c1e/"]
source_url: "https://www.buoyant.io/blog/distributed-inference-benchmarks-kinference-vs-plain-kubernetes"
source_urls: ["https://www.buoyant.io/blog/distributed-inference-benchmarks-kinference-vs-plain-kubernetes"]
schema_version: "newruntime-agent-readable-v0.2"
stable_id: "post:buoyant-publishes-kinference-vs-plain-kubernetes-distributed-benchmarks-de2cef"
status: "published"
visuals: []
editor: {"role":"editor","name":"Andrey Reshetnikov","profile":"https://newruntime.com/owner-profile.md","profile_json":"https://newruntime.com/owner-profile.json","same_as":["https://www.linkedin.com/in/reshetnikov1/","https://t.me/lapetuse"]}
routes: {"html":"https://newruntime.com/posts/buoyant-publishes-kinference-vs-plain-kubernetes-distributed-benchmarks-de2cef/","markdown":"https://newruntime.com/posts/buoyant-publishes-kinference-vs-plain-kubernetes-distributed-benchmarks-de2cef.md","json":"https://newruntime.com/posts/buoyant-publishes-kinference-vs-plain-kubernetes-distributed-benchmarks-de2cef.json"}
markdown_url: "https://newruntime.com/posts/buoyant-publishes-kinference-vs-plain-kubernetes-distributed-benchmarks-de2cef.md"
---

# Buoyant publishes Kinference vs plain Kubernetes distributed benchmarks

## Retrieval answer

Buoyant reports that Kinference, its Kubernetes inference control plane, cut median time-to-first-token by 95% on a shared-prefix workload and increased goodput substantially in tests against a plain Kubernetes Service. For teams serving open-weight models, the result highlights how routing decisions that account for cached prompt context can matter as much as adding replicas.

## Summary
Buoyant reports that Kinference, its Kubernetes inference control plane, cut median time-to-first-token by 95% on a shared-prefix workload and increased goodput substantially in tests against a plain Kubernetes Service. For teams serving open-weight models, the result highlights how routing decisions that account for cached prompt context can matter as much as adding replicas.

## What happened
Buoyant's November 4, 2026 benchmark compares Kinference with a standard Kubernetes Service in front of two identical pools of vLLM 0.31.0. Each pool had three pods, each on its own AWS L40S GPU, running the Qwen-3.8-27B-FP8 open-weight model. One OpenAI-compatible chat-completion endpoint used Kinference; the other pointed to the second pool's Kubernetes Service. For the service arm, kube-proxy in iptables mode distributed new connections randomly across replicas. GuideLLM 0.8.1 generated the benchmark traffic.

Kinference predicts time-to-first-token (TTFT) for each request against candidate vLLM pods, using queue size and KV-cache contents, then routes to the predicted fastest worker. The KV cache stores intermediate prefill results. If a later request shares a prefix with an earlier one, a worker that retains that prefix can reuse the work rather than recomputing it. Buoyant says Kinference can select another GPU when the cache-holding worker is busy if that alternative is predicted to respond faster.

Buoyant tested three traffic types. The control workload used independent 16,256-token prompts with no shared prefix and 256 output tokens. The shared-prefix workload selected one of 24 shared 16,000-token prefixes, then added 256 unique tokens and requested 256 output tokens. The multiturn workload used eight-turn conversations: an 8,000-token first prompt, then 400 new tokens per turn plus the growing conversation history, with 200 output tokens. The unloaded-latency experiment used low-rate open-loop arrivals; each workload received a 60-second warmup and 45-second cooldown, with results measured over 240 seconds.

In that unloaded-latency test, the shared-prefix workload's median TTFT fell from 6.88 seconds through Kubernetes Service to 0.31 seconds through Kinference. Median end-to-end latency fell from 20.5 to 11.3 seconds. For multiturn traffic, median TTFT moved from 1.70 to 0.59 seconds and end-to-end latency from 13.4 to 9.1 seconds. On the control workload, median TTFT was effectively unchanged: 6.92 versus 6.83 seconds. Buoyant reports lower p95 TTFT even in the control case, which it attributes likely to queue-size-aware routing as well as cache overlap.

For capacity, Buoyant used closed-loop traffic at concurrency levels 4, 8, 16, 24 and 32, meaning a client sent another request when its preceding request completed. It defined goodput as requests per minute meeting a selected TTFT service-level objective. The shared-prefix objective was 90% of requests reaching TTFT of eight seconds or less; the multiturn objective was 90% at four seconds or less. Kinference's peak reported goodput was 103 requests per minute on shared-prefix traffic, versus 14 for the service arm, and 82.7 versus 23.8 on multiturn traffic. Kinference met the shared-prefix SLO through the highest tested concurrency, 32, and the multiturn SLO through concurrency 8. The service arm did not meet either SLO at any tested concurrency. Buoyant says it ran three trials for each condition and used a paired hierarchical bootstrap with 2,000 resamples for confidence intervals.

## What the archive adds
New Runtime covered Buoyant's announcement of Kinference on November 7, 2026, describing it as an open-source control plane for vLLM on Kubernetes ([earlier note](https://newruntime.com/posts/buoyant-open-sources-kinference-an-inference-control-plane-aa8c1e/)). This is a new fact on a known story: Buoyant now reports comparative benchmark results for the routing approach.

## Trend signal
This supports [harness-architecture-outlives-model-choice](https://newruntime.com/patterns/harness-architecture-outlives-model-choice/): in production inference, the serving and routing layer can shape user-visible performance independently of the model choice. The benchmark is vendor-run and narrow, so it is evidence for the mechanism rather than proof that the same gains generalize across stacks.

## Why it matters
The practical implication is not simply that Kinference is faster. It is that replica count alone may leave performance on the table when requests reuse long prefixes, as happens across turns in conversations or when many users share system prompts and documents. A random choice of pod can force repeated prefill work, consuming GPU capacity that could otherwise serve new requests. Cache-aware routing can therefore improve both latency and the fraction of requests that meet a service-level objective without changing the model.

The benchmark also separates two effects that are often conflated in inference planning. Output throughput was roughly equal in the unloaded tests, while the large end-to-end gains came principally from reducing TTFT and avoiding repeated prefix computation. That distinction matters for workloads where an agent, user, or downstream service waits for an answer to begin, rather than merely for a fixed number of tokens to be generated.

Teams should treat the reported multipliers as a reason to test their own traffic, not as capacity-planning constants. The gains were largest for a workload designed around shared prefixes, while independent prompts showed no median TTFT advantage. A useful evaluation should include the team's actual prefix reuse, concurrency, latency target, GPU type, vLLM version, and queue behavior. Those measurements can establish whether routing is a meaningful lever before buying more GPUs or changing models.

## Limits and uncertainty
These are Buoyant's own tests, not an independent replication. They use one model, one GPU type, three replicas per arm, a specific vLLM release, and a Kubernetes Service path using kube-proxy in iptables mode. Different hardware, models, numbers of replicas, request distributions, connection behavior, and router configurations may produce different results.

The SLOs were selected from idle uncached TTFT plus padding, and Buoyant notes that they are too high for many interactive chat uses, though potentially plausible for human-out-of-the-loop agentic flows. The reported peak goodput is workload-specific and should not be read as a general capacity multiplier. The comparison also does not independently establish Kinference's operational overhead, cost, reliability, or performance across broader production clusters. The test arms ran one after another rather than simultaneously, although Buoyant says their order alternated between conditions.

## Skill to build now
Instrument prefix reuse before changing inference capacity. Sample requests by shared system prompt, document context, or conversation continuation; then measure TTFT and SLO attainment per group across replicas. A small comparison between ordinary service routing and cache-aware routing can reveal whether prefill duplication, rather than token generation, is the constraint worth addressing.

## Reader takeaway
Kinference's benchmark makes a strong case for measuring prefix-aware routing when serving vLLM workloads with repeated context. The near-unchanged control result is just as useful as the headline improvement: the expected benefit depends on cache reuse. Benchmark real prompts and SLOs before extrapolating Buoyant's latency or capacity numbers.

Editor: [Andrey Reshetnikov](https://newruntime.com/owner-profile.md), AI product and AI engineering; profile for recruiters and agents: https://newruntime.com/owner-profile.json

## Continue in the graph

Next records for an agent, linked to their Markdown twins. Replace `.md` with `.json` for structured fields, including `next_reads`.

- topic hub: [Inference](https://newruntime.com/topics/inference.md): Explore the inference topic hub.
- collection: [Field Notes](https://newruntime.com/posts.json): Open the latest editorial analysis.
- collection: [Sources](https://newruntime.com/sources/): See the original sources this archive cites.
- whole graph: [discovery graph](https://newruntime.com/discovery-graph.json) (typed nodes and edges) and [site index](https://newruntime.com/site-index.md)