A self-hosted LLM endpoint can return 200 OK while the product is already unusable. Akamai’s Du’An Lightfoot demonstrates the failure on a vLLM deployment: traffic rises, aggregate throughput eventually flattens, and tail latency keeps climbing without a conventional server error.
The mismatch starts below HTTP. A web request is mostly stateless and has a roughly stable cost. An inference request holds state in GPU memory, grows with context, and competes with every other active sequence for a fixed memory-bandwidth and KV-cache budget.
Inference also has two different phases. Prefill processes the prompt in parallel and drives time to first token. Decode generates sequentially and repeatedly reads model weights plus the growing cache from memory. CPU and network dashboards do not explain either bottleneck well.
The batching cliff
Continuous batching is still a major efficiency gain. In Akamai’s example, one request produced about 34 tokens per second, while 64 concurrent requests produced roughly 1,041 aggregate tokens per second. The GPU did more useful work per model-weight read.
But batching has a cliff. Once new work arrives faster than the engine can prefill it, throughput stops increasing and the queue grows. Each user receives fewer tokens per second, then P95 latency rises even though the endpoint stays alive.
The memory failure looks similar. The test GPU had 20 GB of VRAM, with about 14 GB occupied by model weights and roughly 38,800 tokens of KV-cache capacity left. Thirty-two requests carrying 4,000-token prompts demanded 128,000 cache tokens. The result was not a crash. Time to first token rose toward 55 seconds while the error rate stayed at zero.
Put the waiting room outside the engine
The corrective pattern is bounded admission. Akamai capped vLLM with --max-num-seqs=8 and --max-model-len=8192, then kept excess work in a gateway queue outside the model server.
An external queue can expose depth, reject quickly, prioritize, rate-limit, or shed work. An internal unbounded queue can only make requests wait invisibly. With the same hardware, admitted work stayed close to baseline latency and the remainder waited predictably.
Autoscaling still matters, but it reacts to measurements after load arrives. Backpressure is the mechanism that keeps the first burst from collapsing the runtime before another replica becomes useful.