Modal describes a production workload in which requests contained roughly 100,000 input tokens and generated roughly 500 output tokens, a 200:1 input-to-output ratio. It also reports that one service handled hundreds of billions of tokens per day and trillions in aggregate. The company says its optimizations increased per-replica performance by 2.8× per user and 5.6× across users. Those are operator-reported results for its chosen model, hardware, engine, and workload, not a general benchmark for all agent systems.
The mechanism is cumulative agent sessions: each turn carries prior conversation, tool results, and model output forward, so much of the new request overlaps with already processed context. KV caching avoids redoing that computation, but turns state placement, HBM capacity and bandwidth, scheduling, and concurrency into first-order runtime concerns. Modal says it used tensor parallelism and custom DFlash speculative decoding while patching SGLang to lift identified bottlenecks.
This matters now because coding agents are extending sessions through iterative tool use, making long-context reuse a routine production workload rather than an edge case. The immediate implication is that agent-platform teams should optimize and observe cache reuse, memory pressure, time-to-first-token, decode rate, and concurrency together rather than treating throughput as a standalone metric. The evidence does not establish that Modal’s gains transfer to other models, hardware fleets, traffic mixes, or inference engines. Watch whether independent operators publish comparable long-session traces, cache-hit data, and cost-per-completed-agent-task results.