Its representative workload had about 100,000 input tokens and 500 output tokens per request, while repeated session history made KV-cache management central. Modal reports 2.8× higher per-user replica performance and 5.6× higher aggregate performance across users after serving optimizations built on SGLang.
Raw SignalSep 23, 20261 min read1 source
How to serve trillions of tokens for trillion
In one minute
Modal published a production engineering account of serving coding-agent workloads with a roughly trillion-parameter Kimi model.