Ramp’s internal LLM gateway processes trillions of tokens per day. That scale gave the company a useful optimization surface: the gateway can learn which provider, model, and service tier is most likely to satisfy a particular request right now.
The routing objective is not simply “choose the cheapest model.” Reliability stays first, while latency depends on the caller. A background automation can wait. An interactive reranker may have a six-second deadline. Callers still express model preference through an ordered candidate list, but the gateway is allowed to reorder acceptable variants when live evidence supports it.
The system first tracks genuine provider-side failures with an exponentially weighted moving average. It does not penalize a model for an invalid caller request. It then uses Thompson Sampling to learn a point-in-time latency distribution for each context slice. For a candidate route, a bad outcome means either a provider failure or a successful response that misses the caller’s deadline.
That probability is combined with relative cost. The gateway sorts the candidates, tries the strongest current route, and keeps the remaining list as fallbacks. The online learner continues exploring enough to notice when a degraded provider recovers or when a discounted service tier becomes fast enough for a deadline-sensitive job.
Ramp first applied the strategy to an LLM reranker. During elevated Gemini rate limiting, traffic shifted to GPT-5.4 Nano, but often to the half-price flex tier because its observed latency remained below the six-second budget. A broader experiment reported 26.3% cost savings with a 0.09 percentage-point reduction in error rate. A streaming version later cut spend for Ramp Inspect by 30%.
For New Runtime, this is a stronger model-router contract than a static fallback list. The caller owns acceptable quality and the deadline. The gateway owns fresh operational evidence. Price is optimized only inside those boundaries.