---
schema_version: "newruntime-agent-readable-v0.2"
type: "post"
stable_id: "post:ramp-adaptive-llm-routing"
slug: "ramp-adaptive-llm-routing"
title: "Ramp Teaches Its Gateway To Route By Failure, Latency, And Cost"
description: "Ramp's internal LLM gateway uses failure-aware online learning to reorder model and service-tier candidates, cutting spend without relaxing request deadlines."
retrieval_nugget: "Ramp's internal LLM gateway uses failure-aware online learning to reorder model and service-tier candidates, cutting spend without relaxing request deadlines. Ramp's internal LLM gateway processes trillions of tokens per day. That scale gave the company a useful optimization surface: the gateway can learn which provider, model, and service tier is most likely to satisfy a particular request right now."
status: "published"
published_at: "2026-08-01"
updated_at: "2026-08-01"
record_date: "2026-08-01"
date_kind: "published_at"
topics: ["inference","agent-economics","model-routing","reliability"]
source_urls: ["https://builders.ramp.com/post/thompson-sampling-model-routing"]
visuals: [{"id":"ramp-adaptive-llm-routing","kind":"editorial-diagram","role":"hero","src":"https://newruntime.com/images/posts/ramp-adaptive-llm-routing.webp","alt":"Hand-drawn gateway flow where caller preferences enter a failure filter and latency sampler, then produce a cost-aware ordered route with fallbacks.","caption":"Ramp turns model routing into a live control loop over provider failure, request deadlines, and price.","credit":"New Runtime synthesis from Ramp Builders","source_url":"https://builders.ramp.com/post/thompson-sampling-model-routing","generated_with":"gemini-3.1-flash-image","width":1600,"height":900,"legend":[{"label":"Caller intent","description":"The application supplies an ordered list of acceptable models and a latency deadline."},{"label":"Live evidence","description":"Provider failures and latency distributions are updated from recent traffic."},{"label":"Adaptive route","description":"The gateway reorders candidates by bad-outcome probability and relative cost, then falls back when needed."}]}]
telegram_message_id: 2882
telegram_url: "https://t.me/qwgai/2882"
telegram_message_ids: [2882,2883]
telegram_delivery_mode: "text_then_media"
telegram_media_url: "https://t.me/qwgai/2883"
routes: {"html":"https://newruntime.com/posts/ramp-adaptive-llm-routing/","markdown":"https://newruntime.com/posts/ramp-adaptive-llm-routing.md","json":"https://newruntime.com/posts/ramp-adaptive-llm-routing.json"}
source_format: "markdown"
---

# Ramp Teaches Its Gateway To Route By Failure, Latency, And Cost

## Retrieval answer

Ramp's internal LLM gateway uses failure-aware online learning to reorder model and service-tier candidates, cutting spend without relaxing request deadlines. Ramp's internal LLM gateway processes trillions of tokens per day. That scale gave the company a useful optimization surface: the gateway can learn which provider, model, and service tier is most likely to satisfy a particular request right now.

Ramp's internal LLM gateway processes trillions of tokens per day. That scale gave the company a useful optimization surface: the gateway can learn which provider, model, and service tier is most likely to satisfy a particular request right now.

The routing objective is not simply "choose the cheapest model." Reliability stays first, while latency depends on the caller. A background automation can wait. An interactive reranker may have a six-second deadline. Callers still express model preference through an ordered candidate list, but the gateway is allowed to reorder acceptable variants when live evidence supports it.

The system first tracks genuine provider-side failures with an exponentially weighted moving average. It does not penalize a model for an invalid caller request. It then uses Thompson Sampling to learn a point-in-time latency distribution for each context slice. For a candidate route, a bad outcome means either a provider failure or a successful response that misses the caller's deadline.

That probability is combined with relative cost. The gateway sorts the candidates, tries the strongest current route, and keeps the remaining list as fallbacks. The online learner continues exploring enough to notice when a degraded provider recovers or when a discounted service tier becomes fast enough for a deadline-sensitive job.

Ramp first applied the strategy to an LLM reranker. During elevated Gemini rate limiting, traffic shifted to GPT-5.4 Nano, but often to the half-price flex tier because its observed latency remained below the six-second budget. A broader experiment reported 26.3% cost savings with a 0.09 percentage-point reduction in error rate. A streaming version later cut spend for Ramp Inspect by 30%.

For New Runtime, this is a stronger model-router contract than a static fallback list. The caller owns acceptable quality and the deadline. The gateway owns fresh operational evidence. Price is optimized only inside those boundaries.