Field noteNov 19, 20265 min read1 source

Switching LLMs mid-conversation is a ski-rental problem

In one minute

Edgee’s account of Self-Learning Routing treats model selection in coding agents as two linked decisions: which capability tier a turn needs, and when a session should switch models without wasting its prompt cache. The design combines a cost threshold with replay of an organization’s own traffic, offering builders a concrete framework for evaluating routing trade-offs.

What changed: It frames mid-session model switching for coding agents as a ski-rental optimization, balancing cache rebuild costs against cheaper-model savings.

What happened

Edgee Research Engineer Khaled Maâmra’s November 10, 2026 article describes routing for long coding-agent sessions. A coding agent resends conversation history with each request, and providers can cache that history. Edgee says changing models loses the warm cache: the destination model must read the conversation cold and write its own cache. That means routing every apparently easy turn to a cheaper model can cost more in cache reconstruction than it saves in inference.

The first part of the system is a capability classifier. It scores the latest user message from zero to one and maps it to low, medium, high, or max tiers, from formatting and typos through multi-file changes to architectural, security-critical, or open-ended work. Requests inside one user turn remain on the model selected at that turn’s start. A request requiring a stronger tier upgrades immediately when confidence is at least 0.75; at 0.5 to 0.75 it needs two consecutive upgrade signals, and below 0.5 it stays put. Edgee frames upgrades as quality constraints rather than cost decisions.

For a downgrade or same-tier model change, the gateway applies a ski-rental-style rule. It estimates the one-off “buy” cost of moving to a candidate with a cold cache and the “rent” represented by per-turn savings forgone by remaining on the current model. It switches when accumulated rent reaches 1.5 times the buy cost. Edgee says the textbook threshold is 1, but uses 1.5 to avoid a downgrade that a hard turn immediately reverses; under its simplified model, this has a 2.5 worst-case competitive ratio. The gateway halves its accumulated total only when already on the target model, imposes a four-turn cooldown after a paid switch, and treats cache expiry or compaction below 60% of the previous conversation length as a free switching opportunity. Session state lasts six hours.

Edgee says organizations that enable classification and learning can run weekly replays over up to 30 days and two million requests. After an administrator accepts a versioned consent notice, the gateway labels user-started turns using Jev with zero data retention or OpenAI Decisions on Edgee’s credential; the latest user message is limited to 4,000 characters. Edgee says it retains labels, not message text, and labeling does not delay model responses. The replay groups traffic by agent, tests up to 81 tier maps per agent, requires candidate models to have at least 200,000-token contexts, and compares policies with a dynamic-programming hindsight optimum. A recommendation requires at least a half-point improvement in estimated billed spend without more than a 0.2-point increase in below-label requests.

What the archive adds

We have not covered this before; it opens a new thread.

Trend signal

This does not yet fit a pattern New Runtime tracks. The article concerns a specific routing design and replay process; it does not by itself establish a broader shift in how production agents are built or evaluated.

Why it matters

The design makes a frequently hidden routing cost explicit: a model’s apparent per-token price is not the full cost of using it mid-session. For teams building coding agents, cache continuity can determine whether per-turn model selection saves money at all. The article’s useful distinction is between capability upgrades, which protect quality, and cost-driven switches, which must earn back their context rewrite.

The replay is also a more grounded alternative to tuning a routing threshold from intuition alone. Comparing a policy against an organization’s own traffic can show whether easy turns arrive in long stretches or isolated blips. But the recommendation remains conditional on classifier labels, catalog ratings, prices, and modeled cache behavior. Teams should track actual spend and quality outcomes rather than treat the ski-rental analogy as a whole-session guarantee.

What happens next

  • Organizations using Edgee’s learning feature will see recommendations shaped by their own labeled session traffic rather than a universal routing threshold. (6 months, 55%). Watch for: published customer results reporting replay coverage, realized savings, and quality outcomes.

Limits and uncertainty

The article is Edgee’s description of its own product and algorithm, not an independent assessment. Its replay reprices past requests using catalog prices and cannot know how a different model would have answered; quality is constrained structurally through tier labels rather than measured by rerunning equivalent tasks. The example session comes from Edgee’s own traffic. Results will depend on labeling coverage, classifier accuracy, session mix, cache behavior, and whether model prices or capabilities change. The article’s simplified competitive analysis assumes stable prices and a stable candidate within one stretch; it does not guarantee optimal cost over a full session.

Skill to build now

For a router that can change models inside a session, instrument two quantities separately: the extra cost of rebuilding the conversation on the candidate model, and the per-turn saving if that model stays appropriate. Replay candidate switching thresholds over completed sessions while enforcing a minimum capability tier for each request. Include cache expiry and compaction in the replay; both can change the cost of switching sharply. Compare estimated savings with observed spend before making a setting permanent.

Reader takeaway

Model routing inside a coding session is not just a classification problem. A cheaper model only saves money when the expected run of suitable turns can repay the cost of rebuilding context—and when the team can verify that the route still meets its capability requirement.

Evidence / sources

  1. 1edgee.ai/blog/posts/switching-llms-mid-conversation-is-a-ski-rental-problem
All Field Notes

Next reads

Read next

Open the graph
  1. 01topicCoding AgentsExplore the coding-agents topic hub.
  2. 02topicClaudeExplore the claude topic hub.
  3. 03topicClaude CodeExplore the claude-code topic hub.
  4. 04archiveField NotesOpen the latest editorial analysis.
  5. 05sourcesSourcesSee the original sources this archive cites.
Who read this page?Machine requests, hidden until opened

Loading the privacy-safe route aggregate...

Open the JSON contract