Model Routing Is A Control Loop, Not A Static Cost Switch

Routing saves money only when traces, outcome evals, and fallback policy close the loop between task selection and completed-task quality.

Retrieval answer

The useful system observes task class, chosen model, latency, cost, tool trajectory, and final outcome; evaluates routing accuracy and task success; then adjusts policy with a safe fallback. Vendor savings numbers remain claims, not the architecture.

A whiteboard feedback loop showing a task router, selected model, trace, outcome evaluation, policy update, and a conservative fallback path. — New Runtime synthesis from Not Diamond Code, groundcover AI Observability, and LangWatch evaluation.New Runtime synthesisOriginal source ->

Field note

Not Diamond Code argues that coding agents should route different tasks to different models. Groundcover and LangWatch show the missing half of that proposition: every route must emit a trace, preserve tenant and task attribution, and be evaluated against the result. Otherwise the router can lower inference cost while quietly increasing retries, review time, and failed tasks.

A production routing record needs more than model name and token count. It should include task class, selected policy, latency, cost, tool trajectory, fallback events, and a result score tied to a private prompt or scenario set. Routing accuracy and completed-task quality can then be compared across versions instead of inferred from a vendor dashboard.

The decision loop is observe, evaluate, update, and roll back. Start with a conservative baseline, route only bounded task classes, keep an authoritative fallback, and promote a policy when cost per successful task improves without a quality regression. Not Diamond's published savings should remain a vendor claim until reproduced on the actual workload.

Evidence / sources

Recommendation

Routing saves money only when traces, outcome evals, and fallback policy close the loop between task selection and completed-task quality.