The Production Observability & Evaluation Loop

In practice, the team does not live like this: “Here we have logs and metrics, and somewhere there once a month evals”. The real cycle is different: we observe production, find strange cases, turn them into datasets, drive offline evals, put release gates, roll out canary, see the result again. This is the operational loop.

Many teams make beautiful dashboards and pairs of benchmarks, but these artifacts do not affect rollout decisions and backlogs. In this case, observability and evals exist formally but do not govern product development.

For a regular backend, request/response is important. For an LLM team, the decision point is often more important: what prompt revision was used, what model, what retriever, what tool policy, what temperature, what retrieved docs got into context. This is what we need to do with the quality of the exit.

When the cycle is built correctly, the team responds more quickly to the questions: “What exactly broke?”, “What cases should I add to the golden set?”, “This new prompt policy really improved the product?”, “Can I release a canary?”. Without this, each change turns into a long argument and a manual check.

That's an important distinction. Vol.2 describes methods and patterns well. Vol.27 describes how they communicate in production workflow: from incident to dataset, from dataset to release gate, from release gate to canary and back to telemetry.

Production traces and associated user signals are the best entry point into a real set of complex cases. This gives rise to “bad” examples: strange tools trajectories, failed retrieval chains, too expensive queries, incorrect structured outputs, unexpected refusals, and bad answers to typical questions.

The production team almost always has a lot of weak signals: alerts, negative feedback, suspicious traces, escalations from support. If they are not folded into a single case inbox, they are lost and do not become material for improving the product.

If you keep only the question and the answer, you lose half the value. For most production cases, you also need to store prompt revision, model/config, retrieval context, tool trajectory, graph version, tenant/product surface, and at least one quality label or user outcome.

Some cases are just noise, one-time oddities or out-of-scope behavior. Golden dataset should not be replenished by everyone, but by cases with high repeatability, product weight or good diagnostic value for the system.

Unsuccessful response, poor groundedness, too mild refusal and incorrect routing are product/quality incidents. Timeout model, broken queue, storage drop, inaccessible tool provider are system incidents. Both are important, but fall into different parts of the eval loop and the ops loop.

If the dataset is not replenished with production cases and is not cleaned of garbage, it quickly ceases to reflect the real product. The right dataset lives next to the system and changes with the user surface, model policy, and product scope.

One monolithic CSV with all the cases quickly becomes unreadable. It is usually useful to have at least: smoke set, regression set, hard cases set and canary set. Then each set has its own operational role.

For open-ended LLM products, it is often more useful to store not one gold output, but rubric: groundedness, completeness, style, safety, formatting, tool correctness, refusal correctness. This makes the case suitable for both offline judge eval and more flexible regression gates.

It is useful for the team to have a clear queue of cases for manual evaluation: ambiguous outputs, high-impact failures, disagreement between heuristics and LLM judge, canary regressions. It’s not just support work, but feedstock for dataset quality and release confidence.

Origin: user complaint, canary failure, production anomaly, manually crafted edge case, synthetic generation. This allows you to understand the credibility of the case and the priority of its use in gates.

Each significant change should be comparable to the baseline. This means that the eval run needs to know what is being compared: new model revision, prompt revision, graph branch, retrieval tweak, or tool policy. Otherwise, the results become beautiful, but operationally useless.

One common score is dangerous: you can improve average quality while breaking down the safety, cost, or important customer segment. Therefore, it is usually useful to build a production gate as a set of conditions: no worse for safety, no more expensive than budget, no slower than p95, no worse on a high-priority set.

Some cases are categorical: if this safety refusal or this P0 business flow is broken, the release should not be allowed. Other cases are more aggregate: quality for 500 support queries, cost for typical traffic, latency profile. These two layers of gate should not be mixed.

If heuristic, reference metric, LLM judge, and human review diverge, this is not necessarily the judge's problem. Often this is a signal that rubric is bad, the case is ambiguous or the product target is informal. Such places are especially useful to pull out in human review queue.

If today’s gate cannot be played in a week, the team loses confidence in the eval process. Therefore, it is useful to version-or dataset, candidate revision, baseline revision, judge prompt, scoring code and the result of the eval run as a separate artifact.

Even a good offline suite doesn’t cover all traffic. Therefore, it is useful to release a new revision first on a small percentage of queries or on a limited segment, observe real traces, user signals, cost and failure modes, and only then expand rollout.

For an LLM system, the error rate itself is too rough. Canary is useful for quality proxies: retry rate, clarification rate, user complaints, interrupt rate, tool explosion, cost/request, latency, human escalation rate.

Human evaluation is too expensive to test the entire flow. Therefore, production teams usually send high-value cases: canary regressions, discrepancies, P0 journeys, safety-sensitive outputs and cases for which they do not trust judge metric. This makes human review part of the release loop rather than endless manual work.

If human review lives apart from the dataset pipeline, the team loses half the benefit. It is more correct when the review result becomes the label for the case, an input to the scorecard, and the material for the next offline run. Then manual work really improves the system.

It is very useful to prescribe the rules of the road: at what offline score thresholds you can go to canary, at what canary signals you need to freeze rollout, when you need a human sign-off, when rollback automatic, and when manual. This makes the release process repetitive.

The production team makes decisions not on 20 disparate dashboards, but on one coherent picture: quality, cost, latency, reliability, human review burden, user satisfaction. Such a scorecard becomes the language of conversation between engineers, PM and ops.

The service can be technically healthy, but product degraded. And vice versa. Therefore, it is often useful to divide scorecard into two halves: system reliability (errors, saturation, queue health) and product quality (judge score, groundedness, user satisfaction, completion rate).

The same change can improve the average score while breaking enterprise-tenants, complex queries, or Spanish. Therefore, it is useful for the team to look at the scorecard by segments: tenant tier, locale, product surface, workflow type, query difficulty.

If after a new model or prompt policy, the number of cases requiring manual verification increases sharply, this is an operational cost even with a good average score. Therefore, it is useful to keep the review queue volume and time-to-resolution in the scorecard next to the cost and latency.

Sometimes the new model improves the judge score, but doubles the cost and latency. Sometimes prompt reduces hallucinations but breaks conversion. A good scorecard protects the team from one-dimensional solutions and forces you to look at the trade-offs as a whole.

The normal operating cycle is almost always distributed by roles: ML/LLM engineers lead eval methods and scorecards, product engineers deliver traces and feature context, PM helps with rubric and business priorities, ops/SRE provides telemetry and release safety, reviewers/analysts help with high-value labeling.

The team needs a regular rhythm: watch scorecard, watch new P0/P1 cases, understand changes in canary and assign actions. Without this feedback loop remains random, and knowledge about the system is held only in the minds of a couple of people.

Fast loops are alerts, traces, canary signals and fast rollback decisions. Deep loop is labeling, dataset curation, rubric refinement, and release gate enhancement. Both circuits are needed: one for service stability today, the other for system quality months from now.

Then traces cease to be just logs, and evals cease to be an academic exercise. Production provides raw materials to improve datasets. Datasets improve offline gates. Gates reduce the number of bad releases. Canary gives new information back. The cycle closes and begins to work on the quality of the product.

Production observability and evals for an LLM team are not a set of disparate best practices. This is the product operating system: how the team sees reality, how it turns failures into knowledge, how knowledge turns into dataset, how dataset drives releases, and how releases come back to production. This is where the subject becomes truly engineering.