{ "type": "new-runtime-knowledge-guide", "version": 1, "generated_at": "2026-07-23", "volume": 27, "slug": "production-observability-eval-loop", "title": "The Production Observability & Evaluation Loop", "description": "This volume does not repeat the general review of evals. It shows the operational cycle of the team: how incident cases, datasets, offline evals, human review, canary gates and rollout/rollback decisions are born from production traces and user signals. It’s not “what evals are,” it’s “how the team lives with evals and observability every day.”", "track": { "slug": "evaluation-and-improvement", "title": "Evaluation & Improvement", "route": "https://newruntime.com/learn/tracks/evaluation-and-improvement/", "position": 3 }, "counts": { "sections": 7, "cards": 35 }, "routes": { "html": "https://newruntime.com/learn/production-observability-eval-loop/", "json": "https://newruntime.com/learn/production-observability-eval-loop.json" }, "sections": [ { "number": "01", "title": "Main Framework", "intro": "", "cards": [ { "number": "01", "title": "Observability and evals are one cycle, not two separate themes.", "tag": "Frame", "description": "In practice, the team does not live like this: “Here we have logs and metrics, and somewhere there once a month evals”. The real cycle is different: we observe production, find strange cases, turn them into datasets, drive offline evals, put release gates, roll out canary, see the result again. This is the operational loop.", "blocks": [ { "kind": "code", "label": "Diagram", "language": "text", "value": "PRODUCTION LOOP\n\nproduction traces / user signals\n ↓\nincident cases / edge cases\n ↓\ncurated dataset\n ↓\noffline eval suite\n ↓\ncanary / release gate\n ↓\ndeploy\n ↓\nobserve again" } ] }, { "number": "02", "title": "Without a closed loop, evals become a presentation, not a tool.", "tag": "Frame", "description": "Many teams make beautiful dashboards and pairs of benchmarks, but these artifacts do not affect rollout decisions and backlogs. In this case, observability and evals exist formally but do not govern product development.", "blocks": [ { "kind": "list", "label": "red flags", "items": [ "Metrics are available, but incidents do not enter datasets.", "eval suite is not involved in the release", "People don’t know which products really break." ] } ] }, { "number": "03", "title": "The main unit of analysis is not only a request, but also a decision.", "tag": "Frame", "description": "For a regular backend, request/response is important. For an LLM team, the decision point is often more important: what prompt revision was used, what model, what retriever, what tool policy, what temperature, what retrieved docs got into context. This is what we need to do with the quality of the exit.", "blocks": [ { "kind": "code", "label": "useful mental model", "language": "text", "value": "request\n→ config / model / prompt revision\n→ graph trajectory\n→ output\n→ user signal / human judgment" } ] }, { "number": "04", "title": "Production eval loop is needed not only for quality but also for team speed.", "tag": "Frame", "description": "When the cycle is built correctly, the team responds more quickly to the questions: “What exactly broke?”, “What cases should I add to the golden set?”, “This new prompt policy really improved the product?”, “Can I release a canary?”. Without this, each change turns into a long argument and a manual check.", "blocks": [ { "kind": "list", "label": "speeding", "items": [ "debugging", "prioritization", "release decisions", "rollback confidence" ] } ] }, { "number": "05", "title": "Vol.2 was a theme map, this volume was a team process map.", "tag": "Frame", "description": "That's an important distinction. Vol.2 describes methods and patterns well. Vol.27 describes how they communicate in production workflow: from incident to dataset, from dataset to release gate, from release gate to canary and back to telemetry.", "blocks": [ { "kind": "list", "label": "short-range", "items": [ "Vol.2 = survey atlas", "Vol.27 = operating model" ] } ] } ] }, { "number": "02", "title": "From Traces to Cases", "intro": "", "cards": [ { "number": "06", "title": "Traces are needed not only for debugging, but also as raw materials for datasets.", "tag": "Tracing", "description": "Production traces and associated user signals are the best entry point into a real set of complex cases. This gives rise to “bad” examples: strange tools trajectories, failed retrieval chains, too expensive queries, incorrect structured outputs, unexpected refusals, and bad answers to typical questions.", "blocks": [ { "kind": "code", "label": "Diagram", "language": "text", "value": "TRACE → CASE\n\ntrace\n + prompt revision\n + model\n + retrieved docs\n + output\n + user signal / thumbs down / escalation\n =\ncandidate eval case" } ] }, { "number": "07", "title": "Incidents are better collected in “case inbox”, rather than leaving them in allertas and Slack.", "tag": "Tracing", "description": "The production team almost always has a lot of weak signals: alerts, negative feedback, suspicious traces, escalations from support. If they are not folded into a single case inbox, they are lost and do not become material for improving the product.", "blocks": [ { "kind": "list", "label": "what could be the source of the case", "items": [ "thumbs down", "support escalation", "quality anomaly", "expensive trace" ] } ] }, { "number": "08", "title": "Case capture should retain not only input/output, but also execution context.", "tag": "Tracing", "description": "If you keep only the question and the answer, you lose half the value. For most production cases, you also need to store prompt revision, model/config, retrieval context, tool trajectory, graph version, tenant/product surface, and at least one quality label or user outcome.", "blocks": [ { "kind": "code", "label": "case-structure", "language": "text", "value": "case_id\nquery\noutput\nexpected_behavior\nprompt_revision\nmodel_name\nthread_id / run_id\nretrieved_context\ntool_trace\nuser_signal\nhuman_label?" } ] }, { "number": "09", "title": "Triage model: Not every bad trace should go to golden set", "tag": "Tracing", "description": "Some cases are just noise, one-time oddities or out-of-scope behavior. Golden dataset should not be replenished by everyone, but by cases with high repeatability, product weight or good diagnostic value for the system.", "blocks": [ { "kind": "list", "label": "triage", "items": [ "repeat", "important user journey", "bright regression signal", "A single noise can be left in the backlog" ] } ] }, { "number": "10", "title": "Distinguish between product incidents and system incidents", "tag": "Tracing", "description": "Unsuccessful response, poor groundedness, too mild refusal and incorrect routing are product/quality incidents. Timeout model, broken queue, storage drop, inaccessible tool provider are system incidents. Both are important, but fall into different parts of the eval loop and the ops loop.", "blocks": [] } ] }, { "number": "03", "title": "Datasets and Labeling Loop", "intro": "", "cards": [ { "number": "11", "title": "Golden dataset should be a living, not an archive of “once collected examples”", "tag": "Datasets", "description": "If the dataset is not replenished with production cases and is not cleaned of garbage, it quickly ceases to reflect the real product. The right dataset lives next to the system and changes with the user surface, model policy, and product scope.", "blocks": [ { "kind": "code", "label": "dateset", "language": "text", "value": "incident found\n→ candidate case added\n→ triage\n→ labeling\n→ enters dataset\n→ participates in release gate\n→ stays until deprecated or replaced" } ] }, { "number": "12", "title": "Dataset is useful to divide into several layers, rather than keeping everything in one heap.", "tag": "Datasets", "description": "One monolithic CSV with all the cases quickly becomes unreadable. It is usually useful to have at least: smoke set, regression set, hard cases set and canary set. Then each set has its own operational role.", "blocks": [ { "kind": "list", "label": "typical", "items": [ "smoke", "regression", "hard cases", "canary monitor set" ] } ] }, { "number": "13", "title": "Labeling needs not only the right answer, but also rubric-level labels.", "tag": "Datasets", "description": "For open-ended LLM products, it is often more useful to store not one gold output, but rubric: groundedness, completeness, style, safety, formatting, tool correctness, refusal correctness. This makes the case suitable for both offline judge eval and more flexible regression gates.", "blocks": [ { "kind": "code", "label": "example of label schema", "language": "text", "value": "groundedness: 0|1\ncomplete: 0|1\nstyle_ok: 0|1\nsafety_ok: 0|1\nshould_refuse: 0|1\nexpected_tool_use: yes|no\npriority: P0|P1|P2" } ] }, { "number": "14", "title": "Human review queue is part of a dataset pipeline, not a side quest", "tag": "Datasets", "description": "It is useful for the team to have a clear queue of cases for manual evaluation: ambiguous outputs, high-impact failures, disagreement between heuristics and LLM judge, canary regressions. It’s not just support work, but feedstock for dataset quality and release confidence.", "blocks": [ { "kind": "list", "label": "Often sent to human review", "items": [ "high-value customers", "safety-sensitive outputs", "judge disagreement", "new feature regressions" ] } ] }, { "number": "15", "title": "It is very important to preserve provenance: where did the case come from", "tag": "Datasets", "description": "Origin: user complaint, canary failure, production anomaly, manually crafted edge case, synthetic generation. This allows you to understand the credibility of the case and the priority of its use in gates.", "blocks": [ { "kind": "code", "label": "Useful origin labels", "language": "text", "value": "origin = \"prod_incident\" | \"support_escalation\" | \"synthetic\" | \"manual\" | \"canary_failure\"\n\nOrigin often influences priority and who owns the case." } ] } ] }, { "number": "04", "title": "Offline Evals as Release Gate", "intro": "", "cards": [ { "number": "16", "title": "Offline eval suite should be linked to change unit: prompt, model, graph, tool policy", "tag": "Offline Evals", "description": "Each significant change should be comparable to the baseline. This means that the eval run needs to know what is being compared: new model revision, prompt revision, graph branch, retrieval tweak, or tool policy. Otherwise, the results become beautiful, but operationally useless.", "blocks": [ { "kind": "code", "label": "Diagram", "language": "text", "value": "CHANGE-ALIGNED EVAL\n\nbaseline revision\n vs\ncandidate revision\n\non the same eval set\nwith the same scoring rules" } ] }, { "number": "17", "title": "Gate is better to build on a scorecard, not on a single aggregate metric", "tag": "Offline Evals", "description": "One common score is dangerous: you can improve average quality while breaking down the safety, cost, or important customer segment. Therefore, it is usually useful to build a production gate as a set of conditions: no worse for safety, no more expensive than budget, no slower than p95, no worse on a high-priority set.", "blocks": [ { "kind": "list", "label": "condition", "items": [ "groundedness not worse", "cost delta within limit", "P0 regression count = 0", "latency within budget" ] } ] }, { "number": "18", "title": "It is very useful to have “must-pass” cases and “statistical” cases separately.", "tag": "Offline Evals", "description": "Some cases are categorical: if this safety refusal or this P0 business flow is broken, the release should not be allowed. Other cases are more aggregate: quality for 500 support queries, cost for typical traffic, latency profile. These two layers of gate should not be mixed.", "blocks": [ { "kind": "code", "label": "double-gate", "language": "text", "value": "must-pass layer:\n P0 business cases\n safety/refusal cases\n schema correctness cases\n\nstatistical layer:\n average judge score\n cost/request\n latency p95\n tool count distribution" } ] }, { "number": "19", "title": "Judge discrepancy is a signal, not just noise", "tag": "Offline Evals", "description": "If heuristic, reference metric, LLM judge, and human review diverge, this is not necessarily the judge's problem. Often this is a signal that rubric is bad, the case is ambiguous or the product target is informal. Such places are especially useful to pull out in human review queue.", "blocks": [ { "kind": "list", "label": "typical causes of disagreement", "items": [ "rubric", "multiple valid answers", "safety vs helpfulness tradeoff" ] } ] }, { "number": "20", "title": "Offline suite must be reproducible and versioned", "tag": "Offline Evals", "description": "If today’s gate cannot be played in a week, the team loses confidence in the eval process. Therefore, it is useful to version-or dataset, candidate revision, baseline revision, judge prompt, scoring code and the result of the eval run as a separate artifact.", "blocks": [ { "kind": "code", "label": "version", "language": "text", "value": "dataset_version\nbaseline_revision\ncandidate_revision\njudge_prompt_version\nscoring_code_version\neval_run_id" } ] } ] }, { "number": "05", "title": "Canary, Human Review and Release Decisions", "intro": "", "cards": [ { "number": "21", "title": "After offline gate, you need an online canary, not “all at once”.", "tag": "Release", "description": "Even a good offline suite doesn’t cover all traffic. Therefore, it is useful to release a new revision first on a small percentage of queries or on a limited segment, observe real traces, user signals, cost and failure modes, and only then expand rollout.", "blocks": [ { "kind": "code", "label": "Diagram", "language": "text", "value": "RELEASE FLOW\n\noffline eval passes\n ↓\nsmall canary\n ↓\nobserve live behavior\n ↓\nexpand traffic or rollback" } ] }, { "number": "22", "title": "Canary monitoring should look not only at errors, but also at quality proxies.", "tag": "Release", "description": "For an LLM system, the error rate itself is too rough. Canary is useful for quality proxies: retry rate, clarification rate, user complaints, interrupt rate, tool explosion, cost/request, latency, human escalation rate.", "blocks": [ { "kind": "list", "label": "very useful canary signals", "items": [ "cost delta", "human escalation rate", "negative user feedback", "tool count inflation" ] } ] }, { "number": "23", "title": "Human review is useful to do point-by-point rather than trying to “check everything manually.”", "tag": "Human Review", "description": "Human evaluation is too expensive to test the entire flow. Therefore, production teams usually send high-value cases: canary regressions, discrepancies, P0 journeys, safety-sensitive outputs and cases for which they do not trust judge metric. This makes human review part of the release loop rather than endless manual work.", "blocks": [ { "kind": "code", "label": "entry point in human review", "language": "text", "value": "candidate change risky\nOR judge disagreement high\nOR canary suspicious\nOR support escalation severe\n→ route to human review queue" } ] }, { "number": "24", "title": "Review queue must be able to return labels back to datasets and scorecards", "tag": "Human Review", "description": "If human review lives apart from the dataset pipeline, the team loses half the benefit. It is more correct when the review result becomes the label for the case, an input to the scorecard, and the material for the next offline run. Then manual work really improves the system.", "blocks": [ { "kind": "list", "label": "The result of the review should be", "items": [ "dataset label store", "release scorecard", "incident backlog" ] } ] }, { "number": "25", "title": "Rollout decisions should be formalized rather than “feeling”", "tag": "Release", "description": "It is very useful to prescribe the rules of the road: at what offline score thresholds you can go to canary, at what canary signals you need to freeze rollout, when you need a human sign-off, when rollback automatic, and when manual. This makes the release process repetitive.", "blocks": [ { "kind": "code", "label": "case-case", "language": "text", "value": "offline gate passes\nAND no P0 regressions\nAND cost delta < 10%\n→ canary 5%\n\nif canary negative feedback +30%\nOR latency p95 > budget\n→ freeze / rollback" } ] } ] }, { "number": "06", "title": "Scorecards and Product Health", "intro": "", "cards": [ { "number": "26", "title": "The team needs a single scorecard, not a scattering of unrelated schedules.", "tag": "Scorecards", "description": "The production team makes decisions not on 20 disparate dashboards, but on one coherent picture: quality, cost, latency, reliability, human review burden, user satisfaction. Such a scorecard becomes the language of conversation between engineers, PM and ops.", "blocks": [ { "kind": "code", "label": "Diagram", "language": "text", "value": "PRODUCT SCORECARD\n\nquality\ncost\nlatency\nreliability\nhuman review load\nuser satisfaction proxy\n\nOne layer of numbers for weekly review and release decisions" } ] }, { "number": "27", "title": "Distinguish “product health” from “system health”", "tag": "Scorecards", "description": "The service can be technically healthy, but product degraded. And vice versa. Therefore, it is often useful to divide scorecard into two halves: system reliability (errors, saturation, queue health) and product quality (judge score, groundedness, user satisfaction, completion rate).", "blocks": [ { "kind": "list", "label": "segmentation", "items": [ "tenant tier", "locale", "surface / feature", "query class" ] }, { "kind": "list", "label": "look out", "items": [ "review cases/day", "median review time", "P0 unresolved count" ] }, { "kind": "code", "label": "tradeoff", "language": "text", "value": "quality up\ncost up a lot\nlatency up\nhuman review unchanged\n\nThis is not a “clearly good release” but a product decision." } ] }, { "number": "28", "title": "Per-segment scorecards are needed because the average hides pain.", "tag": "Scorecards", "description": "The same change can improve the average score while breaking enterprise-tenants, complex queries, or Spanish. Therefore, it is useful for the team to look at the scorecard by segments: tenant tier, locale, product surface, workflow type, query difficulty.", "blocks": [ { "kind": "list", "label": "segmentation", "items": [ "tenant tier", "locale", "surface / feature", "query class" ] } ] }, { "number": "29", "title": "Human review load is also a system quality metric.", "tag": "Scorecards", "description": "If after a new model or prompt policy, the number of cases requiring manual verification increases sharply, this is an operational cost even with a good average score. Therefore, it is useful to keep the review queue volume and time-to-resolution in the scorecard next to the cost and latency.", "blocks": [ { "kind": "list", "label": "look out", "items": [ "review cases/day", "median review time", "P0 unresolved count" ] } ] }, { "number": "30", "title": "A good scorecard helps you say no to beautiful but harmful improvements. n", "tag": "Scorecards", "description": "Sometimes the new model improves the judge score, but doubles the cost and latency. Sometimes prompt reduces hallucinations but breaks conversion. A good scorecard protects the team from one-dimensional solutions and forces you to look at the trade-offs as a whole.", "blocks": [ { "kind": "code", "label": "tradeoff", "language": "text", "value": "quality up\ncost up a lot\nlatency up\nhuman review unchanged\n\nThis is not a “clearly good release” but a product decision." } ] } ] }, { "number": "07", "title": "Team Roles and Master Scheme", "intro": "", "cards": [ { "number": "31", "title": "Production eval loop has ownership: it’s not a “nobody’s theme”", "tag": "Ops", "description": "The normal operating cycle is almost always distributed by roles: ML/LLM engineers lead eval methods and scorecards, product engineers deliver traces and feature context, PM helps with rubric and business priorities, ops/SRE provides telemetry and release safety, reviewers/analysts help with high-value labeling.", "blocks": [ { "kind": "list", "label": "proprietary", "items": [ "engineers → runtime + traces + eval infra", "PM → rubric + priority", "reviewers → labels + edge cases", "ops → alerts + release safety" ] } ] }, { "number": "32", "title": "Weekly review on scorecard is more useful than chaotic analysis of incidents", "tag": "Ops", "description": "The team needs a regular rhythm: watch scorecard, watch new P0/P1 cases, understand changes in canary and assign actions. Without this feedback loop remains random, and knowledge about the system is held only in the minds of a couple of people.", "blocks": [ { "kind": "code", "label": "rhythm", "language": "text", "value": "weekly:\n scorecard review\n new incident cases\n canary changes\n dataset additions\n next release gate decisions" } ] }, { "number": "33", "title": "A good team has a fast loop and a deep loop.", "tag": "Ops", "description": "Fast loops are alerts, traces, canary signals and fast rollback decisions. Deep loop is labeling, dataset curation, rubric refinement, and release gate enhancement. Both circuits are needed: one for service stability today, the other for system quality months from now.", "blocks": [ { "kind": "list", "label": "two-line", "items": [ "fast loop = operational control", "Deep loop = Systemic improvement" ] } ] }, { "number": "34", "title": "If the cycle is built correctly, observability and evals reinforce each other.", "tag": "Ops", "description": "Then traces cease to be just logs, and evals cease to be an academic exercise. Production provides raw materials to improve datasets. Datasets improve offline gates. Gates reduce the number of bad releases. Canary gives new information back. The cycle closes and begins to work on the quality of the product.", "blocks": [ { "kind": "code", "label": "Diagram", "language": "text", "value": "SELF-REINFORCING LOOP\n\ntelemetry\n→ cases\n→ datasets\n→ eval gates\n→ safer releases\n→ better production\n→ better telemetry" } ] }, { "number": "35", "title": "The main idea of the volume", "tag": "Frame", "description": "Production observability and evals for an LLM team are not a set of disparate best practices. This is the product operating system: how the team sees reality, how it turns failures into knowledge, how knowledge turns into dataset, how dataset drives releases, and how releases come back to production. This is where the subject becomes truly engineering.", "blocks": [ { "kind": "code", "label": "Diagram", "language": "text", "value": "ITORMULE\n\nproduction signals\n + case inbox\n + labeled datasets\n + reproducible offline evals\n + canary gates\n + human review\n + scorecards\n =\nReal production loop for the LLM team" } ] } ] } ] }