{ "type": "new-runtime-knowledge-guide", "version": 1, "generated_at": "2026-07-23", "volume": 29, "slug": "data-flywheel-reference", "title": "The Data Flywheel: From Production Cases to Release Gates", "description": "This volume is about the main engine of a high-quality LLM team: how production traces and user pain turn into curated cases, label taxonomy, human review queue, regression set, canary set and decisions about what to fix prompt, what to retriever, and what to train. So it's not \"where to get the dataset,\" but how the team grows it out of their own work and turns the data into a system improvement cycle.", "track": { "slug": "evaluation-and-improvement", "title": "Evaluation & Improvement", "route": "https://newruntime.com/learn/tracks/evaluation-and-improvement/", "position": 4 }, "counts": { "sections": 7, "cards": 35 }, "routes": { "html": "https://newruntime.com/learn/data-flywheel-reference/", "json": "https://newruntime.com/learn/data-flywheel-reference.json" }, "sections": [ { "number": "01", "title": "Main Framework", "intro": "", "cards": [ { "number": "01", "title": "Data flywheel is a cycle where production becomes the next generation of quality.", "tag": "Frame", "description": "The LLM team grows not only from the best model, but also from the best own data loop. Custom cases, incidents, failed tool runs, controversial answers, and edge cases should not die in logs, but instead become examples, labels, regression sets, and training candidates. When this cycle works, quality ceases to depend on random heroic fixes.", "blocks": [ { "kind": "code", "label": "Diagram", "language": "text", "value": "DATA FLYWHEEL\n\nproduction traces / user feedback\n ↓\ncase triage\n ↓\nlabels / taxonomy\n ↓\ndataset layers\n ↓\neval / release gates\n ↓\nfixes / training / prompts\n ↓\ndeploy\n ↓\nobserve again" } ] }, { "number": "02", "title": "The main unit of flywheel is not raw trace, but case", "tag": "Frame", "description": "Raw logs themselves are almost useless for product development. They need to be turned into a case: an understandable artifact with input, context, system response, user signal, problem and expectation. Only such an object can be sent to a review queue, put in a regression set, and used to compare changes.", "blocks": [ { "kind": "code", "label": "case =", "language": "sql", "value": "input\ncontext snapshot\nsystem output\nmetadata\nlabel candidates\ndesired outcome\ndecision: where the case will go next" } ] }, { "number": "03", "title": "Flywheel’s goal is not to “gather more examples” but to improve team decision quality.", "tag": "Frame", "description": "A good flywheel helps you decide what to fix and what to do. Sometimes the problem is prompt packing, sometimes retrieval, sometimes tool schema, sometimes label ambiguity, and sometimes the model itself. If data doesn't help make those decisions, it's an archive, not flywheel.", "blocks": [] }, { "number": "04", "title": "One production case can work simultaneously on evals, prompts, docs and training.", "tag": "Frame", "description": "Strong cases are multiplicative. One failure found can be a regression example, a few-shot example, a test for grader, an example for annotation guide, and a seed for synthetic expansion. Therefore, it is especially important not to lose lineage and not to break the link between the original incident and its derivative artifacts.", "blocks": [] }, { "number": "05", "title": "If the cases do not go into curated sets, the team will step on the same bugs over and over again.", "tag": "Frame", "description": "Many teams know how to find failures, but they do not know how to preserve lessons. As a result, the incident is discussed, repaired and forgotten, and a month later it returns in a different form. Regression set and canary set are needed to make the problem stop being a “memories in Slack” and become a permanent gateway.", "blocks": [] } ] }, { "number": "02", "title": "Intake From Production", "intro": "", "cards": [ { "number": "06", "title": "Raw case sources are wider than thumbs up/down", "tag": "Intake", "description": "Production cases don’t just come from explicit feedback. Sources include escalation to human, user retries, abandonment, unusually long tool loops, low citation coverage, manual QA, support tickets and canary incidents. The richer the intake, the less the team depends on rare manual complaints.", "blocks": [ { "kind": "list", "label": "typical", "items": [ "negative feedback", "retry / reformulation", "human takeover", "tool failure trajectories" ] } ] }, { "number": "07", "title": "A good case record should preserve the input, context, output and environment of the solution.", "tag": "Intake", "description": "For LLM analysis, it is not enough to save only a question and an answer. You need model revision, prompt/template version, retrieved docs, tools, latency, token breakdown, user segment and quality signal. Otherwise, the team will see the failure later, but will not understand in what configuration it occurred. Case record is a forensic snapshot solution, not just an example of text.", "blocks": [ { "kind": "code", "label": "minimum fields", "language": "text", "value": "case_id\ntask / route / segment\nuser input\npacked context snapshot\noutput\nprompt/model/tool versions\nretrieval ids\nuser signal / incident source\ntimestamp\nprivacy status" } ] }, { "number": "08", "title": "Before sending cases to review, they must be deduplicated and clustered.", "tag": "Intake", "description": "If the same failure has occurred a thousand times, the team should not manually read a thousand similar examples. Proper intake clusters using similar error patterns and creates representative exemplars. This helps to separate the “mass bug” from the “single odd case” and saves reviewers time.", "blocks": [] }, { "number": "09", "title": "You need to save slices and segments, otherwise the average metric hides the pain of specific groups.", "tag": "Intake", "description": "It is important to know for which product, language, user tier, domain, region, workflow or model route the case occurred. Without slicing, flywheel slips into overall average quality and does not see that, for example, enterprise users suffer from some failures, and the self-serve segment from others. The correct case intake immediately writes slice metadata.", "blocks": [] }, { "number": "10", "title": "Case priority is defined by the formula severity × frequency × strategic importance.", "tag": "Intake", "description": "Not every case is equally important. Some rare examples are scary because of safety risk, others are massive but tolerant, others are rare but critical for a VIP product. Therefore, triage must take into account the severity, repeatability, and value of the affected flow. It's better than naively sorting just by number of likes or just by subjective drama.", "blocks": [] } ] }, { "number": "03", "title": "Label Taxonomy", "intro": "", "cards": [ { "number": "11", "title": "A good taxonomy answers the question “what exactly happened and why it matters,” not just “good/bad.”", "tag": "Taxonomy", "description": "Binary quality assessment is convenient for a dashboard, but too poor for improvements. The team needs to understand the type of failure: hallucination, citation miss, bad refusal, wrong tool choice, truncation, formatting failure, latency timeout, policy violation, misunderstanding intention. Taxonomy makes cases operational: after it, it is already clear which team and what lever to deal with the problem.", "blocks": [ { "kind": "code", "label": "Diagram", "language": "text", "value": "LABELS SHOULD ANSWER\n\nwhat task was attempted?\nwhat went wrong?\nhow severe was it?\nhow confident is the reviewer?\nwhat likely subsystem caused it?" } ] }, { "number": "12", "title": "Taxonomy is better to build multidimensional: task, failure type, severity, confidence, root cause", "tag": "Taxonomy", "description": "One label rarely captures the whole picture. One and the same case can be \"task=customer support\", \"failure=missing key fact\", \"severity=high\", \"confidence=medium\", \"root cause=retrieval miss\". This axial model is easier for analytics, slices, backlog routing and subsequent graders training. Flat taxonomy breaks down faster and grows into chaos.", "blocks": [] }, { "number": "13", "title": "Taxonomy should be stable, but versioned", "tag": "Taxonomy", "description": "Changing label names too often destroys comparability over time. But the permanently frozen taxonomy becomes inaccurate as the product evolves. Therefore, a good approach is rare, conscious revisions with versioning, mapping old labels and documented changelog. Then historical cases do not lose value.", "blocks": [ { "kind": "list", "label": "antipatterns", "items": [ "Each reviewer comes up with their own labels.", "New label for every rare case", "taxonomy without migration policy" ] } ] }, { "number": "14", "title": "Annotation guide is more important than list of labels", "tag": "Taxonomy", "description": "If the reviewer sees only the label name, he will begin to interpret it in his own way. Taxonomy is defined by definition, inclusion criteria, exclusion criteria, borderline examples and known confusions. Without a guide, even a good label vocabulary spreads quickly over people and time.", "blocks": [ { "kind": "code", "label": "Guide to every important label", "language": "text", "value": "definition\npositive examples\nnegative examples\nnear-miss cases\nwhat severity levels mean\nwhen reviewer may mark uncertain" } ] }, { "number": "15", "title": "The calibration of reviewers is constantly needed, otherwise labels degrade imperceptibly.", "tag": "Taxonomy", "description": "Even good people over time begin to interpret similar cases differently. Calibration loop: shared examples, disagreement review, adjudication, and an update to the disputed zone guide. Data flywheel breaks not only from bad models, but also from the quiet label drift within the team itself.", "blocks": [] } ] }, { "number": "04", "title": "Human Review Queue", "intro": "", "cards": [ { "number": "16", "title": "Review queue is a team’s working system, not a random “watch later” table.", "tag": "Review", "description": "Human review is needed not only for quality assessment, but also for routing work. Therefore, the queue should have statuses, SLA, owner-a, priority score and clear output: dismiss, label, add to regression, add to canary, escalate to safety, seed for training. If the queue is without status and ownership, flywheel quickly turns into a graveyard of incidents.", "blocks": [] }, { "number": "17", "title": "The queue should have different destinations: hotfix, regression, canary, research, training.", "tag": "Review", "description": "After review, the case should not just “get a label.” The team must decide what to do with it. One case requires an immediate prompt/tool fix, another must get permanently into the regression set, a third is useful only as a future canary probe, a fourth becomes a seed for a larger training set. Destination makes review part of engineering, not a rite of evaluation.", "blocks": [ { "kind": "table", "headers": [ "destination", "sense", "type" ], "rows": [ [ "hotfix", "You need to fix the food quickly.", "prompt / retrieval / policy change" ], [ "regression", "Never repeat this failure.", "plug in" ], [ "canary", "rollout", "separate watch set" ], [ "training", "Materials for Model Improvement", "curate / expand / relabel" ] ] } ] }, { "number": "18", "title": "The reviewer should not only see output but also the decision context.", "tag": "Review", "description": "Evaluating an LLM response in a vacuum is dangerous. Reviewer is useful to see retrieved docs, tool trace summary, prompt revision, user segment and query history. Otherwise, he may correctly notice that the answer is bad, but make a mistake in root cause and send the case to the wrong team. Review UI should support diagnosis, not just voting.", "blocks": [] }, { "number": "19", "title": "Disagreement between reviewers is a signal to improve the guide, not noise", "tag": "Review", "description": "If two strong reviewers diverge, the problem is often not people, but vague definitions, slice-specific nuance, or a really controversial product policy. Such cases should go to adjudication and help improve the annotation guide. In a mature system, disagreement is an input into methodology, not just a “difference of opinion.”", "blocks": [] }, { "number": "20", "title": "The queue should have health metrics: backlog age, SLA miss, agreement, promotion rate.", "tag": "Review", "description": "If review queue grows faster than it gets, flywheel chokes. Therefore, the queue itself should be observed: the average age of the case, the proportion of overdue high-priority incidents, inter-evaluation agreement, how many cases go to regression/canary/training. Otherwise, it seems that there is a process, but in fact it does not have time to affect the product.", "blocks": [] } ] }, { "number": "05", "title": "Dataset Layers", "intro": "", "cards": [ { "number": "21", "title": "Smoke set for quick checks, but does not replace real regression", "tag": "Sets", "description": "Smoke set is a small set of representative cases that quickly shows that the release did not break something basic. It is needed for speed, especially in CI or local variation comparison. But the smoke set is too small and too obvious to be the only gate. It catches disaster, not subtle degradation.", "blocks": [] }, { "number": "22", "title": "Regression set is the curated memory of a team that has already been sick once.", "tag": "Sets", "description": "It is the regression set that makes organizational memory machine-testable. This includes real important cases that have already broken users or product targets. These cases should be stable, well-marked and run as a mandatory gateway for releases, rather than simply being archived “just in case.”", "blocks": [ { "kind": "list", "label": "usually gets", "items": [ "real-life incidents", "high-severity failures", "historically flaky flows", "Not everything from review queue" ] } ] }, { "number": "23", "title": "Canary set is not needed for the final exam, but for an early signal on the rollout.", "tag": "Sets", "description": "Canary set often consists of cases that are sensitive to specific release risk: a new prompt policy, a new retriever, a new model, a new tool strategy. It can be less stable and more acute than regression set. The point is to quickly notice that it is this type of change that began to float, even before the wide roll.", "blocks": [ { "kind": "code", "label": "Diagram", "language": "text", "value": "SET ROLES\n\nsmoke set → basic sanity\nregression set → must not break known pain\ncanary set → watch specific rollout risk\nhard set → stretch / stress / edge cases\ntraining set → improve underlying behavior" } ] }, { "number": "24", "title": "Hard set and adversarial set are needed to ensure that the team is not optimized for the average day.", "tag": "Sets", "description": "If the sets consist only of normal, clean queries, the system will look beautiful in a sign and fall in real life. Therefore, you need a layer of complex cases: ambiguous asks, adversarial wording, messy contexts, multilingual noise, borderline policy cases. This is not necessarily a must-pass set, but it perfectly shows the safety margin of the system.", "blocks": [] }, { "number": "25", "title": "Training set should not be confused with eval set.", "tag": "Risk", "description": "The same initial case can give rise to both training material and eval example, but the final sets must be diluted carefully. Otherwise, the team will begin to measure success on what the system has already built. Separation of eval and training data is a basic defense against self-deception and a false sense of progress.", "blocks": [] } ] }, { "number": "06", "title": "Release Loop", "intro": "", "cards": [ { "number": "26", "title": "The path of the case in the regression set must be clear and rare, otherwise the gate will suffocate.", "tag": "Release", "description": "If you put everything in regression, the set quickly becomes noisy, expensive and poorly maintained. Promotion policy: severity, reproducibility, label confidence, importance for business flow, availability of expected answer and uniqueness in relation to existing cases. Regression set should be a curated museum of important pains, not an endless dump.", "blocks": [ { "kind": "code", "label": "promotion policy", "language": "text", "value": "if case is\n high-value\n reproducible\n clearly labeled\n not duplicate\nthen\n promote to regression\nelse\n keep in exploratory / review pool" } ] }, { "number": "27", "title": "Comparison of prompt/model/retriever options should go on the same curated sets, not random demo examples.", "tag": "Release", "description": "The essence of flywheel is that it delivers single work kits for choosing between system options. If a team changes a model, a prompt template, or a retrieval strategy, and only tests it on “a couple of favorite examples,” progress is almost always overrated. Curated sets turn the comparison of options into a repeatable procedure.", "blocks": [] }, { "number": "28", "title": "Canary release is only useful when there is a meaningful canary watchlist behind it.", "tag": "Release", "description": "By itself, the percentage of traffic to a new release doesn’t tell you what you’re watching. You need a linked canary set and a set of product/risk metrics that are particularly sensitive to current change. Canary then becomes not a “wait and see,” but a controlled experiment with known failure modes.", "blocks": [] }, { "number": "29", "title": "Synthetic expansion is useful when it grows from real seed cases, rather than living in isolation from food pains.", "tag": "Release", "description": "The LLM can help expand the dataset around found product failure: paraphrasing, neighboring edge cases, variations in language, context, and complexity. This greatly increases the coverage. But if synthetic generation is detached from real-world seeds and review, it begins to teach the system to pass the exam on artificial problems, rather than solve real-world problems of users.", "blocks": [] }, { "number": "30", "title": "From one flywheel can be born different levers of improvement: prompt fix, retrieval fix, guardrail, fine-tune", "tag": "Release", "description": "Not every case should lead to training. Sometimes it is enough to repackage the context, improve the reranker, change the tool policy or add a better refusal pattern. And sometimes repeated cluster already clearly says that you need a trained model or distillation. A good flywheel isn’t tied to a single fix, it supplies the evidence to pick the right lever.", "blocks": [] } ] }, { "number": "07", "title": "Ops and Governance", "intro": "", "cards": [ { "number": "31", "title": "Dataset versioning and lineage are mandatory, otherwise no one will understand what the releases were compared on.", "tag": "Ops", "description": "For each set, it should be clear: when it was created, from which cases it was collected, which taxonomy was marked, which guideline version was used and what changed in the last revision. Without lineage, any comparison of quality becomes slippery: the team seems to see an improvement, but does not understand whether the system has improved or the dataset has quietly changed.", "blocks": [] }, { "number": "32", "title": "Privacy and redaction are part of flywheel data, not an external compliance ritual", "tag": "Ops", "description": "Production cases often contain PII, commercial details, and internal documents. If editing and redaction are not built into the intake pipeline, flywheel will either become dangerous or the team will stop using the cases altogether. Privacy should be factored directly into the case record, review UI and export sets.", "blocks": [] }, { "number": "33", "title": "It is necessary to measure not only the size of the set, but also the coverage, freshness and label health.", "tag": "Ops", "description": "The big dataset itself says nothing. More importantly, which task types are covered, which segments are underrepresented, how quickly new error types fall into curated sets, how stable the reviewer agreement is, what the age of regression cases is. Data flywheel is only useful when its quality is monitored and managed.", "blocks": [ { "kind": "list", "label": "Useful framing", "items": [ "coverage by slice", "review agreement", "promotion rate", "dataset freshness" ] } ] }, { "number": "34", "title": "Data flywheel dies without clear roles: triage owner, reviewer, adjudicator, release owner", "tag": "Ops", "description": "Even a good process falls apart if you don’t know who takes the cases, who oversees taxonomy, who solves controversial labels, and who translates the results into release gates. As a mature team, flywheel has not only tools but also shared responsibility. Otherwise, cases hang at the junction between product, ML, safety and platform.", "blocks": [] }, { "number": "35", "title": "A sign of a mature AI team: it can turn user pain into reproducible kits and system solutions", "tag": "Ops", "description": "Data flywheel is not just a dataset conveyor. This is a way to make the product reality manageable: real failure is not lost, but gets labels, gets in line, becomes a gate, affects the release and sometimes the training of the next generation of the system. In this sense, datasets are not a by-product of the team, but its memory, method, and quality engine.", "blocks": [ { "kind": "code", "label": "Diagram", "language": "text", "value": "TEAM MATURITY\n\nlogs exist\n ↓\ncases are curated\n ↓\nlabels are consistent\n ↓\nreview queue has owners\n ↓\nregression/canary sets exist\n ↓\nreleases are gated by real cases\n ↓\nproduct quality compounds over time" } ] } ] } ] }