{ "type": "new-runtime-knowledge-guide", "version": 1, "generated_at": "2026-07-23", "volume": 12, "slug": "eval-process-llm-judge-reference", "title": "Evaluation Process & LLM-as-Judge", "description": "How to build the process of evaluating LLM systems from scratch: minimum viable eval, guardrails vs evaluators, why a binary assessment is better than a scale of 1–5, negative scenarios, LLM-as-judge with correlation to people, detection of degradation after release, synthetic data - when it helps and when it harms.", "track": { "slug": "evaluation-and-improvement", "title": "Evaluation & Improvement", "route": "https://newruntime.com/learn/tracks/evaluation-and-improvement/", "position": 2 }, "counts": { "sections": 7, "cards": 23 }, "routes": { "html": "https://newruntime.com/learn/eval-process-llm-judge-reference/", "json": "https://newruntime.com/learn/eval-process-llm-judge-reference.json" }, "sections": [ { "number": "01", "title": "Eval Process: where to start", "intro": "", "cards": [ { "number": "01", "title": "Minimum Viable Eval: from scratch in 3 steps", "tag": "Process", "description": "Most teams put off evaluation because it seems like expensive infrastructure is needed. No: the minimum working eval is 20–50 examples, pass/fail criteria and one “benevolent dictator” - a person with domain expertise, whose opinion is considered the standard.", "blocks": [ { "kind": "code", "label": "Diagram", "language": "sql", "value": "PHASE 1 - LAUNCH (Day 1-3)\n\nStep 1: Collect 20–50 Real Requests\n └─ from production logs, from stakeholders, invented by hand\n └─ cover: standard cases + 5–10 edge cases\n └─ DO NOT generate LLM at this stage\n\nStep 2: one expert marks manually\n └─ \"benevolent dictator\": PM / domain expert / TL\n └─ for each example: pass / fail + 1 line why\n └─ DO NOT discuss the criteria - first mark it intuitively\n └─ then - extract patterns from your solutions into criteria\n\nStep 3: run the system, compare\n └─ count pass_rate = passes / total\n └─ look at specific fails - WHAT exactly is wrong\n └─ this is your baseline - remember it\n\n───────────────────────────────── ─────────────────────────────────\nPHASE 2 - GROWTH (after the first launch)\n\n every prod bug → add to dataset\n each prompt change → run eval\n every 2 weeks → review of criteria\n with 50+ examples → you can think about automation" }, { "kind": "list", "label": "start antipatterns", "items": [ "We are waiting for the “correct” infrastructure", "we immediately build LLM-judge without baseline", "generate a dataset LLM without verification", "20 manual examples > 1000 synthetic ones" ] } ] }, { "number": "02", "title": "Error Analysis: hands before automation", "tag": "Process", "description": "After each significant change to the system (prompt, model, chunking, retriever), be sure to read 20–50 outputs manually. Automated metrics hide error patterns that are instantly visible to the eye.", "blocks": [ { "kind": "code", "label": "error analysis protocol", "language": "sql", "value": "# After any change:\nsample = random_sample(outputs, n=30)\n# or: all fails from eval for the period\n\n# For each read:\n✓ Is the answer correct in meaning?\n✓ Is the answer based on context or fictitious?\n✓ Is the formatting as expected?\n✓ Is the answer too long/short/evasive?\n✓ Does the model respond when it should fail?\n✓ Does the model refuse when it should respond?\n\n# Group errors by type:\nhallucination | wrong_refusal\noff_topic | format_error\nincomplete | tone_mismatch\n\n# Most common category → fix first" }, { "kind": "list", "label": "practice", "items": [ "read the fails yourself - don’t delegate to the junior", "30 examples give 80% of the picture", "every bug → test case in the dataset" ] } ] }, { "number": "03", "title": "Response metrics vs pipeline metrics", "tag": "Process", "description": "Key principle: a good bottom line metric doesn't tell you where it's broken. Measure each pipeline node separately - retrieval, rerank, prompt, generation - otherwise you won’t understand what to fix.", "blocks": [ { "kind": "code", "label": "Diagram", "language": "sql", "value": "PIPLINE NODE METRICS SUMMARY METRICS\n\nQuery\n │\n ▼\nRetrieval ──→ Recall@k, MRR, nDCG@k ┐\n │ │\n ▼ │\nRerank ──→ Precision@k, MRR delta │──→ Faithfulness\n │ │Answer Relevance\n ▼ │ Task Success Rate\nPrompt ──→ token count, format │\n │ few-shot hit rate │\n ▼ │\nGeneration ──→ hallucination rate │\n refusal rate │\n latency, cost ┘\n\nThe final metric is growing → but from where?\nNodal metrics → we know exactly what has changed" }, { "kind": "list", "label": "rule", "items": [ "change one variable at a time", "record the dataset, model, index when comparing", "comparing on different datasets = pointless" ] } ] }, { "number": "04", "title": "Trace: what must be logged", "tag": "Process", "description": "Trace - a record of everything that happened in the pipeline for one request. Without traces, you cannot debug, build eval and detect degradation. The minimum set of fields is the one without which it is impossible to answer “why the answer is incorrect.”", "blocks": [ { "kind": "code", "label": "required trace fields", "language": "json", "value": "{\n \"trace_id\": \"uuid\", # end-to-end ID\n \"timestamp\": \"ISO-8601\",\n \"user_id\": \"...\", # for user-level analysis\n\n // Input\n \"query\": \"original query\",\n \"query_rewrite\":\"after augmentation\", # if any\n\n // Retrieval\n \"retrieved_chunks\": [\"chunk_id\", \"score\", \"text\"],\n \"reranked_chunks\": [\"chunk_id\", \"score\"],\n\n //Generation\n \"prompt_tokens\": 1240,\n \"completion_tokens\": 312,\n \"model\": \"cloud-sonnet-4\",\n \"response\": \"final response\",\n \"latency_ms\": 1840,\n\n // Eval (async, after request)\n \"faithfulness\": 0.92,\n \"refusal\": false\n}" }, { "kind": "list", "label": "tools", "items": [ "Langfuse - open-source, self-hosting", "Phoenix Arize - traces + eval", "the trace structure is more important than the tool" ] } ] } ] }, { "number": "02", "title": "Binary vs Scale: why pass/fail wins", "intro": "", "cards": [ { "number": "05", "title": "Why binary rating is better than 1-5 scale", "tag": "Binary", "description": "Intuition suggests: a scale of 1–5 is more accurate, because it contains more information. Experience suggests the opposite: numerical scales are unstable and there is little agreement between raters (humans and LLMs) on what a “3” vs “4” means.", "blocks": [ { "kind": "code", "label": "Diagram", "language": "sql", "value": "THE PROBLEM OF SCALES\n\nSame answer, three evaluators:\n Ivan: 4 - “good, but a little incomplete”\n Maria: 3 — “average, expected more details”\n LLM: 5 - “completely answers the question”\n\n Cohen's κ = 0.18 ← weak agreement, noise\n\n───────────────────────────────── ─────────────────────────────────\nWHY BINARY WORKS\n\nThe same answer, worded as a clear criterion:\n \"Does the answer contain a specific number?\" → pass / fail\n\n Ivan: pass\n Maria: pass\n LLM: pass\n\n Cohen's κ = 0.91 ← excellent agreement\n\n───────────────────────────────── ─────────────────────────────────\nPRINCIPLE: One criterion = one binary test\n\nBad: \"how good is the answer from 1 to 5?\"\n → mixes: relevance + completeness + tone + facts\n\nOkay: four separate questions:\n 1. Does the answer answer the user's question? Y/N\n 2. Is the answer supported by the context (not made up)? Y/N\n 3. Does the tone match the brand style? Y/N\n 4. Does the response contain the requested format? Y/N\n\n pass_rate for each → it’s clear what exactly is broken" }, { "kind": "list", "label": "when the scale is still justified", "items": [ "ranking of multiple options (A/B/C/D)", "quality gradations with clear categories and examples", "only if κ > 0.6 on the pilot marking", "never for automation without verification" ] } ] }, { "number": "06", "title": "How to formulate pass/fail criteria", "tag": "Binary", "description": "The criterion must be formulated so that any reasonable person (or LLM) would give the same answer. Test: give a criterion to two people without explanation - if they don’t match, the criterion is bad.", "blocks": [ { "kind": "code", "label": "good criterion template", "language": "sql", "value": "# Structure: \"The answer [does X]?\" where X is specifically\n\n✗ Bad (blurry):\n \"Is the answer quality?\"\n \"Is the answer useful to the user?\"\n \"Is the answer correct?\"\n\n✓ Good (specific):\n \"Does the answer contain at least one specific number?\"\n \"The answer doesn't mention a competitor company?\"\n \"Is the answer written in Russian?\"\n “The answer begins with a direct answer, not with introductory words?”\n \"Does the answer contain statements that are not in the context?\"\n\n# Criterion test:\nsample = 20 examples\nhuman_1_labels = [...]\nhuman_2_labels = [...]\nκ = cohens_kappa(human_1_labels, human_2_labels)\n# κ > 0.6 → criterion is valid\n# κ < 0.4 → reformulate" }, { "kind": "list", "label": "practice", "items": [ "one criterion → one question", "formulation through specific behavior", "abstract words: “quality”, “useful”" ] } ] }, { "number": "07", "title": "False Refusal vs Hallucination: how to choose a balance", "tag": "Binary", "description": "These are two opposite types of errors. If the model is too careful, it refuses to answer when the answer would be correct (false refusal). If she is too bold, she makes up facts (hallucination). The balance depends on the domain and the cost of the error.", "blocks": [ { "kind": "code", "label": "Diagram", "language": "text", "value": "TWO TYPES OF ERROR\n\nFALSE REFUSAL (type I error)\n Situation: the answer would be correct, but the model refused\n Example: “I can’t answer this question” - but I could\n Price: bad UX, user did not receive help\n Metric: refusal_rate on \"safe\" questions\n\nHALLUCINATION (error of the second type)\n Situation: the model answered, but made up the facts\n Example: “The company was founded in 1998” is incorrect\n Cost: loss of trust, legal risks, harm\n Metrics: faithfulness_score, citation_accuracy\n\n───────────────────────────────── ─────────────────────────────────\nHOW TO CHOOSE A BALANCE\n\nDomain with a high hallucination price → hard refusal\n medicine, law, finance:\n It is BETTER to refuse than to give out an incorrect fact.\n → policy: “if not in the context, say “I don’t know””\n\nDomain with high failure cost → tolerance for inaccuracies\n chatbotonsite, creative assistant:\n It is BETTER to give an approximate answer than to refuse\n → policy: \"answer with caveat 'check with a specialist'\"\n\n───────────────────────────────── ─────────────────────────────────\nHOW TO MEASURE\n\nrefusal_rate = refusals / all requests\n — monitor separately for “safe” and “unsafe” requests!\n\nhallucination_rate = unsubstantiated claims / all claims\n\nSeparate dataset for each class:\n safe_questions → refusal_rate must be < threshold\n unsafe_questions → refusal_rate must be > threshold" }, { "kind": "list", "label": "typical thresholds", "items": [ "medicine: hallucination < 1%, refusal ok", "e-commerce: refusal < 5%, hallucination < 10%", "always measure separately - one final figure hides" ] } ] } ] }, { "number": "03", "title": "LLM-as-Judge: correlation with people", "intro": "", "cards": [ { "number": "08", "title": "Correlation of LLM-judge with human eval", "tag": "Judge", "description": "LLM-judge is only useful if his scores match how people score. Validation of judge is a mandatory step before trusting automatic scores. The binary score gives a significantly better correlation than the numerical scale.", "blocks": [ { "kind": "code", "label": "Diagram", "language": "text", "value": "WHY BINARY CORRELATES BETTER WITH PEOPLE\n\nExperiment: 200 examples, 3 experts + LLM-judge\n\nScale 1–5 (LLM vs expert average):\n Pearson r = 0.52 ← weak correlation\n Cohen's κ = 0.21 ← poor agreement\n LLM: \"4\", Expert 1: \"3\", Expert 2: \"5\" - are all three right?\n\nBinary pass/fail (LLM vs expert consensus):\n Accuracy = 0.84 ← much better\n Cohen's κ = 0.67 ← good agreement\n pass/fail clearly - “is the fact in the source? Y/N”\n\n───────────────────────────────── ─────────────────────────────────\nHOW TO VALIDATE JUDGE\n\nStep 1: Collect 100–200 examples of varying quality\nStep 2: Mark manually (2–3 experts)\nStep 3: run LLM-judge using the same examples\nStep 4: Read Agreement Metrics\n\nGood judge (binary): κ > 0.6, accuracy > 0.80\nAcceptable: κ 0.4–0.6\nBad - do not use: κ < 0.4\n\nReview judge when changing domain/model/criteria" }, { "kind": "list", "label": "known problems of LLM-judge", "items": [ "self-enhancement bias: the model prefers its own answers", "verbose bias: long answer scores higher", "position bias: the first in a pair gets an advantage", "use a stronger model as judge" ] } ] }, { "number": "09", "title": "Judge Prompt: Structure for Reliability", "tag": "Judge", "description": "The quality of judge critically depends on the structure of its prompt. Template: clear criterion → CoT reasoning → final verdict. CoT before the verdict significantly improves quality and provides an explanation for the debug.", "blocks": [ { "kind": "code", "label": "judge prompt template", "language": "text", "value": "SYSTEM:\n You evaluate the quality of the AI assistant's answers.\n Answer strictly according to the format below.\n\nUSER:\n USER QUESTION: {question}\n CONTEXT (sources): {context}\n ASSISTANT'S REPLY: {answer}\n\n CRITERIA:\n Are all the facts in the answer supported by context?\n A fact is considered confirmed if it is clearly\n is present in one of the sources above.\n\n Step 1. Explain: List all statements in your answer.\n and for each, indicate whether it is in the context.\n\n Step 2. Verdict: PASS or FAIL.\n (PASS only if ALL statements are confirmed)\n\nEXPECTED FORMAT:\n Analysis: [chain of reasoning]\n Verdict: PASS/FAIL" }, { "kind": "list", "label": "how to increase stability", "items": [ "Temperature=0 for reproducibility", "3 runs → majority vote at T>0", "swap A/B in pairwise comparison", "CoT before the verdict - not after" ] } ] }, { "number": "10", "title": "When BERT, when LLM as a judge", "tag": "Judge", "description": "Not every check needs to be done through an LLM call. Many criteria are checked cheaper and more reliably by deterministic code, regex or BERT classifier. LLM-judge - only where understanding of the meaning is needed.", "blocks": [ { "kind": "table", "headers": [ "What we check", "Best tool", "Why" ], "rows": [ [ "JSON validity", "json.parse()", "deterministic, 0ms" ], [ "Response Length", "len(text)", "deterministic" ], [ "Availability of keywords", "regex / str.contains", "quickly, accurately" ], [ "PII (email, phone)", "regex / spaCy NER", "cheaper, more reliable" ], [ "Toxicity", "BERT classifier", "fast, calibrated" ], [ "Semantic similarity", "cosine(embed_A, embed_B)", "without LLM call" ], [ "Faithfulness (facts vs source)", "LLM-judge / NLI", "need understanding" ], [ "Tone, style, \"Useful framing\"", "LLM-judge with rubric", "only if κ>0.6" ] ] }, { "kind": "list", "label": "principle", "items": [ "cheap first: regex → BERT → LLM", "LLM-judge everywhere = expensive and unstable" ] } ] }, { "number": "11", "title": "LLM-judge as guardrail: why not", "tag": "Judge", "description": "It would seem logical: put LLM-judge directly into the pipeline as a filter for each answer. In practice, this is almost always a bad idea - due to latency and instability.", "blocks": [ { "kind": "code", "label": "why not", "language": "text", "value": "Problem 1: Latency\n Main call: 800ms\n + LLM-judge: +600ms ← +75% latency\n = total: 1400ms ← user notices\n\nProblem 2: Instability\n The same request, judge in the pipeline:\n Run 1: PASS → response sent\n Run 2: FAIL → response blocked\n ← non-deterministic UX\n\nProblem 3: False locks\n judge is wrong → the correct answer is blocked\n the user receives a refusal to a valid question\n\n──────────────────────── ─────────────────────────\nWhat to do instead:\n\nIn-pipeline (guardrail): regex + BERT (fast)\nAsync eval (evaluator): LLM-judge on sample\n → estimate 10% of traffic asynchronously\n → do not block the user\n → alert when metrics drop" }, { "kind": "list", "label": "exceptions", "items": [ "very high rates (medicine) → latency can be tolerated", "then: cache judge on similar queries" ] } ] } ] }, { "number": "04", "title": "Guardrails vs Evaluators: different roles", "intro": "", "cards": [ { "number": "12", "title": "Guardrail vs Evaluator: architectural difference", "tag": "Guardrail", "description": "Guardrail is part of the production pipeline, works synchronously, blocks or modifies output in real time. Evaluator is a separate quality assessment service that works asynchronously or offline and does not directly affect the user’s UX.", "blocks": [ { "kind": "code", "label": "Diagram", "language": "text", "value": "GUARDRAIL - in the pipeline\n\nUSER → [INPUT GUARDRAIL] → LLM → [OUTPUT GUARDRAIL] → USER\n │ │\n ▼ BLOCK ▼ BLOCK/MODIFY\n reject replace/warn\n\nSpecifications:\n ✓ Synchronous - blocks until answered\n ✓ Deterministic - no instability\n ✓ Fast - regex/BERT, not LLM\n ✓ Affects EVERY request\n\n───────────────────────────────── ─────────────────────────────────\nEVALUATOR - outside the pipeline\n\nUSER → LLM → USER ← user does not wait\n │\n └──→ LOG → [EVALUATOR] → METRICS → DASHBOARD\n ▼\n ALERT if degraded\n\nSpecifications:\n ✓ Asynchronous - does not affect latency\n ✓ Can use LLM-judge (expensive - ok)\n ✓ Evaluates a sample (10–100% of traffic)\n ✓ Gives a signal to improve the system" }, { "kind": "list", "label": "key rule", "items": [ "Guardrail = security and format", "Evaluator = quality and improvement", "do not confuse the roles - LLM-judge is not guardrail" ] } ] }, { "number": "13", "title": "What to install in Guardrail: tools", "tag": "Guardrail", "description": "The rule of choice: the cheaper and more deterministic the tool, the better for guardrail. Layered approach: first cheap checks, then expensive ones - only if you pass the cheap ones.", "blocks": [ { "kind": "code", "label": "guardrails stack by cost", "language": "text", "value": "Layer 1 - deterministic (0ms, $0)\n regex: prohibited words, PII patterns\n schema: JSON validation, data type\n length: min/max tokens\n format: expected response structure\n\nLayer 2 - BERT/classifier (5–20ms, ~$)\n toxicity: BERT toxicity classifier\n PII: spaCy NER for email/phone/passport\n topic: zero-shot classifier (not our topic?)\n lang: detect_language(response) == \"ru\"\n\nLayer 3 - Light LLM (50–200ms, $$)\n safety: haiku/mini checks safety\n schema: \"response is valid JSON with fields X,Y?\"\n ← only if the first layers were skipped\n\nLayer 4 - heavy LLM (never in guardrail)\n faithfulness, helpfulness → evaluator only" } ] }, { "number": "14", "title": "Evaluator as a separate service", "tag": "Guardrail", "description": "Evaluator is an asynchronous service subscribed to a log queue. Evaluates a random sample of traffic and builds metrics. The main task: to give a signal that “the system has degraded” even before users start complaining.", "blocks": [ { "kind": "code", "label": "evaluator service architecture", "language": "python", "value": "# Data stream\nPROD TRAFFIC → message queue (Kafka/SQS)\n │\n sampling: 10% of requests\n │\n ▼\n EVALUATOR WORKER\n for each trace:\n faithfulness = llm_judge(...)\n format_ok = regex_check(...)\n refusal = detect_refusal(...)\n │\n ▼\n METRICS STORE (TimeSeries DB)\n │\n ▼\n ALERT if score_7d_avg drops > 5%" }, { "kind": "list", "label": "what to monitor", "items": [ "faithfulness_score (moving average)", "refusal_rate (growth = problem)", "latency p50/p95/p99", "alert when deviation > 2σ from baseline" ] } ] } ] }, { "number": "05", "title": "Negative Scenarios: dataset for failures", "intro": "", "cards": [ { "number": "15", "title": "Negative Scenarios: why and how to build a dataset", "tag": "Negative", "description": "Most teams only test the happy path: “ask a question, get an answer.” But the system must be able to intelligently refuse: when there is no information in the context, when the request is outside the domain, when the request is unsafe. Each type requires a separate dataset.", "blocks": [ { "kind": "code", "label": "Diagram", "language": "sql", "value": "TAXONOMY OF NEGATIVE SCENARIOS\n\nType 1: No answer in context\n Query: \"What is the commission for category X?\"\n Context: does not contain information about X\n Expectedly: “There is no information in the provided sources...”\n Error: The model is making up an answer.\n\nType 2: Out-of-domain\n Request: \"Write me a poem\"\n Domain: Business Assistant for Analytics\n Expected: polite refusal + redirect\n Error: executing request\n\nType 3: Unsafe / jailbreak\n Request: attempt to bypass restrictions\n Expected: failure without system prompt expansion\n Error: Executes or expands prompt\n\nType 4: Ambiguous/unanswerable\n Request: too vague, no point in answering\n Expected: clarifying question, not a hallucination\n Error: confident answer to an unclear question\n\n───────────────────────────────── ─────────────────────────────────\nHOW TO BUILD A DATASET\n\nStep 1: Collect real cases\n from logs: requests without context, user complaints\n\nStep 2: synthetically expand\n take positive examples → rephrase → remove answer from context\n take your domain → generate off-topic queries\n\nStep 3: verify markup\n for each example: expected = REFUSE / ANSWER\n κ between two markers > 0.7" }, { "kind": "list", "label": "metrics on negative dataset", "items": [ "correct_refusal_rate → should be high", "false_answer_rate (hallucination) → should be low", "measure separately for each type" ] } ] }, { "number": "16", "title": "Dataset for “whether the model can not know”", "tag": "Negative", "description": "A special class of tests: queries that the model has no reason to answer. The goal is to check that the model is not making things up, but admitting ignorance. Pattern: correct question + obviously empty or irrelevant context.", "blocks": [ { "kind": "code", "label": "construction patterns", "language": "sql", "value": "# Pattern 1: there is context, no answer\ncontext = \"WB commission for clothes: 12%. Shoes: 15%.\"\nquestion = \"What is the commission for electronics?\"\nexpected = REFUSE # \"The source contains no electronics data\"\n\n# Pattern 2: question about the future/non-existent\nquestion = \"What will the dollar exchange rate be in 2030?\"\nexpected = REFUSE # \"Can't predict the future\"\n\n# Pattern 3: Conflicting Context\ncontext = \"Commission 12%. Commission 18%.\"\nquestion = \"What is the commission?\"\nexpected = REFUSE/CLARIFY # indicate a contradiction\n\n# How to evaluate automatically:\njudge_prompt = \"\"\"\n Did the assistant answer the question using\n ONLY information from context?\n If the context does not contain an answer, is it correct?\n the assistant refused to answer? PASS/FAIL\n\"\"\"" } ] }, { "number": "17", "title": "Hallucination Probing: test for fiction", "tag": "Negative", "description": "Active testing: intentionally give the model a false or empty context and see if it uses its “prior knowledge” instead of the context. This test often reveals problems that are not visible in a normal eval.", "blocks": [ { "kind": "code", "label": "probing techniques", "language": "text", "value": "# Technique 1: False Fact Context\ncontext = \"The company was founded in 2031.\" # lies\nquestion = \"When was the company founded?\"\n\n✓ Correct: “In 2031” ← follows the context\n✗ Problem: “In 1998” ← ignores context, returns prior\n\n# Technique 2: empty context\ncontext = \"\"\nquestion = \"What is the VAT rate in Russia?\"\n\n✓ Correct: \"There is no data in the provided context\"\n✗ Problem: “VAT in Russia is 20%” ← hallucination\n\n#Technique 3: Irrelevant context\ncontext = \"Borscht recipe: beets, potatoes...\"\nquestion = \"What are the delivery terms?\"\n\n✓ Correct: refusal / clarification\n✗ Problem: invents delivery terms" }, { "kind": "list", "label": "starting frequency", "items": [ "every time the prompt changes - mandatory", "when changing model - required", "~10% of the dataset = hallucination probes" ] } ] } ] }, { "number": "06", "title": "Drift Detection: degradation after release", "intro": "", "cards": [ { "number": "18", "title": "Types of system degradation after release", "tag": "Drift", "description": "The system can degrade without a single code change - due to changes in the external environment: new documents in the knowledge base, changes in user request patterns, changes in the API of the model provider. Understanding the sources of drift is critical to proper diagnosis.", "blocks": [ { "kind": "table", "headers": [ "Drift type", "Reason", "Detector", "Treatment" ], "rows": [ [ "Data drift", "New documents in KB do not match the old chunking", "retrieval recall@k crashes", "re-embed new docs" ], [ "Query drift", "Users started asking about new topics", "out-of-domain rate is growing", "extend KB / update prompt" ], [ "Model drift", "The provider has changed the model (silent update)", "response format/tone changes", "pin version of the model" ], [ "Concept drift", "Reality has changed (new law, prices)", "faithfulness falls due to new facts", "update KB, add date" ], [ "Prompt drift", "The prompt was changed and eval did not run", "pass_rate for golden set", "CI for prompts" ] ] }, { "kind": "code", "label": "degradation signals that are visible in metrics", "language": "text", "value": "Early signals (fast): rising → bad sign\n ↑ refusal_rate — the model refuses more often\n ↑ response_length — “inflates” responses\n ↑ latency_p99 - something in the pipeline has slowed down\n ↑ token_count - context is growing (context pollution)\n\nDelayed signals (via eval): falling → bad sign\n ↓ faithfulness_score\n ↓ recall@k (retriever)\n ↓ task_success_rate (agent)\n ↓ user_thumbs_up rate" } ] }, { "number": "19", "title": "Golden Set: monitoring without a person", "tag": "Drift", "description": "Golden dataset is a fixed set of examples with known expected answers. Run regularly (daily) on a production system without changes - any drop in pass_rate signals drift.", "blocks": [ { "kind": "code", "label": "monitoring structure", "language": "python", "value": "# Daily scheduled job\nfor example in golden_set:\n response = system.run(example.query)\n score = evaluate(response, example.criteria)\n metrics_store.record(score, timestamp=today)\n\n# Comparison with baseline (release day)\ndelta = today_pass_rate - baseline_pass_rate\n\nif delta < -0.05: # fell by 5%\n alert(channel=\"#ml-alerts\",\n msg=f\"Golden set: {delta:+.1%}\")\n\n# What should be in the golden set:\n# - typical cases (50%)\n# - edge cases (30%)\n# - negative scenarios (20%)" }, { "kind": "list", "label": "practice", "items": [ "50–200 examples are enough for a signal", "update golden set when changing domain", "don’t update with every improvement - you lose baseline" ] } ] }, { "number": "20", "title": "Controlled experiments: what to record", "tag": "Drift", "description": "The purity of the experiment is the main condition for meaningful comparison. Change one variable at a time. Without this, you won’t understand what exactly improved or broke the system.", "blocks": [ { "kind": "code", "label": "controlled experiment protocol", "language": "text", "value": "# Fix (do not change):\nFIXED = {\n \"eval_dataset\": \"v3.1\", # one dataset\n \"vector_index\": \"2025-06-01\", # index snapshot\n \"embed_model\": \"text-emb-3\",\n \"llm_model\": \"cloud-sonnet-4-6\",\n \"temperature\": 0.0, # reproducible\n \"k\": 5,\n}\n\n# Change (one at a time):\nVARIABLE = \"chunk_size\" #512 -> 256\n\n# Run both options on the same dataset\nresults_A = run(system_A, FIXED, dataset)\nresults_B = run(system_B, FIXED, dataset)\n\n# Compare metrics\n# Make sure the difference is statistically significant\n# p-value < 0.05 for n >= 100" } ] } ] }, { "number": "07", "title": "Synthetic Data: when it helps, when it harms", "intro": "", "cards": [ { "number": "21", "title": "Structured Synthetic Generation: not just “give me 100 questions”", "tag": "Synthetic", "description": "A naive approach—asking an LLM to generate 100 questions—yields a homogeneous, biased dataset. Structural approach: first a taxonomy of complexity and types, then generation for each matrix cell, then verification.", "blocks": [ { "kind": "code", "label": "Diagram", "language": "sql", "value": "GENERATION MATRIX\n\n Simple Medium Difficult\nThe actual \"what is X?\" \"how is X different\" why does X work\n from Y?\" is different from Z?\"\n\nAnalytical \"list...\" \"compare A and B\" \"what will happen\n if...\"\n\nRefusal \"tell\" what will happen in \"how to get around\n joke \"2040?\" restriction?\n\n→ Generate N examples in each cell\n→ This gives variety and coverage of edge cases\n\n───────────────────────────────── ─────────────────────────────────\nPIPELINE\n\nStep 1: Real documents → LLM generates QA pairs\n prompt = \"Generate 3 questions from the text below:\n one simple, one requiring reasoning,\n one for which there is no answer in the text.\"\n\nStep 2: Deduplication by embedding cosine > 0.92\n # remove semantic duplicates\n\nStep 3: Auto-filtering by quality score\n quality_score = LLM(\"Is this question clear and unambiguous? Y/N\")\n\nStep 4: Selective Human Verification\n sample 10–20% → one expert checks\n approval_rate > 80% → dataset is good\n approval_rate < 80% → revise generation prompt" }, { "kind": "list", "label": "practice", "items": [ "matrix approach → diversity", "sample verification is mandatory", "100% LLM without verification → confirmation bias", "start with real examples anyway" ] } ] }, { "number": "22", "title": "When is synthetic data harmful?", "tag": "Synthetic", "description": "Synthetic data carries risks that are easy to overlook, especially when the same model is used for both dataset generation and evaluation. This is called model collapse or circular evaluation.", "blocks": [ { "kind": "code", "label": "cases when synthetic is a bad choice", "language": "text", "value": "1. Circular eval (main problem)\n GPT-4 generates QA → GPT-4 evaluates QA\n ← the model evaluates “itself”, there will always be a good score\n ← does not detect model system errors\n\n2. Distribution mismatch\n Synthetic questions - literary Russian\n Real users - colloquial, with typos\n ← eval shows good, cont shows bad\n\n3. Coverage gaps\n LLM does not know what NOT to generate (edge cases in prod)\n ← does not cover real problem queries\n\n4. Incorrect reference answers\n LLM generates \"correct answer\" with error\n Let's use it as ground truth for eval\n ← testing on obviously incorrect examples\n\nWhen synthetic is acceptable:\n ✓ expansion of a small dataset of real examples\n ✓ type coverage (not replacing real examples)\n ✓ with verification by an expert" } ] }, { "number": "23", "title": "Keeping the golden dataset up to date", "tag": "Synthetic", "description": "The Golden dataset is becoming obsolete: the business domain is changing, new types of questions appear, the product is changing. An updating process is needed - without it, eval begins to measure something else.", "blocks": [ { "kind": "code", "label": "lifecycle dataset", "language": "text", "value": "# Update triggers:\n→ A bug was found in the product → add as a fail test\n→ Product has changed → update expected\n→ New feature → add new cases\n→ Once a quarter → review of outdated examples\n\n# Versioning:\ndatasets/\n golden_v1.jsonl # ← do not delete! baseline for comparison\n golden_v2.jsonl\n golden_current → golden_v2.jsonl # symlink\n\n# Metadata for each example:\n{\n \"id\": \"tc_087\",\n \"added\": \"2025-04-12\",\n \"source\": \"prod_bug_#2341\",\n \"last_reviewed\": \"2025-06-01\",\n \"tags\": [\"refusal\", \"no-context\"]\n}" }, { "kind": "list", "label": "practice", "items": [ "every product bug → test case", "versioning the dataset as code", "keep old versions for comparison", "update golden set without committing baseline" ] } ] } ] } ] }