Evaluation Process & LLM-as-Judge

PHASE 1 - LAUNCH (Day 1-3)

Step 1: Collect 20–50 Real Requests
  └─ from production logs, from stakeholders, invented by hand
  └─ cover: standard cases + 5–10 edge cases
  └─ DO NOT generate LLM at this stage

Step 2: one expert marks manually
  └─ "benevolent dictator": PM / domain expert / TL
  └─ for each example: pass / fail + 1 line why
  └─ DO NOT discuss the criteria - first mark it intuitively
  └─ then - extract patterns from your solutions into criteria

Step 3: run the system, compare
  └─ count pass_rate = passes / total
  └─ look at specific fails - WHAT exactly is wrong
  └─ this is your baseline - remember it

───────────────────────────────── ─────────────────────────────────
PHASE 2 - GROWTH (after the first launch)

  every prod bug → add to dataset
  each prompt change → run eval
  every 2 weeks → review of criteria
  with 50+ examples → you can think about automation

# After any change:
sample = random_sample(outputs, n=30)
# or: all fails from eval for the period

# For each read:
✓ Is the answer correct in meaning?
✓ Is the answer based on context or fictitious?
✓ Is the formatting as expected?
✓ Is the answer too long/short/evasive?
✓ Does the model respond when it should fail?
✓ Does the model refuse when it should respond?

# Group errors by type:
hallucination |  wrong_refusal
off_topic |  format_error
incomplete |  tone_mismatch

# Most common category → fix first

PIPLINE NODE METRICS SUMMARY METRICS

Query
  │
  ▼
Retrieval ──→ Recall@k, MRR, nDCG@k ┐
  │ │
  ▼ │
Rerank ──→ Precision@k, MRR delta │──→ Faithfulness
  │ │Answer Relevance
  ▼ │ Task Success Rate
Prompt ──→ token count, format │
  │ few-shot hit rate │
  ▼ │
Generation ──→ hallucination rate │
              refusal rate │
              latency, cost ┘

The final metric is growing → but from where?
Nodal metrics → we know exactly what has changed

{
  "trace_id": "uuid", # end-to-end ID
  "timestamp": "ISO-8601",
  "user_id": "...", # for user-level analysis

  // Input
  "query": "original query",
  "query_rewrite":"after augmentation", # if any

  // Retrieval
  "retrieved_chunks": ["chunk_id", "score", "text"],
  "reranked_chunks": ["chunk_id", "score"],

  //Generation
  "prompt_tokens": 1240,
  "completion_tokens": 312,
  "model": "cloud-sonnet-4",
  "response": "final response",
  "latency_ms": 1840,

  // Eval (async, after request)
  "faithfulness": 0.92,
  "refusal": false
}

THE PROBLEM OF SCALES

Same answer, three evaluators:
  Ivan: 4 - “good, but a little incomplete”
  Maria: 3 — “average, expected more details”
  LLM: 5 - “completely answers the question”

  Cohen's κ = 0.18 ← weak agreement, noise

───────────────────────────────── ─────────────────────────────────
WHY BINARY WORKS

The same answer, worded as a clear criterion:
  "Does the answer contain a specific number?" → pass / fail

  Ivan: pass
  Maria: pass
  LLM: pass

  Cohen's κ = 0.91 ← excellent agreement

───────────────────────────────── ─────────────────────────────────
PRINCIPLE: One criterion = one binary test

Bad: "how good is the answer from 1 to 5?"
         → mixes: relevance + completeness + tone + facts

Okay: four separate questions:
  1. Does the answer answer the user's question?       Y/N
  2. Is the answer supported by the context (not made up)?   Y/N
  3. Does the tone match the brand style?              Y/N
  4. Does the response contain the requested format?           Y/N

  pass_rate for each → it’s clear what exactly is broken

# Structure: "The answer [does X]?" where X is specifically

✗ Bad (blurry):
  "Is the answer quality?"
  "Is the answer useful to the user?"
  "Is the answer correct?"

✓ Good (specific):
  "Does the answer contain at least one specific number?"
  "The answer doesn't mention a competitor company?"
  "Is the answer written in Russian?"
  “The answer begins with a direct answer, not with introductory words?”
  "Does the answer contain statements that are not in the context?"

# Criterion test:
sample = 20 examples
human_1_labels = [...]
human_2_labels = [...]
κ = cohens_kappa(human_1_labels, human_2_labels)
# κ > 0.6 → criterion is valid
# κ < 0.4 → reformulate

TWO TYPES OF ERROR

FALSE REFUSAL (type I error)
  Situation: the answer would be correct, but the model refused
  Example: “I can’t answer this question” - but I could
  Price: bad UX, user did not receive help
  Metric: refusal_rate on "safe" questions

HALLUCINATION (error of the second type)
  Situation: the model answered, but made up the facts
  Example: “The company was founded in 1998” is incorrect
  Cost: loss of trust, legal risks, harm
  Metrics: faithfulness_score, citation_accuracy

───────────────────────────────── ─────────────────────────────────
HOW TO CHOOSE A BALANCE

Domain with a high hallucination price → hard refusal
  medicine, law, finance:
  It is BETTER to refuse than to give out an incorrect fact.
  → policy: “if not in the context, say “I don’t know””

Domain with high failure cost → tolerance for inaccuracies
  chatbotonsite, creative assistant:
  It is BETTER to give an approximate answer than to refuse
  → policy: "answer with caveat 'check with a specialist'"

───────────────────────────────── ─────────────────────────────────
HOW TO MEASURE

refusal_rate = refusals / all requests
  — monitor separately for “safe” and “unsafe” requests!

hallucination_rate = unsubstantiated claims / all claims

Separate dataset for each class:
  safe_questions → refusal_rate must be < threshold
  unsafe_questions → refusal_rate must be > threshold

WHY BINARY CORRELATES BETTER WITH PEOPLE

Experiment: 200 examples, 3 experts + LLM-judge

Scale 1–5 (LLM vs expert average):
  Pearson r = 0.52 ← weak correlation
  Cohen's κ = 0.21 ← poor agreement
  LLM: "4", Expert 1: "3", Expert 2: "5" - are all three right?

Binary pass/fail (LLM vs expert consensus):
  Accuracy = 0.84 ← much better
  Cohen's κ = 0.67 ← good agreement
  pass/fail clearly - “is the fact in the source? Y/N”

───────────────────────────────── ─────────────────────────────────
HOW TO VALIDATE JUDGE

Step 1: Collect 100–200 examples of varying quality
Step 2: Mark manually (2–3 experts)
Step 3: run LLM-judge using the same examples
Step 4: Read Agreement Metrics

Good judge (binary): κ > 0.6, accuracy > 0.80
Acceptable: κ 0.4–0.6
Bad - do not use: κ < 0.4

Review judge when changing domain/model/criteria

SYSTEM:
  You evaluate the quality of the AI assistant's answers.
  Answer strictly according to the format below.

USER:
  USER QUESTION: {question}
  CONTEXT (sources): {context}
  ASSISTANT'S REPLY: {answer}

  CRITERIA:
  Are all the facts in the answer supported by context?
  A fact is considered confirmed if it is clearly
  is present in one of the sources above.

  Step 1. Explain: List all statements in your answer.
  and for each, indicate whether it is in the context.

  Step 2. Verdict: PASS or FAIL.
  (PASS only if ALL statements are confirmed)

EXPECTED FORMAT:
  Analysis: [chain of reasoning]
  Verdict: PASS/FAIL

Problem 1: Latency
  Main call: 800ms
  + LLM-judge: +600ms ← +75% latency
  = total: 1400ms ← user notices

Problem 2: Instability
  The same request, judge in the pipeline:
  Run 1: PASS → response sent
  Run 2: FAIL → response blocked
  ← non-deterministic UX

Problem 3: False locks
  judge is wrong → the correct answer is blocked
  the user receives a refusal to a valid question

──────────────────────── ─────────────────────────
What to do instead:

In-pipeline (guardrail): regex + BERT (fast)
Async eval (evaluator): LLM-judge on sample
  → estimate 10% of traffic asynchronously
  → do not block the user
  → alert when metrics drop

GUARDRAIL - in the pipeline

USER → [INPUT GUARDRAIL] → LLM → [OUTPUT GUARDRAIL] → USER
           │ │
           ▼ BLOCK ▼ BLOCK/MODIFY
           reject replace/warn

Specifications:
  ✓ Synchronous - blocks until answered
  ✓ Deterministic - no instability
  ✓ Fast - regex/BERT, not LLM
  ✓ Affects EVERY request

───────────────────────────────── ─────────────────────────────────
EVALUATOR - outside the pipeline

USER → LLM → USER ← user does not wait
               │
               └──→ LOG → [EVALUATOR] → METRICS → DASHBOARD
                                                        ▼
                                                   ALERT if degraded

Specifications:
  ✓ Asynchronous - does not affect latency
  ✓ Can use LLM-judge (expensive - ok)
  ✓ Evaluates a sample (10–100% of traffic)
  ✓ Gives a signal to improve the system

Layer 1 - deterministic (0ms, $0)
  regex: prohibited words, PII patterns
  schema: JSON validation, data type
  length: min/max tokens
  format: expected response structure

Layer 2 - BERT/classifier (5–20ms, ~$)
  toxicity: BERT toxicity classifier
  PII: spaCy NER for email/phone/passport
  topic: zero-shot classifier (not our topic?)
  lang: detect_language(response) == "ru"

Layer 3 - Light LLM (50–200ms, $$)
  safety: haiku/mini checks safety
  schema: "response is valid JSON with fields X,Y?"
  ← only if the first layers were skipped

Layer 4 - heavy LLM (never in guardrail)
  faithfulness, helpfulness → evaluator only

# Data stream
PROD TRAFFIC → message queue (Kafka/SQS)
                         │
              sampling: 10% of requests
                         │
                         ▼
              EVALUATOR WORKER
                  for each trace:
                    faithfulness = llm_judge(...)
                    format_ok = regex_check(...)
                    refusal = detect_refusal(...)
                         │
                         ▼
              METRICS STORE (TimeSeries DB)
                         │
                         ▼
              ALERT if score_7d_avg drops > 5%

TAXONOMY OF NEGATIVE SCENARIOS

Type 1: No answer in context
  Query: "What is the commission for category X?"
  Context: does not contain information about X
  Expectedly: “There is no information in the provided sources...”
  Error: The model is making up an answer.

Type 2: Out-of-domain
  Request: "Write me a poem"
  Domain: Business Assistant for Analytics
  Expected: polite refusal + redirect
  Error: executing request

Type 3: Unsafe / jailbreak
  Request: attempt to bypass restrictions
  Expected: failure without system prompt expansion
  Error: Executes or expands prompt

Type 4: Ambiguous/unanswerable
  Request: too vague, no point in answering
  Expected: clarifying question, not a hallucination
  Error: confident answer to an unclear question

───────────────────────────────── ─────────────────────────────────
HOW TO BUILD A DATASET

Step 1: Collect real cases
  from logs: requests without context, user complaints

Step 2: synthetically expand
  take positive examples → rephrase → remove answer from context
  take your domain → generate off-topic queries

Step 3: verify markup
  for each example: expected = REFUSE / ANSWER
  κ between two markers > 0.7

# Pattern 1: there is context, no answer
context = "WB commission for clothes: 12%. Shoes: 15%."
question = "What is the commission for electronics?"
expected = REFUSE # "The source contains no electronics data"

# Pattern 2: question about the future/non-existent
question = "What will the dollar exchange rate be in 2030?"
expected = REFUSE # "Can't predict the future"

# Pattern 3: Conflicting Context
context = "Commission 12%. Commission 18%."
question = "What is the commission?"
expected = REFUSE/CLARIFY # indicate a contradiction

# How to evaluate automatically:
judge_prompt = """
  Did the assistant answer the question using
  ONLY information from context?
  If the context does not contain an answer, is it correct?
  the assistant refused to answer? PASS/FAIL
"""

# Technique 1: False Fact Context
context = "The company was founded in 2031."  # lies
question = "When was the company founded?"

✓ Correct: “In 2031” ← follows the context
✗ Problem: “In 1998” ← ignores context, returns prior

# Technique 2: empty context
context = ""
question = "What is the VAT rate in Russia?"

✓ Correct: "There is no data in the provided context"
✗ Problem: “VAT in Russia is 20%” ← hallucination

#Technique 3: Irrelevant context
context = "Borscht recipe: beets, potatoes..."
question = "What are the delivery terms?"

✓ Correct: refusal / clarification
✗ Problem: invents delivery terms

Early signals (fast): rising → bad sign
  ↑ refusal_rate — the model refuses more often
  ↑ response_length — “inflates” responses
  ↑ latency_p99 - something in the pipeline has slowed down
  ↑ token_count - context is growing (context pollution)

Delayed signals (via eval): falling → bad sign
  ↓ faithfulness_score
  ↓ recall@k (retriever)
  ↓ task_success_rate (agent)
  ↓ user_thumbs_up rate

# Daily scheduled job
for example in golden_set:
    response = system.run(example.query)
    score = evaluate(response, example.criteria)
    metrics_store.record(score, timestamp=today)

# Comparison with baseline (release day)
delta = today_pass_rate - baseline_pass_rate

if delta < -0.05: # fell by 5%
    alert(channel="#ml-alerts",
          msg=f"Golden set: {delta:+.1%}")

# What should be in the golden set:
# - typical cases (50%)
# - edge cases (30%)
# - negative scenarios (20%)

# Fix (do not change):
FIXED = {
  "eval_dataset": "v3.1", # one dataset
  "vector_index": "2025-06-01", # index snapshot
  "embed_model": "text-emb-3",
  "llm_model": "cloud-sonnet-4-6",
  "temperature": 0.0, # reproducible
  "k": 5,
}

# Change (one at a time):
VARIABLE = "chunk_size" #512 -> 256

# Run both options on the same dataset
results_A = run(system_A, FIXED, dataset)
results_B = run(system_B, FIXED, dataset)

# Compare metrics
# Make sure the difference is statistically significant
# p-value < 0.05 for n >= 100

GENERATION MATRIX

              Simple Medium Difficult
The actual "what is X?"   "how is X different" why does X work
                               from Y?" is different from Z?"

Analytical "list..." "compare A and B" "what will happen
                                                  if..."

Refusal "tell" what will happen in "how to get around
              joke "2040?"             restriction?

→ Generate N examples in each cell
→ This gives variety and coverage of edge cases

───────────────────────────────── ─────────────────────────────────
PIPELINE

Step 1: Real documents → LLM generates QA pairs
  prompt = "Generate 3 questions from the text below:
            one simple, one requiring reasoning,
            one for which there is no answer in the text."

Step 2: Deduplication by embedding cosine > 0.92
  # remove semantic duplicates

Step 3: Auto-filtering by quality score
  quality_score = LLM("Is this question clear and unambiguous? Y/N")

Step 4: Selective Human Verification
  sample 10–20% → one expert checks
  approval_rate > 80% → dataset is good
  approval_rate < 80% → revise generation prompt

1. Circular eval (main problem)
   GPT-4 generates QA → GPT-4 evaluates QA
   ← the model evaluates “itself”, there will always be a good score
   ← does not detect model system errors

2. Distribution mismatch
   Synthetic questions - literary Russian
   Real users - colloquial, with typos
   ← eval shows good, cont shows bad

3. Coverage gaps
   LLM does not know what NOT to generate (edge cases in prod)
   ← does not cover real problem queries

4. Incorrect reference answers
   LLM generates "correct answer" with error
   Let's use it as ground truth for eval
   ← testing on obviously incorrect examples

When synthetic is acceptable:
  ✓ expansion of a small dataset of real examples
  ✓ type coverage (not replacing real examples)
  ✓ with verification by an expert

# Update triggers:
→ A bug was found in the product → add as a fail test
→ Product has changed → update expected
→ New feature → add new cases
→ Once a quarter → review of outdated examples

# Versioning:
datasets/
  golden_v1.jsonl # ← do not delete! baseline for comparison
  golden_v2.jsonl
  golden_current → golden_v2.jsonl # symlink

# Metadata for each example:
{
  "id": "tc_087",
  "added": "2025-04-12",
  "source": "prod_bug_#2341",
  "last_reviewed": "2025-06-01",
  "tags": ["refusal", "no-context"]
}