Evaluation Process & LLM-as-Judge

# After any change:
sample = random_sample(outputs, n=30)
# or: all fails from eval for the period

# For each read:
✓ Is the answer correct in meaning?
✓ Is the answer based on context or fictitious?
✓ Is the formatting as expected?
✓ Is the answer too long/short/evasive?
✓ Does the model respond when it should fail?
✓ Does the model refuse when it should respond?

# Group errors by type:
hallucination |  wrong_refusal
off_topic |  format_error
incomplete |  tone_mismatch

# Most common category → fix first

{
  "trace_id": "uuid", # end-to-end ID
  "timestamp": "ISO-8601",
  "user_id": "...", # for user-level analysis

  // Input
  "query": "original query",
  "query_rewrite":"after augmentation", # if any

  // Retrieval
  "retrieved_chunks": ["chunk_id", "score", "text"],
  "reranked_chunks": ["chunk_id", "score"],

  //Generation
  "prompt_tokens": 1240,
  "completion_tokens": 312,
  "model": "claude-sonnet-4",
  "response": "final response",
  "latency_ms": 1840,

  // Eval (async, after request)
  "faithfulness": 0.92,
  "refusal": false
}

# Structure: "The answer [does X]?" where X is specifically

✗ Bad (blurry):
  "Is the answer quality?"
  "Is the answer useful to the user?"
  "Is the answer correct?"

✓ Good (specific):
  "Does the answer contain at least one specific number?"
  "The answer doesn't mention a competitor company?"
  "Is the answer written in Russian?"
  “The answer begins with a direct answer, not with introductory words?”
  "Does the answer contain statements that are not in the context?"

# Criterion test:
sample = 20 examples
human_1_labels = [...]
human_2_labels = [...]
κ = cohens_kappa(human_1_labels, human_2_labels)
# κ > 0.6 → criterion is valid
# κ < 0.4 → reformulate

SYSTEM:
  You evaluate the quality of the AI assistant's answers.
  Answer strictly according to the format below.

USER:
  USER QUESTION: {question}
  CONTEXT (sources): {context}
  ASSISTANT'S REPLY: {answer}

  CRITERIA:
  Are all the facts in the answer supported by context?
  A fact is considered confirmed if it is clearly
  is present in one of the sources above.

  Step 1. Explain: List all statements in your answer.
  and for each, indicate whether it is in the context.

  Step 2. Verdict: PASS or FAIL.
  (PASS only if ALL statements are confirmed)

EXPECTED FORMAT:
  Analysis: [chain of reasoning]
  Verdict: PASS/FAIL

Problem 1: Latency
  Main call: 800ms
  + LLM-judge: +600ms ← +75% latency
  = total: 1400ms ← user notices

Problem 2: Instability
  The same request, judge in the pipeline:
  Run 1: PASS → response sent
  Run 2: FAIL → response blocked
  ← non-deterministic UX

Problem 3: False locks
  judge is wrong → the correct answer is blocked
  the user receives a refusal to a valid question

──────────────────────── ─────────────────────────
What to do instead:

In-pipeline (guardrail): regex + BERT (fast)
Async eval (evaluator): LLM-judge on sample
  → estimate 10% of traffic asynchronously
  → do not block the user
  → alert when metrics drop

Layer 1 - deterministic (0ms, $0)
  regex: prohibited words, PII patterns
  schema: JSON validation, data type
  length: min/max tokens
  format: expected response structure

Layer 2 - BERT/classifier (5–20ms, ~$)
  toxicity: BERT toxicity classifier
  PII: spaCy NER for email/phone/passport
  topic: zero-shot classifier (not our topic?)
  lang: detect_language(response) == "ru"

Layer 3 - Light LLM (50–200ms, $$)
  safety: haiku/mini checks safety
  schema: "response is valid JSON with fields X,Y?"
  ← only if the first layers were skipped

Layer 4 - heavy LLM (never in guardrail)
  faithfulness, helpfulness → evaluator only

# Data stream
PROD TRAFFIC → message queue (Kafka/SQS)
                         │
              sampling: 10% of requests
                         │
                         ▼
              EVALUATOR WORKER
                  for each trace:
                    faithfulness = llm_judge(...)
                    format_ok = regex_check(...)
                    refusal = detect_refusal(...)
                         │
                         ▼
              METRICS STORE (TimeSeries DB)
                         │
                         ▼
              ALERT if score_7d_avg drops > 5%

# Pattern 1: there is context, no answer
context = "WB commission for clothes: 12%. Shoes: 15%."
question = "What is the commission for electronics?"
expected = REFUSE # "The source contains no electronics data"

# Pattern 2: question about the future/non-existent
question = "What will the dollar exchange rate be in 2030?"
expected = REFUSE # "Can't predict the future"

# Pattern 3: Conflicting Context
context = "Commission 12%. Commission 18%."
question = "What is the commission?"
expected = REFUSE/CLARIFY # indicate a contradiction

# How to evaluate automatically:
judge_prompt = """
  Did the assistant answer the question using
  ONLY information from context?
  If the context does not contain an answer, is it correct?
  the assistant refused to answer? PASS/FAIL
"""

# Technique 1: False Fact Context
context = "The company was founded in 2031."  # lies
question = "When was the company founded?"

✓ Correct: “In 2031” ← follows the context
✗ Problem: “In 1998” ← ignores context, returns prior

# Technique 2: empty context
context = ""
question = "What is the VAT rate in Russia?"

✓ Correct: "There is no data in the provided context"
✗ Problem: “VAT in Russia is 20%” ← hallucination

#Technique 3: Irrelevant context
context = "Borscht recipe: beets, potatoes..."
question = "What are the delivery terms?"

✓ Correct: refusal / clarification
✗ Problem: invents delivery terms

Early signals (fast): rising → bad sign
  ↑ refusal_rate — the model refuses more often
  ↑ response_length — “inflates” responses
  ↑ latency_p99 - something in the pipeline has slowed down
  ↑ token_count - context is growing (context pollution)

Delayed signals (via eval): falling → bad sign
  ↓ faithfulness_score
  ↓ recall@k (retriever)
  ↓ task_success_rate (agent)
  ↓ user_thumbs_up rate

# Daily scheduled job
for example in golden_set:
    response = system.run(example.query)
    score = evaluate(response, example.criteria)
    metrics_store.record(score, timestamp=today)

# Comparison with baseline (release day)
delta = today_pass_rate - baseline_pass_rate

if delta < -0.05: # fell by 5%
    alert(channel="#ml-alerts",
          msg=f"Golden set: {delta:+.1%}")

# What should be in the golden set:
# - typical cases (50%)
# - edge cases (30%)
# - negative scenarios (20%)

# Fix (do not change):
FIXED = {
  "eval_dataset": "v3.1", # one dataset
  "vector_index": "2025-06-01", # index snapshot
  "embed_model": "text-emb-3",
  "llm_model": "claude-sonnet-4-6",
  "temperature": 0.0, # reproducible
  "k": 5,
}

# Change (one at a time):
VARIABLE = "chunk_size" #512 -> 256

# Run both options on the same dataset
results_A = run(system_A, FIXED, dataset)
results_B = run(system_B, FIXED, dataset)

# Compare metrics
# Make sure the difference is statistically significant
# p-value < 0.05 for n >= 100

1. Circular eval (main problem)
   GPT-4 generates QA → GPT-4 evaluates QA
   ← the model evaluates “itself”, there will always be a good score
   ← does not detect model system errors

2. Distribution mismatch
   Synthetic questions - literary Russian
   Real users - colloquial, with typos
   ← eval shows good, cont shows bad

3. Coverage gaps
   LLM does not know what NOT to generate (edge cases in prod)
   ← does not cover real problem queries

4. Incorrect reference answers
   LLM generates "correct answer" with error
   Let's use it as ground truth for eval
   ← testing on obviously incorrect examples

When synthetic is acceptable:
  ✓ expansion of a small dataset of real examples
  ✓ type coverage (not replacing real examples)
  ✓ with verification by an expert

# Update triggers:
→ A bug was found in the product → add as a fail test
→ Product has changed → update expected
→ New feature → add new cases
→ Once a quarter → review of outdated examples

# Versioning:
datasets/
  golden_v1.jsonl # ← do not delete! baseline for comparison
  golden_v2.jsonl
  golden_current → golden_v2.jsonl # symlink

# Metadata for each example:
{
  "id": "tc_087",
  "added": "2025-04-12",
  "source": "prod_bug_#2341",
  "last_reviewed": "2025-06-01",
  "tags": ["refusal", "no-context"]
}