LLM Observability & Evaluations

trace_id: a3f9... total: 4.2s · 3,147 tokens · $0.0089

├── span: query_rewrite 0ms → 380ms · 212 tok
│ input: “why the pump doesn’t work”
│ output: "reasons for centrifugal pump failure"
│
├── span: vector_search 380ms → 510ms · 0 tok
│ results: 12 chunks, top_score: 0.847
│
├── span: rerank 510ms → 890ms · 0 tok
│ input: 12 → output: 5 chunks
│
└── span: generate_answer 890ms → 4200ms · 2935 tok
      model: cloud-sonnet finish: stop

{
  span_id, trace_id, parent_span_id,
  step_name: "rerank",
  model: "cloud-sonnet-4",
  prompt_tokens: 1240,
  completion_tokens: 312,
  latency_ms: 780,
  status: "success" | "error",
  input: {truncated to 10k chars},
  output: {truncated to 10k chars},
  // custom:
  user_id, session_id, product_line,
  num_retrieved_docs, top_retrieval_score
}

session (user_id + session_id)
 ├── trace (turn 1)  → spans...
 ├── trace (turn 2)  → spans...
 └── trace (turn 3)  → spans...

session_metrics:
  avg_turns_to_resolution: 2.3
  total_cost_per_session:  $0.042

if status == "error":       sample_rate = 1.0
elif latency_ms > 5000:    sample_rate = 1.0
elif score_low_quality:    sample_rate = 1.0
elif user_flagged:         sample_rate = 1.0
else:                      sample_rate = 0.05  // 5%

Metric Norm Alert if
────────────────────────── ───────────────────────────
p50 latency (e2e) < 2s > 5s
p99 latency (e2e) < 8s > 15s
token / request < 3000 > 8000
cost/request < $0.01 > $0.05
error rate < 0.5% > 2%
retry rate < 2% > 10%
avg quality score > 0.75 < 0.60
hallucination rate < 3% > 8%
retrieval recall@5 > 0.80 < 0.65
rerank MRR > 0.70 < 0.55

TTFT = time_to_first_token // UX metric
TGT = total_generation_time // model
Latency breakdown:
  embed: 45ms █░
  search: 120ms ████░
  rerank: 350ms █████████░ ← bottleneck
  generate: 2100ms ████████████████████

data drift: distribution of incoming requests
concept drift: correct answers have changed
model drift: model behavior after update
prompt drift: prompt performs worse due to context changes

Detect: rolling window 7d vs baseline 30d
  KL-divergence on query embeddings
  Mann-Whitney on score metrics

copy_to_clipboard → answer is helpful
no_followup_in_30s → question closed
session_ended → task completed
immediate_rephrase → response not understood
thumbs_down → obvious signal of poor quality
rage_click / abandon → frustration

len(response) < 20 → flag "too_short"
len(response) > 4000 → flag "too_long"
"I cannot" in response → "refusal" flag
cosine(response, query) < 0.3 → flag "off_topic"
repetition_ratio > 0.4 → "degenerate" flag

Method Who evaluates Speed Cost Reliability
────────────────────────────────── ───────────────────────────────────
Reference-based metric vs GT ms $0 average
LLM-as-Judge GPT-4/Claude seconds $$ high
Human eval markers days $$$ standard
Model-based classifier fine-tuned ms $0 high*
Heuristic / rule code ms $0 low

* after training on human tags

Extraction: Exact Match, F1 token overlap
Summarization: ROUGE-L, BERTScore
Translation: BLEU, ChrF
Generation: Semantic similarity (cos)
Classification: Accuracy, F1, MCC

// BERTScore is better than BLEU - takes into account the meaning

✗ “Rate the answer from 1 to 10” → subjective
✓ Rubrics with specific criteria → objectively

✗ One challenge, one result → unstable
✓ 3 runs → majority vote → reliable

✗ A is always the first in the pair → position bias
✓ Swap A↔B, average → neutral

✗ Score without explanation → opaque
✓ CoT → explanation → score → debuggable

Faithfulness response supported by context?
                       // detect hallucinations
Answer Relevance is the answer relevant to the question?
                       // detect off-topic
Context Precision found chunks useful?
                       // retrieval quality
Context Recall have all the required chunks been found?
                       // requires GT

prompt_v1 vs prompt_v2 → judge: "v2 is better"
prompt_v2 vs prompt_v3 → judge: "v2 is better"
prompt_v3 vs prompt_v1 → judge: "v3 is better"

ELO: v2=1250, v3=1180, v1=1070
// → select v2 for deployment

MFT (Minimum Functionality):
  "answer a simple question" → pass/fail

INV (Invariance):
  "the result does not change when changing names"
  Ivan bought → Anna bought → same intent

DIR (Directional):
  "adding negativity reduces sentiment"
  "good" → score 0.9
  "not very good" → score < 0.9 ✓

[
  {
    id: "tc_001",
    input: {query: "...", context: "..."},
    expected_output: "...", // or null
    eval_criteria: [
      "contains a specific number"
      "does not mention competitors"
      "answers in Russian"
    ],
    tags: ["edge_case", "pricing"],
    added_because: "bug from 2024-11-03"
  }
]

git push prompt_v2
  → CI trigger
  → run_eval(dataset, prompt_v2)
  → compare(prompt_v2.scores, baseline.scores)
  → if regression > 5%: BLOCK merge
  → if improvement > 3%: AUTO APPROVE
  → else: REQUEST human review

smoke tests: 5-10 critical cases, <30 sec
regression suite: 50-200 cases, run on PR
full eval: 500+ cases, night run
adversarial: attempts to break, jailbreak

for doc in knowledge_base:
    qa_pairs = LLM(f"""
        From the document below, generate 3 question-answer pairs.
        Include: simple, complex, edge case.
        Document: {doc}
        Format: JSON [{{"q":..., "a":...}}]
    """)
    dataset.extend(qa_pairs)

request
  ├── prod_pipeline(request) → user ✓
  └── shadow_pipeline(request) → /dev/null
                                → logs + metrics

after N requests:
  compare(prod.metrics, shadow.metrics)
  → decision to deploy

Day 1: 1% traffic → 24h monitoring
Day 2: 5% traffic → 24h monitoring
Day 3: 20% traffic → 24h monitoring
Day 5: 50% traffic → monitoring 48h
Day 7: 100% → prompt_v2 = new baseline

auto-rollback if:
  error_rate > 3% OR quality_score drop > 10%

1. Determine the victory metric BEFORE the test
2. Calculate the required sample size (power=0.8)
3. Run → wait for N=calculated requests
4. Conduct t-test / chi-square
5. p < 0.05 → deploy winner
   p > 0.05 → no difference, keep the old one

config = feature_flags.get(user_id, {
  "model": "cloud-sonnet-4",
  "prompt_ver": "v3.2",
  "temperature": 0.3,
  "rerank": True,
  "top_k": 5
})
// change without re-deploying the service

watch(metrics, window=5min):
  if error_rate > threshold:      rollback()
  if p99_latency > threshold:     rollback()
  if avg_quality_score < floor:  rollback()
  if refusal_rate spike:          rollback()

rollback() → feature_flag.set(prev_version)
           → alert(oncall)

Metric What it does the Formula count?
───────────────────────────────── ─────────────────────────────────
Precision@k share of relevant ones among top-k |rel ∩ top-k| /k
Recall@k share of those found among all relevant ones |rel ∩ top-k| / |rel|
MRR how high is the first relevant 1/rank_first_rel
MAP average precision for all relevant avg(precision@k for rel)
NDCG@k takes into account the order and degree of DCG/IDCG relativity

for chunk in index:
    score = LLM(f"""
        Rate the chunk according to 3 criteria (0-1):
        1. Self-sufficiency (understandable without context?)
        2. Information content (are there specific facts?)
        3. Clean (no garbage, headers, artifacts?)
        Chunk: {chunk}
    """)
    if score < 0.5: flag_for_rechunking(chunk)

test: pass context with intentionally false fact
  "The Eiffel Tower was built in 1995"

if LLM answers "in 1889" → ignores context
if LLM answers "in 1995" → follows context ✓

// real-time: citation tracking
claims_in_answer → attribution_check → source_chunks

answer → LLM decompose → [claim1, claim2, claim3]

for claim in claims:
    evidence = search(claim, in=source_docs)
    claim = nli(evidence, claim)
    // entail=1, neutral=0.5, contradict=0

groundedness = mean(entailment_scores)
// 0.9+ excellent, 0.7-0.9 acceptable, <0.7 problem

Binary: problem solved / not solved
Partial: 0.0 – 1.0 (percentage of completed steps)
Weighted: critical steps are more important than minor ones

Example for code agent:
  does the code compile?     → +0.3
  are the tests passing?        → +0.4
  Does it match the spec?   → +0.3

steps_taken: 5 (optimal: 3) → unnecessary steps: 2
tool_accuracy: the correct tool is selected 4/5 times
backtrack_rate: agent changed plan 2 times
redundant_calls: 1 extra call search
efficiency_score: optimal_steps / actual_steps = 0.6

production: agent → real_db, real_api, real_email
testing: agent → mock_db, mock_api, mock_email
                       ↓ ↓ ↓
                    records all calls with arguments

assert called_with("SELECT", table="products")
assert not called("DELETE")
assert email_sent_to == "test@sandbox.com"

recent_actions = deque(maxlen=10)

def check_loop(action):
    if recent_actions.count(action) >= 3:
        raise AgentLoopError(f"Loop: {action} x3")
    if len(all_actions) > MAX_STEPS:
        raise AgentStuckError("Step limit exceeded")
    recent_actions.append(action)

step input_tok output_tok $cost
────────────────────── ───────────────────────
query_rewrite 120 45 $0.0001
generate_answer 2800 420 $0.0098 ← main
validation 650 80 $0.0020
total 3570 545 $0.0119

→ 82% of the cost in generate_answer → optimize there

cost_per_user_per_day = sum(costs, by=user_id) / DAU
cost_per_feature = sum(costs, by=feature)

insights:
  "feature X consumes 60% of the budget, but 10% of the users"
  "The Free plan costs us $0.8/month per user"
  "enterprise plan pays off x4"

per_request: max_tokens_input = 8000
              max_tokens_output = 2000
per_session: max_cost = $0.50
per_user_day: max_cost = $2.00
per_hour: max_requests = 100

if exceeded:
  → truncate context (softly)
  → rate limit (hard)
  → oncall alert (critical)