--- type: "new-runtime-knowledge-guide" stable_id: "knowledge_guide:llm-observability-evals" version: 1 generated_at: "2026-07-23" record_date: "2026-07-23" date_kind: "generated_at" slug: "llm-observability-evals" title: "LLM Observability & Evaluations" description: "How to understand what is happening inside the pipeline - and what exactly is broken. Tracing, metrics, automatic evaluations, testing prompts and secure deployment of changes." retrieval_nugget: "How to understand what is happening inside the pipeline - and what exactly is broken. Tracing, metrics, automatic evaluations, testing prompts and secure deployment of changes. Trace — the full path of one request through the system. Span is an atomic step inside a trace (one LLM call, one search, one tool)." track: "evaluation-and-improvement" volume: 2 --- # LLM Observability & Evaluations ## Retrieval answer How to understand what is happening inside the pipeline - and what exactly is broken. Tracing, metrics, automatic evaluations, testing prompts and secure deployment of changes. Trace — the full path of one request through the system. Span is an atomic step inside a trace (one LLM call, one search, one tool). ## 01. Distributed Tracing ### 01.01 Trace / Span - anatomy of the observed pipeline Trace — the full path of one request through the system. Span is an atomic step inside a trace (one LLM call, one search, one tool). Each span contains: start/end time, tokens, input/output data, status, metadata. Without this, debugging a multi-step pipeline is like looking at a black box. #### Diagram ```text trace_id: a3f9... total: 4.2s · 3,147 tokens · $0.0089 ├── span: query_rewrite 0ms → 380ms · 212 tok │ input: “why the pump doesn’t work” │ output: "reasons for centrifugal pump failure" │ ├── span: vector_search 380ms → 510ms · 0 tok │ results: 12 chunks, top_score: 0.847 │ ├── span: rerank 510ms → 890ms · 0 tok │ input: 12 → output: 5 chunks │ └── span: generate_answer 890ms → 4200ms · 2935 tok model: claude-sonnet finish: stop ``` #### Tools - Langfuse - LangSmith - Arize Phoenix - Weights & Biases Traces - Helicone ### 01.02 Span Instrumentation - what exactly to log For each LLM call, you need to log a minimum set of data in order to later reproduce and debug any request. Plus custom fields for business context. #### Minimum set of span fields ```text { span_id, trace_id, parent_span_id, step_name: "rerank", model: "claude-sonnet-4", prompt_tokens: 1240, completion_tokens: 312, latency_ms: 780, status: "success" | "error", input: {truncated to 10k chars}, output: {truncated to 10k chars}, // custom: user_id, session_id, product_line, num_retrieved_docs, top_retrieval_score } ``` ### 01.03 Session / Thread - tracing of multi-pass dialogues Several traces are combined into a session - the entire user dialogue. This allows you to see: how many moves before the solution, where the user clarified the question, which turn was expensive. Critical for chat products. #### Hierarchy ```text session (user_id + session_id) ├── trace (turn 1) → spans... ├── trace (turn 2) → spans... └── trace (turn 3) → spans... session_metrics: avg_turns_to_resolution: 2.3 total_cost_per_session: $0.042 ``` #### Applicable - chatbots - support assistants - conversion funnels ### 01.04 Sampling Strategy - don't log everything When traffic is high, logging 100% trace is expensive. Strategies: head-based sampling (random%), tail-based (log only errors and slow requests), adaptive (more logs for anomalies). #### Sampling rules ```python if status == "error": sample_rate = 1.0 elif latency_ms > 5000: sample_rate = 1.0 elif score_low_quality: sample_rate = 1.0 elif user_flagged: sample_rate = 1.0 else: sample_rate = 0.05 // 5% ``` #### Applicable - high-load production - cost control ## 02. Metrics & Monitoring ### 02.05 Golden Signals for LLM systems Adaptation of classic SRE golden signals (latency, traffic, errors, saturation) to the specifics of LLM pipelines. Each step of the pipeline should have its own metrics - not just the final answer. #### Diagram ```text Metric Norm Alert if ────────────────────────── ─────────────────────────── p50 latency (e2e) < 2s > 5s p99 latency (e2e) < 8s > 15s token / request < 3000 > 8000 cost/request < $0.01 > $0.05 error rate < 0.5% > 2% retry rate < 2% > 10% avg quality score > 0.75 < 0.60 hallucination rate < 3% > 8% retrieval recall@5 > 0.80 < 0.65 rerank MRR > 0.70 < 0.55 ``` ### 02.06 Latency Breakdown step by step Measure the latency of each step separately. TTFT (Time To First Token) - separately. This allows you to understand where the pipeline actually slows down: in search, in rerank, in generation or in the network. #### Latency decomposition ```text TTFT = time_to_first_token // UX metric TGT = total_generation_time // model Latency breakdown: embed: 45ms █░ search: 120ms ████░ rerank: 350ms █████████░ ← bottleneck generate: 2100ms ████████████████████ ``` ### 02.07 Drift Detection - degradation over time The quality of the LLM pipeline degrades imperceptibly: the data, the behavior of the model after the update, and the distribution of requests change. We need rolling monitoring of metrics with alerts for statistically significant deviations. #### What's drifting ```text data drift: distribution of incoming requests concept drift: correct answers have changed model drift: model behavior after update prompt drift: prompt performs worse due to context changes Detect: rolling window 7d vs baseline 30d KL-divergence on query embeddings Mann-Whitney on score metrics ``` #### Applicable - long-running products - after API model updates ### 02.08 Implicit User Signals User behavior as a proxy quality metric. There is no need to explicitly ask for an assessment—whether they copy the answer, ask a clarifying question right away, or return to the session. This is ground truth without markup. #### Signals and what they mean ```text copy_to_clipboard → answer is helpful no_followup_in_30s → question closed session_ended → task completed immediate_rephrase → response not understood thumbs_down → obvious signal of poor quality rage_click / abandon → frustration ``` #### Applicable - B2C products - scalable eval without markers ### 02.09 Anomaly Detection on spikes Automatically flag anomalous responses: too short, too long, with unexpected patterns (refused, repeated, off-topic). Response embedding + OOD detection. #### Simple heuristics ```text len(response) < 20 → flag "too_short" len(response) > 4000 → flag "too_long" "I cannot" in response → "refusal" flag cosine(response, query) < 0.3 → flag "off_topic" repetition_ratio > 0.4 → "degenerate" flag ``` ## 03. Evaluation Methods ### 03.10 Taxonomy of assessment methods Methods for evaluating LLM systems are divided along three axes: who evaluates, when to evaluate, and what exactly is evaluated. Each method has its own trade-offs in terms of cost, speed and reliability. #### Diagram ```bash Method Who evaluates Speed Cost Reliability ────────────────────────────────── ─────────────────────────────────── Reference-based metric vs GT ms $0 average LLM-as-Judge GPT-4/Claude seconds $$ high Human eval markers days $$$ standard Model-based classifier fine-tuned ms $0 high* Heuristic / rule code ms $0 low * after training on human tags ``` ### 03.11 Reference-based Metrics Comparing the output with the ground truth answer using deterministic metrics. Fast and cheap, but requires a dataset with correct answers and does not work well for open-ended tasks. #### Metrics by task ```text Extraction: Exact Match, F1 token overlap Summarization: ROUGE-L, BERTScore Translation: BLEU, ChrF Generation: Semantic similarity (cos) Classification: Accuracy, F1, MCC // BERTScore is better than BLEU - takes into account the meaning ``` #### Applicable - regression tests - problems with a clear answer ### 03.12 LLM-as-Judge - correct implementation The LLM judge works reliably only with the right configuration: a clear rubric, a scale with examples, CoT before the assessment, neutralization of position bias. Without this, estimates are unstable and biased. #### Anti-patterns and fixes ```sql ✗ “Rate the answer from 1 to 10” → subjective ✓ Rubrics with specific criteria → objectively ✗ One challenge, one result → unstable ✓ 3 runs → majority vote → reliable ✗ A is always the first in the pair → position bias ✓ Swap A↔B, average → neutral ✗ Score without explanation → opaque ✓ CoT → explanation → score → debuggable ``` ### 03.13 RAGAS - eval framework for RAG A specialized set of metrics for RAG systems: evaluates not only the final response, but also the quality of the retrieval separately. Automatically through LLM - does not require GT responses for some metrics. #### 4 RAGAS metrics ```text Faithfulness response supported by context? // detect hallucinations Answer Relevance is the answer relevant to the question? // detect off-topic Context Precision found chunks useful? // retrieval quality Context Recall have all the required chunks been found? // requires GT ``` #### Applicable - RAG QA - knowledge bases - document assistants ### 03.14 Pairwise / Comparative Eval Instead of an absolute assessment, a comparison of two options: “which answer is better, A or B?” It is much easier for people and LLM judges to make comparisons than to give absolute marks. Used to compare prompts/models. #### ELO rating of prompts ```sql prompt_v1 vs prompt_v2 → judge: "v2 is better" prompt_v2 vs prompt_v3 → judge: "v2 is better" prompt_v3 vs prompt_v1 → judge: "v3 is better" ELO: v2=1250, v3=1180, v1=1070 // → select v2 for deployment ``` #### Applicable - prompt selection - model selection - A/B test of configs ### 03.15 Behavioral Testing - CheckList approach A set of specific behavioral tests: “the model must X given Y.” An analogue of unit tests for NLP (CheckList, Microsoft). Tests specific abilities rather than general quality. #### Types of tests ```text MFT (Minimum Functionality): "answer a simple question" → pass/fail INV (Invariance): "the result does not change when changing names" Ivan bought → Anna bought → same intent DIR (Directional): "adding negativity reduces sentiment" "good" → score 0.9 "not very good" → score < 0.9 ✓ ``` #### Applicable - regression suite - edge case coverage ## 04. Prompt Testing & CI/CD ### 04.16 Golden Dataset is the foundation of everything A labeled dataset of inputs with expected outputs or evaluation criteria. It is built iteratively: start with 20-50 cases covering the main scenarios, add new edge cases when found. #### Dataset structure ```sql [ { id: "tc_001", input: {query: "...", context: "..."}, expected_output: "...", // or null eval_criteria: [ "contains a specific number" "does not mention competitors" "answers in Russian" ], tags: ["edge_case", "pricing"], added_because: "bug from 2024-11-03" } ] ``` ### 04.17 Prompt CI - tests for every change Changing a prompt is a change in code. Before merging into production: run the golden dataset, compare metrics with baseline, block deployment during regression. Prompts are stored in git with versioning. #### Pipeline in CI ```bash git push prompt_v2 → CI trigger → run_eval(dataset, prompt_v2) → compare(prompt_v2.scores, baseline.scores) → if regression > 5%: BLOCK merge → if improvement > 3%: AUTO APPROVE → else: REQUEST human review ``` #### Tools - Langfuse Experiments - PromptFlow - Braintrust - GitHub Actions ### 04.18 Regression Suite - protection against degradation A set of tests that capture current "good" behavior and automatically check that it hasn't broken. It is updated every time a bug is found in the product. #### Regression layers ```text smoke tests: 5-10 critical cases, <30 sec regression suite: 50-200 cases, run on PR full eval: 500+ cases, night run adversarial: attempts to break, jailbreak ``` ### 04.19 Synthetic Eval Data Generation The LLM generates test cases to evaluate another LLM. From the documents - questions + answers. Allows you to quickly create a dataset without manual marking. Quality must be verified selectively manually. #### Generation scheme ```sql for doc in knowledge_base: qa_pairs = LLM(f""" From the document below, generate 3 question-answer pairs. Include: simple, complex, edge case. Document: {doc} Format: JSON [{{"q":..., "a":...}}] """) dataset.extend(qa_pairs) ``` #### Applicable - start without markers - RAG testing - RAGAS ### 04.20 Shadow Mode - testing on real traffic The new version of the pipeline receives the same requests as the production version, generates responses, but does not give them to the user. Metrics of two versions are compared on the same traffic without risk to users. #### Scheme ```text request ├── prod_pipeline(request) → user ✓ └── shadow_pipeline(request) → /dev/null → logs + metrics after N requests: compare(prod.metrics, shadow.metrics) → decision to deploy ``` #### Applicable - model change - major pipeline refactor ## 05. Safe Deployment ### 05.21 Canary Deployment prompts The new prompt is rolled out to a small percentage of traffic. With normal metrics, it gradually expands. In case of an anomaly - instant rollback. This is the safest way to deploy changes to LLM systems. #### Rollout stages ```text Day 1: 1% traffic → 24h monitoring Day 2: 5% traffic → 24h monitoring Day 3: 20% traffic → 24h monitoring Day 5: 50% traffic → monitoring 48h Day 7: 100% → prompt_v2 = new baseline auto-rollback if: error_rate > 3% OR quality_score drop > 10% ``` ### 05.22 A/B Test - statistically correct Random division of traffic between versions, measurement of business metrics, statistical testing of significance before making a decision. You can’t stop an A/B test prematurely - p-hacking. #### The right process ```text 1. Determine the victory metric BEFORE the test 2. Calculate the required sample size (power=0.8) 3. Run → wait for N=calculated requests 4. Conduct t-test / chi-square 5. p < 0.05 → deploy winner p > 0.05 → no difference, keep the old one ``` ### 05.23 Feature Flags for prompts/models Prompts and model configurations are stored in the feature flag system, not in the code. Switching occurs without deployment - runtime. You can target by user_id, region, plan, experiment_group. #### Config at runtime ```text config = feature_flags.get(user_id, { "model": "claude-sonnet-4", "prompt_ver": "v3.2", "temperature": 0.3, "rerank": True, "top_k": 5 }) // change without re-deploying the service ``` #### Tools - LaunchDarkly - Unleash - Langfuse Prompt Management ### 05.24 Automated Rollback The system automatically rolls back to the previous version of the prompt/config when alerts are triggered. No manual intervention. Critical for night deployments and high-load systems. #### Automatic rollback triggers ```python watch(metrics, window=5min): if error_rate > threshold: rollback() if p99_latency > threshold: rollback() if avg_quality_score < floor: rollback() if refusal_rate spike: rollback() rollback() → feature_flag.set(prev_version) → alert(oncall) ``` ## 06. RAG-specific Evaluation ### 06.25 Retrieval Metrics - evaluation of the search part separately It is critically important to evaluate retrieval regardless of generation - otherwise you will not understand where the problem is: in search or in LLM. To do this, you need a dataset with known relevant documents for each question. #### Diagram ```text Metric What it does the Formula count? ───────────────────────────────── ───────────────────────────────── Precision@k share of relevant ones among top-k |rel ∩ top-k| /k Recall@k share of those found among all relevant ones |rel ∩ top-k| / |rel| MRR how high is the first relevant 1/rank_first_rel MAP average precision for all relevant avg(precision@k for rel) NDCG@k takes into account the order and degree of DCG/IDCG relativity ``` #### When to improve retrieval vs generation - Recall@5 low → improve search - Faithfulness is low → improve prompt - Precision@5 low → add rerank ### 06.26 Chunk Quality Evaluation Assessing the quality of individual chunks in the index: how self-sufficient a chunk is (understandable without context), whether it’s informative enough, or whether it’s too noisy. Identifies bad chunking strategies. #### Automatic chunk evaluation ```python for chunk in index: score = LLM(f""" Rate the chunk according to 3 criteria (0-1): 1. Self-sufficiency (understandable without context?) 2. Information content (are there specific facts?) 3. Clean (no garbage, headers, artifacts?) Chunk: {chunk} """) if score < 0.5: flag_for_rechunking(chunk) ``` ### 06.27 Context Utilization - does LLM use context? Checking: does the final answer rely on the passed chunks or does the model “hallucinate” its weights, ignoring the context. A common problem is when the context conflicts with the prior knowledge of the model. #### Diagnostics ```text test: pass context with intentionally false fact "The Eiffel Tower was built in 1995" if LLM answers "in 1889" → ignores context if LLM answers "in 1995" → follows context ✓ // real-time: citation tracking claims_in_answer → attribution_check → source_chunks ``` ### 06.28 Groundedness Score - according to statements Break down the answer into atomic statements, check each against the sources. Gives the exact percentage of "substantiated" vs "hallucinated" facts. Better than binary hallucination/no-hallucination. #### Pipeline ```python answer → LLM decompose → [claim1, claim2, claim3] for claim in claims: evidence = search(claim, in=source_docs) claim = nli(evidence, claim) // entail=1, neutral=0.5, contradict=0 groundedness = mean(entailment_scores) // 0.9+ excellent, 0.7-0.9 acceptable, <0.7 problem ``` ## 07. Agent-specific Evaluation ### 07.29 Task Completion Rate The main metric for agents: whether the agent solved the problem completely. Evaluated either by a deterministic test of the result (unit test for an artifact) or by an LLM final state estimator. #### Assessment levels ```text Binary: problem solved / not solved Partial: 0.0 – 1.0 (percentage of completed steps) Weighted: critical steps are more important than minor ones Example for code agent: does the code compile? → +0.3 are the tests passing? → +0.4 Does it match the spec? → +0.3 ``` ### 07.30 Trajectory Evaluation - evaluation of the process, not the result Evaluate not only the final answer, but also the path to it: whether the agent used the right tools, in the right order, without unnecessary steps. An agent can give the correct answer in a bad way. #### Trajectory metrics ```text steps_taken: 5 (optimal: 3) → unnecessary steps: 2 tool_accuracy: the correct tool is selected 4/5 times backtrack_rate: agent changed plan 2 times redundant_calls: 1 extra call search efficiency_score: optimal_steps / actual_steps = 0.6 ``` #### Applicable - cost optimization - improvement planning - loop detection ### 07.31 Sandbox Environment Testing For agents with real tools - testing in an isolated sandbox with fake versions of the tools. The agent “thinks” that it is working with real data, but all actions are safe and recordable. #### Scheme ```sql production: agent → real_db, real_api, real_email testing: agent → mock_db, mock_api, mock_email ↓ ↓ ↓ records all calls with arguments assert called_with("SELECT", table="products") assert not called("DELETE") assert email_sent_to == "test@sandbox.com" ``` ### 07.32 Loop & Stuck Detection The agent may get stuck in an endless loop or become fixated on one tool. We need loop detectors and a forced stop with logging of the cause. #### Loop detector ```python recent_actions = deque(maxlen=10) def check_loop(action): if recent_actions.count(action) >= 3: raise AgentLoopError(f"Loop: {action} x3") if len(all_actions) > MAX_STEPS: raise AgentStuckError("Step limit exceeded") recent_actions.append(action) ``` ## 08. Cost Observability ### 08.33 Token Budget Tracking step by step Measuring the consumption of tokens at each step of the pipeline separately. Allows you to find unexpectedly expensive steps and optimize them, rather than guess where the money is leaking. #### breakdown example ```text step input_tok output_tok $cost ────────────────────── ─────────────────────── query_rewrite 120 45 $0.0001 generate_answer 2800 420 $0.0098 ← main validation 650 80 $0.0020 total 3570 545 $0.0119 → 82% of the cost in generate_answer → optimize there ``` ### 08.34 Cost Attribution - by features and users Link the cost to business attributes: user_id, feature, plan, product_line. This allows you to understand unit economics - how much one active user, one feature, one scenario actually costs. #### Unit Economics ```text cost_per_user_per_day = sum(costs, by=user_id) / DAU cost_per_feature = sum(costs, by=feature) insights: "feature X consumes 60% of the budget, but 10% of the users" "The Free plan costs us $0.8/month per user" "enterprise plan pays off x4" ``` ### 08.35 Budget Alerts & Hard Limits Strict limits on token consumption at the user, session and request levels. Without this, one bad prompt or one user can generate an unexpected bill. #### Limit levels ```python per_request: max_tokens_input = 8000 max_tokens_output = 2000 per_session: max_cost = $0.50 per_user_day: max_cost = $2.00 per_hour: max_requests = 100 if exceeded: → truncate context (softly) → rate limit (hard) → oncall alert (critical) ```