{
  "type": "new-runtime-knowledge-guide",
  "version": 1,
  "generated_at": "2026-07-23",
  "volume": 2,
  "slug": "llm-observability-evals",
  "title": "LLM Observability & Evaluations",
  "description": "How to understand what is happening inside the pipeline - and what exactly is broken. Tracing, metrics, automatic evaluations, testing prompts and secure deployment of changes.",
  "track": {
    "slug": "evaluation-and-improvement",
    "title": "Evaluation & Improvement",
    "route": "https://newruntime.com/learn/tracks/evaluation-and-improvement/",
    "position": 1
  },
  "counts": {
    "sections": 8,
    "cards": 35
  },
  "routes": {
    "html": "https://newruntime.com/learn/llm-observability-evals/",
    "json": "https://newruntime.com/learn/llm-observability-evals.json"
  },
  "sections": [
    {
      "number": "01",
      "title": "Distributed Tracing",
      "intro": "",
      "cards": [
        {
          "number": "01",
          "title": "Trace / Span - anatomy of the observed pipeline",
          "tag": "tracing",
          "description": "Trace — the full path of one request through the system. Span is an atomic step inside a trace (one LLM call, one search, one tool). Each span contains: start/end time, tokens, input/output data, status, metadata. Without this, debugging a multi-step pipeline is like looking at a black box.",
          "blocks": [
            {
              "kind": "code",
              "label": "Diagram",
              "language": "text",
              "value": "trace_id: a3f9... total: 4.2s · 3,147 tokens · $0.0089\n\n├── span: query_rewrite 0ms → 380ms · 212 tok\n│ input: “why the pump doesn’t work”\n│ output: \"reasons for centrifugal pump failure\"\n│\n├── span: vector_search 380ms → 510ms · 0 tok\n│ results: 12 chunks, top_score: 0.847\n│\n├── span: rerank 510ms → 890ms · 0 tok\n│ input: 12 → output: 5 chunks\n│\n└── span: generate_answer 890ms → 4200ms · 2935 tok\n      model: cloud-sonnet finish: stop"
            },
            {
              "kind": "list",
              "label": "Tools",
              "items": [
                "Langfuse",
                "LangSmith",
                "Arize Phoenix",
                "Weights & Biases Traces",
                "Helicone"
              ]
            }
          ]
        },
        {
          "number": "02",
          "title": "Span Instrumentation - what exactly to log",
          "tag": "tracing",
          "description": "For each LLM call, you need to log a minimum set of data in order to later reproduce and debug any request. Plus custom fields for business context.",
          "blocks": [
            {
              "kind": "code",
              "label": "Minimum set of span fields",
              "language": "text",
              "value": "{\n  span_id, trace_id, parent_span_id,\n  step_name: \"rerank\",\n  model: \"cloud-sonnet-4\",\n  prompt_tokens: 1240,\n  completion_tokens: 312,\n  latency_ms: 780,\n  status: \"success\" | \"error\",\n  input: {truncated to 10k chars},\n  output: {truncated to 10k chars},\n  // custom:\n  user_id, session_id, product_line,\n  num_retrieved_docs, top_retrieval_score\n}"
            }
          ]
        },
        {
          "number": "03",
          "title": "Session / Thread - tracing of multi-pass dialogues",
          "tag": "tracing",
          "description": "Several traces are combined into a session - the entire user dialogue. This allows you to see: how many moves before the solution, where the user clarified the question, which turn was expensive. Critical for chat products.",
          "blocks": [
            {
              "kind": "code",
              "label": "Hierarchy",
              "language": "text",
              "value": "session (user_id + session_id)\n ├── trace (turn 1)  → spans...\n ├── trace (turn 2)  → spans...\n └── trace (turn 3)  → spans...\n\nsession_metrics:\n  avg_turns_to_resolution: 2.3\n  total_cost_per_session:  $0.042"
            },
            {
              "kind": "list",
              "label": "Applicable",
              "items": [
                "chatbots",
                "support assistants",
                "conversion funnels"
              ]
            }
          ]
        },
        {
          "number": "04",
          "title": "Sampling Strategy - don't log everything",
          "tag": "tracing",
          "description": "When traffic is high, logging 100% trace is expensive. Strategies: head-based sampling (random%), tail-based (log only errors and slow requests), adaptive (more logs for anomalies).",
          "blocks": [
            {
              "kind": "code",
              "label": "Sampling rules",
              "language": "python",
              "value": "if status == \"error\":       sample_rate = 1.0\nelif latency_ms > 5000:    sample_rate = 1.0\nelif score_low_quality:    sample_rate = 1.0\nelif user_flagged:         sample_rate = 1.0\nelse:                      sample_rate = 0.05  // 5%"
            },
            {
              "kind": "list",
              "label": "Applicable",
              "items": [
                "high-load production",
                "cost control"
              ]
            }
          ]
        }
      ]
    },
    {
      "number": "02",
      "title": "Metrics & Monitoring",
      "intro": "",
      "cards": [
        {
          "number": "05",
          "title": "Golden Signals for LLM systems",
          "tag": "metrics",
          "description": "Adaptation of classic SRE golden signals (latency, traffic, errors, saturation) to the specifics of LLM pipelines. Each step of the pipeline should have its own metrics - not just the final answer.",
          "blocks": [
            {
              "kind": "code",
              "label": "Diagram",
              "language": "text",
              "value": "Metric Norm Alert if\n────────────────────────── ───────────────────────────\np50 latency (e2e) < 2s > 5s\np99 latency (e2e) < 8s > 15s\ntoken / request < 3000 > 8000\ncost/request < $0.01 > $0.05\nerror rate < 0.5% > 2%\nretry rate < 2% > 10%\navg quality score > 0.75 < 0.60\nhallucination rate < 3% > 8%\nretrieval recall@5 > 0.80 < 0.65\nrerank MRR > 0.70 < 0.55"
            }
          ]
        },
        {
          "number": "06",
          "title": "Latency Breakdown step by step",
          "tag": "metrics",
          "description": "Measure the latency of each step separately. TTFT (Time To First Token) - separately. This allows you to understand where the pipeline actually slows down: in search, in rerank, in generation or in the network.",
          "blocks": [
            {
              "kind": "code",
              "label": "Latency decomposition",
              "language": "text",
              "value": "TTFT = time_to_first_token // UX metric\nTGT = total_generation_time // model\nLatency breakdown:\n  embed: 45ms █░\n  search: 120ms ████░\n  rerank: 350ms █████████░ ← bottleneck\n  generate: 2100ms ████████████████████"
            }
          ]
        },
        {
          "number": "07",
          "title": "Drift Detection - degradation over time",
          "tag": "metrics",
          "description": "The quality of the LLM pipeline degrades imperceptibly: the data, the behavior of the model after the update, and the distribution of requests change. We need rolling monitoring of metrics with alerts for statistically significant deviations.",
          "blocks": [
            {
              "kind": "code",
              "label": "What's drifting",
              "language": "text",
              "value": "data drift: distribution of incoming requests\nconcept drift: correct answers have changed\nmodel drift: model behavior after update\nprompt drift: prompt performs worse due to context changes\n\nDetect: rolling window 7d vs baseline 30d\n  KL-divergence on query embeddings\n  Mann-Whitney on score metrics"
            },
            {
              "kind": "list",
              "label": "Applicable",
              "items": [
                "long-running products",
                "after API model updates"
              ]
            }
          ]
        },
        {
          "number": "08",
          "title": "Implicit User Signals",
          "tag": "metrics",
          "description": "User behavior as a proxy quality metric. There is no need to explicitly ask for an assessment—whether they copy the answer, ask a clarifying question right away, or return to the session. This is ground truth without markup.",
          "blocks": [
            {
              "kind": "code",
              "label": "Signals and what they mean",
              "language": "text",
              "value": "copy_to_clipboard → answer is helpful\nno_followup_in_30s → question closed\nsession_ended → task completed\nimmediate_rephrase → response not understood\nthumbs_down → obvious signal of poor quality\nrage_click / abandon → frustration"
            },
            {
              "kind": "list",
              "label": "Applicable",
              "items": [
                "B2C products",
                "scalable eval without markers"
              ]
            }
          ]
        },
        {
          "number": "09",
          "title": "Anomaly Detection on pins",
          "tag": "metrics",
          "description": "Automatically flag anomalous responses: too short, too long, with unexpected patterns (refused, repeated, off-topic). Response embedding + OOD detection.",
          "blocks": [
            {
              "kind": "code",
              "label": "Simple heuristics",
              "language": "text",
              "value": "len(response) < 20 → flag \"too_short\"\nlen(response) > 4000 → flag \"too_long\"\n\"I cannot\" in response → \"refusal\" flag\ncosine(response, query) < 0.3 → flag \"off_topic\"\nrepetition_ratio > 0.4 → \"degenerate\" flag"
            }
          ]
        }
      ]
    },
    {
      "number": "03",
      "title": "Evaluation Methods",
      "intro": "",
      "cards": [
        {
          "number": "10",
          "title": "Taxonomy of assessment methods",
          "tag": "evals",
          "description": "Methods for evaluating LLM systems are divided along three axes: who evaluates, when to evaluate, and what exactly is evaluated. Each method has its own trade-offs in terms of cost, speed and reliability.",
          "blocks": [
            {
              "kind": "code",
              "label": "Diagram",
              "language": "bash",
              "value": "Method Who evaluates Speed Cost Reliability\n────────────────────────────────── ───────────────────────────────────\nReference-based metric vs GT ms $0 average\nLLM-as-Judge GPT-4/Claude seconds $$ high\nHuman eval markers days $$$ standard\nModel-based classifier fine-tuned ms $0 high*\nHeuristic / rule code ms $0 low\n\n* after training on human tags"
            }
          ]
        },
        {
          "number": "11",
          "title": "Reference-based Metrics",
          "tag": "evals",
          "description": "Comparing the output with the ground truth answer using deterministic metrics. Fast and cheap, but requires a dataset with correct answers and does not work well for open-ended tasks.",
          "blocks": [
            {
              "kind": "code",
              "label": "Metrics by task",
              "language": "text",
              "value": "Extraction: Exact Match, F1 token overlap\nSummarization: ROUGE-L, BERTScore\nTranslation: BLEU, ChrF\nGeneration: Semantic similarity (cos)\nClassification: Accuracy, F1, MCC\n\n// BERTScore is better than BLEU - takes into account the meaning"
            },
            {
              "kind": "list",
              "label": "Applicable",
              "items": [
                "regression tests",
                "problems with a clear answer"
              ]
            }
          ]
        },
        {
          "number": "12",
          "title": "LLM-as-Judge - correct implementation",
          "tag": "evals",
          "description": "The LLM judge works reliably only with the right configuration: a clear rubric, a scale with examples, CoT before the assessment, neutralization of position bias. Without this, estimates are unstable and biased.",
          "blocks": [
            {
              "kind": "code",
              "label": "Anti-patterns and fixes",
              "language": "sql",
              "value": "✗ “Rate the answer from 1 to 10” → subjective\n✓ Rubrics with specific criteria → objectively\n\n✗ One challenge, one result → unstable\n✓ 3 runs → majority vote → reliable\n\n✗ A is always the first in the pair → position bias\n✓ Swap A↔B, average → neutral\n\n✗ Score without explanation → opaque\n✓ CoT → explanation → score → debuggable"
            }
          ]
        },
        {
          "number": "13",
          "title": "RAGAS - eval framework for RAG",
          "tag": "rag eval",
          "description": "A specialized set of metrics for RAG systems: evaluates not only the final response, but also the quality of the retrieval separately. Automatically through LLM - does not require GT responses for some metrics.",
          "blocks": [
            {
              "kind": "code",
              "label": "4 RAGAS metrics",
              "language": "text",
              "value": "Faithfulness response supported by context?\n                       // detect hallucinations\nAnswer Relevance is the answer relevant to the question?\n                       // detect off-topic\nContext Precision found chunks useful?\n                       // retrieval quality\nContext Recall have all the required chunks been found?\n                       // requires GT"
            },
            {
              "kind": "list",
              "label": "Applicable",
              "items": [
                "RAG QA",
                "knowledge bases",
                "document assistants"
              ]
            }
          ]
        },
        {
          "number": "14",
          "title": "Pairwise / Comparative Eval",
          "tag": "evals",
          "description": "Instead of an absolute assessment, a comparison of two options: “which answer is better, A or B?” It is much easier for people and LLM judges to make comparisons than to give absolute marks. Used to compare prompts/models.",
          "blocks": [
            {
              "kind": "code",
              "label": "ELO rating of prompts",
              "language": "sql",
              "value": "prompt_v1 vs prompt_v2 → judge: \"v2 is better\"\nprompt_v2 vs prompt_v3 → judge: \"v2 is better\"\nprompt_v3 vs prompt_v1 → judge: \"v3 is better\"\n\nELO: v2=1250, v3=1180, v1=1070\n// → select v2 for deployment"
            },
            {
              "kind": "list",
              "label": "Applicable",
              "items": [
                "prompt selection",
                "model selection",
                "A/B test of configs"
              ]
            }
          ]
        },
        {
          "number": "15",
          "title": "Behavioral Testing - CheckList approach",
          "tag": "testing",
          "description": "A set of specific behavioral tests: “the model must X given Y.” An analogue of unit tests for NLP (CheckList, Microsoft). Tests specific abilities rather than general quality.",
          "blocks": [
            {
              "kind": "code",
              "label": "Types of tests",
              "language": "text",
              "value": "MFT (Minimum Functionality):\n  \"answer a simple question\" → pass/fail\n\nINV (Invariance):\n  \"the result does not change when changing names\"\n  Ivan bought → Anna bought → same intent\n\nDIR (Directional):\n  \"adding negativity reduces sentiment\"\n  \"good\" → score 0.9\n  \"not very good\" → score < 0.9 ✓"
            },
            {
              "kind": "list",
              "label": "Applicable",
              "items": [
                "regression suite",
                "edge case coverage"
              ]
            }
          ]
        }
      ]
    },
    {
      "number": "04",
      "title": "Prompt Testing & CI/CD",
      "intro": "",
      "cards": [
        {
          "number": "16",
          "title": "Golden Dataset is the foundation of everything",
          "tag": "testing",
          "description": "A labeled dataset of inputs with expected outputs or evaluation criteria. It is built iteratively: start with 20-50 cases covering the main scenarios, add new edge cases when found.",
          "blocks": [
            {
              "kind": "code",
              "label": "Dataset structure",
              "language": "sql",
              "value": "[\n  {\n    id: \"tc_001\",\n    input: {query: \"...\", context: \"...\"},\n    expected_output: \"...\", // or null\n    eval_criteria: [\n      \"contains a specific number\"\n      \"does not mention competitors\"\n      \"answers in Russian\"\n    ],\n    tags: [\"edge_case\", \"pricing\"],\n    added_because: \"bug from 2024-11-03\"\n  }\n]"
            }
          ]
        },
        {
          "number": "17",
          "title": "Prompt CI - tests for every change",
          "tag": "CI/CD",
          "description": "Changing a prompt is a change in code. Before merging into production: run the golden dataset, compare metrics with baseline, block deployment during regression. Prompts are stored in git with versioning.",
          "blocks": [
            {
              "kind": "code",
              "label": "Pipeline in CI",
              "language": "bash",
              "value": "git push prompt_v2\n  → CI trigger\n  → run_eval(dataset, prompt_v2)\n  → compare(prompt_v2.scores, baseline.scores)\n  → if regression > 5%: BLOCK merge\n  → if improvement > 3%: AUTO APPROVE\n  → else: REQUEST human review"
            },
            {
              "kind": "list",
              "label": "Tools",
              "items": [
                "Langfuse Experiments",
                "PromptFlow",
                "Braintrust",
                "GitHub Actions"
              ]
            }
          ]
        },
        {
          "number": "18",
          "title": "Regression Suite - protection against degradation",
          "tag": "testing",
          "description": "A set of tests that capture current \"good\" behavior and automatically check that it hasn't broken. It is updated every time a bug is found in the product.",
          "blocks": [
            {
              "kind": "code",
              "label": "Regression layers",
              "language": "text",
              "value": "smoke tests: 5-10 critical cases, <30 sec\nregression suite: 50-200 cases, run on PR\nfull eval: 500+ cases, night run\nadversarial: attempts to break, jailbreak"
            }
          ]
        },
        {
          "number": "19",
          "title": "Synthetic Eval Data Generation",
          "tag": "testing",
          "description": "The LLM generates test cases to evaluate another LLM. From the documents - questions + answers. Allows you to quickly create a dataset without manual marking. Quality must be verified selectively manually.",
          "blocks": [
            {
              "kind": "code",
              "label": "Generation scheme",
              "language": "sql",
              "value": "for doc in knowledge_base:\n    qa_pairs = LLM(f\"\"\"\n        From the document below, generate 3 question-answer pairs.\n        Include: simple, complex, edge case.\n        Document: {doc}\n        Format: JSON [{{\"q\":..., \"a\":...}}]\n    \"\"\")\n    dataset.extend(qa_pairs)"
            },
            {
              "kind": "list",
              "label": "Applicable",
              "items": [
                "start without markers",
                "RAG testing",
                "RAGAS"
              ]
            }
          ]
        },
        {
          "number": "20",
          "title": "Shadow Mode - testing on real traffic",
          "tag": "deployment",
          "description": "The new version of the pipeline receives the same requests as the production version, generates responses, but does not give them to the user. Metrics of two versions are compared on the same traffic without risk to users.",
          "blocks": [
            {
              "kind": "code",
              "label": "Scheme",
              "language": "text",
              "value": "request\n  ├── prod_pipeline(request) → user ✓\n  └── shadow_pipeline(request) → /dev/null\n                                → logs + metrics\n\nafter N requests:\n  compare(prod.metrics, shadow.metrics)\n  → decision to deploy"
            },
            {
              "kind": "list",
              "label": "Applicable",
              "items": [
                "model change",
                "major pipeline refactor"
              ]
            }
          ]
        }
      ]
    },
    {
      "number": "05",
      "title": "Safe Deployment",
      "intro": "",
      "cards": [
        {
          "number": "21",
          "title": "Canary Deployment prompts",
          "tag": "deployment",
          "description": "The new prompt is rolled out to a small percentage of traffic. With normal metrics, it gradually expands. In case of an anomaly - instant rollback. This is the safest way to deploy changes to LLM systems.",
          "blocks": [
            {
              "kind": "code",
              "label": "Rollout stages",
              "language": "text",
              "value": "Day 1: 1% traffic → 24h monitoring\nDay 2: 5% traffic → 24h monitoring\nDay 3: 20% traffic → 24h monitoring\nDay 5: 50% traffic → monitoring 48h\nDay 7: 100% → prompt_v2 = new baseline\n\nauto-rollback if:\n  error_rate > 3% OR quality_score drop > 10%"
            }
          ]
        },
        {
          "number": "22",
          "title": "A/B Test - statistically correct",
          "tag": "deployment",
          "description": "Random division of traffic between versions, measurement of business metrics, statistical testing of significance before making a decision. You can’t stop an A/B test prematurely - p-hacking.",
          "blocks": [
            {
              "kind": "code",
              "label": "The right process",
              "language": "text",
              "value": "1. Determine the victory metric BEFORE the test\n2. Calculate the required sample size (power=0.8)\n3. Run → wait for N=calculated requests\n4. Conduct t-test / chi-square\n5. p < 0.05 → deploy winner\n   p > 0.05 → no difference, keep the old one"
            }
          ]
        },
        {
          "number": "23",
          "title": "Feature Flags for prompts/models",
          "tag": "deployment",
          "description": "Prompts and model configurations are stored in the feature flag system, not in the code. Switching occurs without deployment - runtime. You can target by user_id, region, plan, experiment_group.",
          "blocks": [
            {
              "kind": "code",
              "label": "Config at runtime",
              "language": "text",
              "value": "config = feature_flags.get(user_id, {\n  \"model\": \"cloud-sonnet-4\",\n  \"prompt_ver\": \"v3.2\",\n  \"temperature\": 0.3,\n  \"rerank\": True,\n  \"top_k\": 5\n})\n// change without re-deploying the service"
            },
            {
              "kind": "list",
              "label": "Tools",
              "items": [
                "LaunchDarkly",
                "Unleash",
                "Langfuse Prompt Management"
              ]
            }
          ]
        },
        {
          "number": "24",
          "title": "Automated Rollback",
          "tag": "deployment",
          "description": "The system automatically rolls back to the previous version of the prompt/config when alerts are triggered. No manual intervention. Critical for night deployments and high-load systems.",
          "blocks": [
            {
              "kind": "code",
              "label": "Automatic rollback triggers",
              "language": "python",
              "value": "watch(metrics, window=5min):\n  if error_rate > threshold:      rollback()\n  if p99_latency > threshold:     rollback()\n  if avg_quality_score < floor:  rollback()\n  if refusal_rate spike:          rollback()\n\nrollback() → feature_flag.set(prev_version)\n           → alert(oncall)"
            }
          ]
        }
      ]
    },
    {
      "number": "06",
      "title": "RAG-specific Evaluation",
      "intro": "",
      "cards": [
        {
          "number": "25",
          "title": "Retrieval Metrics - evaluation of the search part separately",
          "tag": "rag eval",
          "description": "It is critically important to evaluate retrieval regardless of generation - otherwise you will not understand where the problem is: in search or in LLM. To do this, you need a dataset with known relevant documents for each question.",
          "blocks": [
            {
              "kind": "code",
              "label": "Diagram",
              "language": "text",
              "value": "Metric What it does the Formula count?\n───────────────────────────────── ─────────────────────────────────\nPrecision@k share of relevant ones among top-k |rel ∩ top-k| /k\nRecall@k share of those found among all relevant ones |rel ∩ top-k| / |rel|\nMRR how high is the first relevant 1/rank_first_rel\nMAP average precision for all relevant avg(precision@k for rel)\nNDCG@k takes into account the order and degree of DCG/IDCG relativity"
            },
            {
              "kind": "list",
              "label": "When to improve retrieval vs generation",
              "items": [
                "Recall@5 low → improve search",
                "Faithfulness is low → improve prompt",
                "Precision@5 low → add rerank"
              ]
            }
          ]
        },
        {
          "number": "26",
          "title": "Chunk Quality Evaluation",
          "tag": "rag eval",
          "description": "Assessing the quality of individual chunks in the index: how self-sufficient a chunk is (understandable without context), whether it’s informative enough, or whether it’s too noisy. Identifies bad chunking strategies.",
          "blocks": [
            {
              "kind": "code",
              "label": "Automatic chunk evaluation",
              "language": "python",
              "value": "for chunk in index:\n    score = LLM(f\"\"\"\n        Rate the chunk according to 3 criteria (0-1):\n        1. Self-sufficiency (understandable without context?)\n        2. Information content (are there specific facts?)\n        3. Clean (no garbage, headers, artifacts?)\n        Chunk: {chunk}\n    \"\"\")\n    if score < 0.5: flag_for_rechunking(chunk)"
            }
          ]
        },
        {
          "number": "27",
          "title": "Context Utilization - does LLM use context?",
          "tag": "rag eval",
          "description": "Checking: does the final answer rely on the passed chunks or does the model “hallucinate” its weights, ignoring the context. A common problem is when the context conflicts with the prior knowledge of the model.",
          "blocks": [
            {
              "kind": "code",
              "label": "Diagnostics",
              "language": "text",
              "value": "test: pass context with intentionally false fact\n  \"The Eiffel Tower was built in 1995\"\n\nif LLM answers \"in 1889\" → ignores context\nif LLM answers \"in 1995\" → follows context ✓\n\n// real-time: citation tracking\nclaims_in_answer → attribution_check → source_chunks"
            }
          ]
        },
        {
          "number": "28",
          "title": "Groundedness Score - according to statements",
          "tag": "rag eval",
          "description": "Break down the answer into atomic statements, check each against the sources. Gives the exact percentage of \"substantiated\" vs \"hallucinated\" facts. Better than binary hallucination/no-hallucination.",
          "blocks": [
            {
              "kind": "code",
              "label": "Pipeline",
              "language": "python",
              "value": "answer → LLM decompose → [claim1, claim2, claim3]\n\nfor claim in claims:\n    evidence = search(claim, in=source_docs)\n    claim = nli(evidence, claim)\n    // entail=1, neutral=0.5, contradict=0\n\ngroundedness = mean(entailment_scores)\n// 0.9+ excellent, 0.7-0.9 acceptable, <0.7 problem"
            }
          ]
        }
      ]
    },
    {
      "number": "07",
      "title": "Agent-specific Evaluation",
      "intro": "",
      "cards": [
        {
          "number": "29",
          "title": "Task Completion Rate",
          "tag": "agent eval",
          "description": "The main metric for agents: whether the agent solved the problem completely. Evaluated either by a deterministic test of the result (unit test for an artifact) or by an LLM final state estimator.",
          "blocks": [
            {
              "kind": "code",
              "label": "Assessment levels",
              "language": "text",
              "value": "Binary: problem solved / not solved\nPartial: 0.0 – 1.0 (percentage of completed steps)\nWeighted: critical steps are more important than minor ones\n\nExample for code agent:\n  does the code compile?     → +0.3\n  are the tests passing?        → +0.4\n  Does it match the spec?   → +0.3"
            }
          ]
        },
        {
          "number": "30",
          "title": "Trajectory Evaluation - evaluation of the process, not the result",
          "tag": "agent eval",
          "description": "Evaluate not only the final answer, but also the path to it: whether the agent used the right tools, in the right order, without unnecessary steps. An agent can give the correct answer in a bad way.",
          "blocks": [
            {
              "kind": "code",
              "label": "Trajectory metrics",
              "language": "text",
              "value": "steps_taken: 5 (optimal: 3) → unnecessary steps: 2\ntool_accuracy: the correct tool is selected 4/5 times\nbacktrack_rate: agent changed plan 2 times\nredundant_calls: 1 extra call search\nefficiency_score: optimal_steps / actual_steps = 0.6"
            },
            {
              "kind": "list",
              "label": "Applicable",
              "items": [
                "cost optimization",
                "improvement planning",
                "loop detection"
              ]
            }
          ]
        },
        {
          "number": "31",
          "title": "Sandbox Environment Testing",
          "tag": "agent eval",
          "description": "For agents with real tools - testing in an isolated sandbox with fake versions of the tools. The agent “thinks” that it is working with real data, but all actions are safe and recordable.",
          "blocks": [
            {
              "kind": "code",
              "label": "Scheme",
              "language": "sql",
              "value": "production: agent → real_db, real_api, real_email\ntesting: agent → mock_db, mock_api, mock_email\n                       ↓ ↓ ↓\n                    records all calls with arguments\n\nassert called_with(\"SELECT\", table=\"products\")\nassert not called(\"DELETE\")\nassert email_sent_to == \"test@sandbox.com\""
            }
          ]
        },
        {
          "number": "32",
          "title": "Loop & Stuck Detection",
          "tag": "agent eval",
          "description": "The agent may get stuck in an endless loop or become fixated on one tool. We need loop detectors and a forced stop with logging of the cause.",
          "blocks": [
            {
              "kind": "code",
              "label": "Loop detector",
              "language": "python",
              "value": "recent_actions = deque(maxlen=10)\n\ndef check_loop(action):\n    if recent_actions.count(action) >= 3:\n        raise AgentLoopError(f\"Loop: {action} x3\")\n    if len(all_actions) > MAX_STEPS:\n        raise AgentStuckError(\"Step limit exceeded\")\n    recent_actions.append(action)"
            }
          ]
        }
      ]
    },
    {
      "number": "08",
      "title": "Cost Observability",
      "intro": "",
      "cards": [
        {
          "number": "33",
          "title": "Token Budget Tracking step by step",
          "tag": "cost",
          "description": "Measuring the consumption of tokens at each step of the pipeline separately. Allows you to find unexpectedly expensive steps and optimize them, rather than guess where the money is leaking.",
          "blocks": [
            {
              "kind": "code",
              "label": "breakdown example",
              "language": "text",
              "value": "step input_tok output_tok $cost\n────────────────────── ───────────────────────\nquery_rewrite 120 45 $0.0001\ngenerate_answer 2800 420 $0.0098 ← main\nvalidation 650 80 $0.0020\ntotal 3570 545 $0.0119\n\n→ 82% of the cost in generate_answer → optimize there"
            }
          ]
        },
        {
          "number": "34",
          "title": "Cost Attribution - by features and users",
          "tag": "cost",
          "description": "Link the cost to business attributes: user_id, feature, plan, product_line. This allows you to understand unit economics - how much one active user, one feature, one scenario actually costs.",
          "blocks": [
            {
              "kind": "code",
              "label": "Unit Economics",
              "language": "text",
              "value": "cost_per_user_per_day = sum(costs, by=user_id) / DAU\ncost_per_feature = sum(costs, by=feature)\n\ninsights:\n  \"feature X consumes 60% of the budget, but 10% of the users\"\n  \"The Free plan costs us $0.8/month per user\"\n  \"enterprise plan pays off x4\""
            }
          ]
        },
        {
          "number": "35",
          "title": "Budget Alerts & Hard Limits",
          "tag": "cost",
          "description": "Strict limits on token consumption at the user, session and request levels. Without this, one bad prompt or one user can generate an unexpected bill.",
          "blocks": [
            {
              "kind": "code",
              "label": "Limit levels",
              "language": "python",
              "value": "per_request: max_tokens_input = 8000\n              max_tokens_output = 2000\nper_session: max_cost = $0.50\nper_user_day: max_cost = $2.00\nper_hour: max_requests = 100\n\nif exceeded:\n  → truncate context (softly)\n  → rate limit (hard)\n  → oncall alert (critical)"
            }
          ]
        }
      ]
    }
  ]
}