{ "type": "new-runtime-knowledge-guide", "version": 1, "generated_at": "2026-07-23", "volume": 33, "slug": "base-metrics", "title": "Core Metrics for LLM Pipelines", "description": "One volume about the most useful metrics for LLM systems: from average, variance and standard deviation to accuracy, precision, recall, F1, MRR, MAP, nDCG@k, calibration, pass@k, latency and cost per success. Each card answers four questions: what it measures, how to calculate it, how to read, and where to apply it in the pipeline.", "track": { "slug": "evaluation-and-improvement", "title": "Evaluation & Improvement", "route": "https://newruntime.com/learn/tracks/evaluation-and-improvement/", "position": 6 }, "counts": { "sections": 5, "cards": 26 }, "routes": { "html": "https://newruntime.com/learn/base-metrics/", "json": "https://newruntime.com/learn/base-metrics.json" }, "sections": [ { "number": "01", "title": "Basic statistics", "intro": "", "cards": [ { "number": "01", "title": "Mean and weighted average", "tag": "Base", "description": "The average answers the question “how much on average.” This is the most common aggregating metric for judge score, latency, cost, reward and batch quality. A weighted average is needed when observations are unequal: for example, requests differ in the number of tokens or importance.", "blocks": [ { "kind": "code", "label": "formula", "language": "tex", "value": "mean = Σx_i / n\nweighted mean = Σ(w_i * x_i) / Σw_i\n\nIf all observations are equally important -> the usual average.\nIf the weight is different, use weights." }, { "kind": "code", "label": "Diagram", "language": "tex", "value": "latency = [1.2, 1.6, 3.2] sec\nmean = (1.2 + 1.6 + 3.2) / 3 = 2.0 sec\n\ntoken-weighted loss:\n100 tokens with loss 0. 2.\n900 tokens with loss 0.5\n\nweighted mean = (100*0.2 + 900*0.5) / 1000 = 0.47\n\nIf you just average 0.2 and 0.5, you get 0.35 -> that's wrong." }, { "kind": "list", "label": "Where it helps", "items": [ "average judge score", "cost / request", "middle-tail", "Don't just look at the mean latency." ] } ] }, { "number": "02", "title": "Median and percentiles: p50, p95, p99", "tag": "Base", "description": "The median shows a \"typical\" value that doesn't break down from rare emissions. Percentile p95 means 95% of observations are as good as this. For latency and token count, they are usually more important than average.", "blocks": [ { "kind": "code", "label": "formula", "language": "tex", "value": "p50 = median\np95 = value below which lies 95% of observations\np99 = value below which lies 99% of observations\n\nQuestion to the metric:\n\"how bad is the system's tail?\"" }, { "kind": "code", "label": "Diagram", "language": "tex", "value": "sorted latency = [100, 120, 140, 180, 900] ms\n\nmean = 288 ms\np50 = 140 ms\np80 = 180 ms\np95 ~= 900 ms\n\nThe average says \"like 288 ms.\"\np95 shows real tail pain" }, { "kind": "list", "label": "Where it helps", "items": [ "p95 latency", "tail token count", "median does not show worst-case", "p99 for SLO" ] } ] }, { "number": "03", "title": "Dispersion and standard deviation", "tag": "Base", "description": "The variance and standard deviation show a spread around the mean. They are needed when it is important not only what the average score is, but also how stable it is from example, for example, from launch to launch and from annotator to annotator.", "blocks": [ { "kind": "code", "label": "formula", "language": "tex", "value": "variance = Σ(x_i - mean)^2 / n\nstd = sqrt(variance)\n\nThe more std, the less predictable the outcome.\nThe std is measured in the same units as the reference metric." }, { "kind": "code", "label": "Diagram", "language": "tex", "value": "System A: [0.7, 0.8, 0.9]\nmean = 0.8, std ~= 0.08\n\nSystem B: [0.2, 0.8, 1.4]\nmean = 0.8, std ~= 0.49\n\nThe average is the same, the stability is different." }, { "kind": "list", "label": "Where it helps", "items": [ "run-to-run variance", "judge consistency", "std without a mean of little information", "compare" ] } ] }, { "number": "04", "title": "Share, success rate, error rate", "tag": "Base", "description": "A lot of product and eval metrics are actually just fractions: how many queries worked, how many JSONs were valid, how many answers were grounded, how many times the system failed. This is the most universal type of metric.", "blocks": [ { "kind": "code", "label": "formula", "language": "tex", "value": "rate = successes / total\nerror rate = errors / total = 1 - success rate\n\nThe main rule:\nAlways indicate the denominator.\n\"87 errors\" without \"how many\" is almost useless." }, { "kind": "code", "label": "Diagram", "language": "tex", "value": "100 requests\n87 correct answers\n13 incorrect\n\nsuccess rate = 87 / 100 = 0.87\nerror rate = 13 / 100 = 0.13" }, { "kind": "list", "label": "Where it helps", "items": [ "task success rate", "schema-valid rate", "hallucination rate", "You need a size sample." ] } ] }, { "number": "05", "title": "Confidence interval, delta and lift", "tag": "Base", "description": "One number without uncertainty often overestimates confidence. The confidence interval shows the range of plausible values. Delta shows an absolute difference, lift shows relative growth relative to baseline.", "blocks": [ { "kind": "code", "label": "formula", "language": "tex", "value": "For fraction p:\n95% CI ~= p +/- 1.96 * sqrt(p(1-p)/n)\n\ndelta = metric_B - metric_A\nlift = (metric_B - metric_A) / metric_A" }, { "kind": "code", "label": "Diagram", "language": "tex", "value": "A: success = 78%\nB: success = 81%\n\ndelta = 81% - 78% = +3 pp\nlift = 3 / 78 = +3.8%\n\nIf p = 0.80 and n = 1000:\n95% CI ~= 0.80 +/- 1.96 * sqrt(0.8*0.2/1000)\n ~= 0.80 +/- 0.025\n ~= [0.775, 0.825]" }, { "kind": "list", "label": "Where it helps", "items": [ "A/B comparison", "report uncertainty", "delta in pp!= lift in %", "Do not publish a point estimate" ] } ] } ] }, { "number": "02", "title": "Classification and filters", "intro": "", "cards": [ { "number": "06", "title": "Confusion Matrix: TP, FP, TN, FN", "tag": "Classification", "description": "Almost all binary metrics are derived from four numbers. If you do not fix what is a positive class, then it is easy to confuse the meaning of precision, recall and false positive rate. For safety and routing, this is the basic table from which everything starts.", "blocks": [ { "kind": "code", "label": "read", "language": "text", "value": "TP: Model says 'positive' and it's true\nFP: Model says 'positive' but it's false alarm\nTN: Model says \"negative\" and it's true\nFN: Model said \"negative\" but missed positive\n\nAccuracy/precision/remember/F1\nIt's just a different relationship between these four numbers." }, { "kind": "code", "label": "Diagram", "language": "text", "value": " Truth\n Positive Negative\nPred Positive TP FP\nPred Negative FN TN\n\nExample of safety-filter:\nTP = catching a harmful request\nFP = block the normal request\nFN = missed harmful request\nTN = missed a normal request" }, { "kind": "list", "label": "Where it helps", "items": [ "toxicity filter", "intent classifier", "spam / abuse detector", "You have to ask positive class." ] } ] }, { "number": "07", "title": "Accuracy and error rate", "tag": "Classification", "description": "Accuracy shows a proportion of correct predictions. It is convenient when the classes are more or less balanced. With a strong imbalance, accuracy often looks more beautiful than the model is actually useful.", "blocks": [ { "kind": "code", "label": "formula", "language": "tex", "value": "accuracy = (TP + TN) / (TP + FP + TN + FN)\nerror rate = 1 - accuracy" }, { "kind": "code", "label": "Diagram", "language": "text", "value": "990 safe, 10 unsafe\nModel says \"safe\" to everyone\n\naccuracy = 990 / 1000 = 99%\nrecall unsafe = 0 / 10 = 0%\n\nFor safety, such accuracy is almost useless." }, { "kind": "list", "label": "when to use", "items": [ "balanced-class", "class imbalance makes sense", "Do not use it alone for safety" ] } ] }, { "number": "08", "title": "Precision and Recall", "tag": "Classification", "description": "Precision answers the question, “If a model flagged something, how often is she right?” Recall answers the question, “What proportion of really important cases did the model find?” This is almost always the main pair of metrics for filters, detectors, and routers.", "blocks": [ { "kind": "code", "label": "formula", "language": "tex", "value": "precision = TP / (TP + FP)\nrecall = TP / (TP + FN)\n\nPrecision -> Price of false alarm\nrecall -> skip price" }, { "kind": "code", "label": "Diagram", "language": "tex", "value": "There are 10 harmful queries and 90 normal ones.\nThe filter caught 9 harmful and 3 normal hit in vain\n\nTP = 9, FP = 3, FN = 1\n\nprecision = 9 / (9 + 3) = 0.75\nrecall = 9 / (9 + 1) = 0.90" }, { "kind": "list", "label": "choose", "items": [ "High recall for safety", "High precision for routing", "You can't improve one for free.", "threshold" ] } ] }, { "number": "09", "title": "Specificity, FPR and FNR", "tag": "Classification", "description": "These metrics are useful when it is separately important to control false locks and dangerous omissions. For safety-review, it is often more convenient to talk about the false negative rate, and for antifraud and moderation, false positive rate is sometimes more important.", "blocks": [ { "kind": "code", "label": "formula", "language": "tex", "value": "specificity = TN / (TN + FP)\nFPR = FP / (FP + TN) = 1 - specificity\nFNR = FN / (FN + TP) = 1 - recall" }, { "kind": "code", "label": "Diagram", "language": "sql", "value": "From a previous example:\nTN = 87, FP = 3, FN = 1, TP = 9\n\nspecificity = 87 / 90 = 0.967\nFPR = 3 / 90 = 0.033\nFNR = 1 / 10 = 0.10\n\nFNR is especially important if the pass is expensive." }, { "kind": "list", "label": "Where it helps", "items": [ "guardrails", "moderation", "abuse detection", "It is important to fix the threshold" ] } ] }, { "number": "10", "title": "F1 and F-beta", "tag": "Classification", "description": "F1 is needed when both precision and recall are important at the same time, and you want one final digit. This is a harmonic average: a metric will not allow one side to mask the failure of the other. F-beta allows you to appreciate recall or precision.", "blocks": [ { "kind": "code", "label": "formula", "language": "tex", "value": "F1 = 2PR / (P + R)\nF_beta = (1 + beta^2)PR / (beta^2 P + R)\n\nBeta > 1 - More weight recall\nBeta < 1 - More precision weight" }, { "kind": "code", "label": "Diagram", "language": "text", "value": "If P = 0.75, R = 0.90:\n\nF1 = 2 * 0.75 * 0.90 / (0.75 + 0.90)\n = 0.818\n\nF2 = 5 * 0.75 * 0.90 / (4*0.75 + 0.90)\n = 0.865\n\nF2 is higher because it rewards recall more" }, { "kind": "list", "label": "when to use", "items": [ "single headline metric", "model selection", "F1 hides which compromise is chosen", "F2 for safety/triage" ] } ] }, { "number": "11", "title": "Macro, Micro, Weighted, Balanced Accuracy and MCC", "tag": "Classification", "description": "Once there are more than two classes or the data is unbalanced, the method of averaging is important. Macro gives all classes the same voice. Weighted gives more weight to frequent classes. Micro counts errors globally. Balanced accuracy and MCC are useful when looking more honestly at imbalanced data.", "blocks": [ { "kind": "code", "label": "formula", "language": "tex", "value": "macro = mean(metric c by class)\nweighted = Σ(support_c * metric_c) / Σsupport_c\nmicro = count TP/FP/FN globally, then metric\n\nbalanced accuracy = (recall + specificity) / 2\nMCC = (TP*TN - FP*FN) / sqrt((TP+FP)(TP+FN)(TN+FP)(TN+FN))" }, { "kind": "code", "label": "Diagram", "language": "text", "value": "Class A: 990 examples, F1 = 0.99\nClass B: 10 examples, F1 = 0.20\n\nmacro F1 = (0.99 + 0.20) / 2 = 0.595\nweighted F1 = (990*0.99 + 10*0.20) / 1000 = 0.982\n\nWeighted almost does not notice a failure in a rare class\n\nIf the model is always predicting the majority class:\nbalanced accuracy ~= 0.50\nMCC = 0 is more accurate than 99% accuracy" }, { "kind": "list", "label": "selection rule", "items": [ "macro for fairness by class", "micro", "MCC for imbalance", "Weighted hides minority fail" ] } ] } ] }, { "number": "03", "title": "Retrieval and Ranking", "intro": "", "cards": [ { "number": "12", "title": "Hit@k / Success@k", "tag": "Ranking", "description": "Hit@k answers a simple question: is there at least one useful document in the top-k? For the first retrieval stage, this metric is especially convenient when the generator needs one good support, rather than a complete set of all relevant documents.", "blocks": [ { "kind": "code", "label": "formula", "language": "tex", "value": "Hit@k for one request:\n1 if there is at least one relevant item in the top-k\n0 if not\n\nHit@k = Average of Requests" }, { "kind": "code", "label": "Diagram", "language": "text", "value": "10 requests for eval set\n8 of them in the top 5 found at least 1 useful passage\n\nHit@5 = 8 / 10 = 0.80\n\nIt doesn't matter if it's 1st or 5th. n" }, { "kind": "list", "label": "when to use", "items": [ "first-stage retrieval", "RAG candidate search", "Do not see the quality of order inside the top k", "handy as a ceiling for a generator" ] } ] }, { "number": "13", "title": "Precision@k and Recall@k", "tag": "Ranking", "description": "Precision@k shows how clean top-k is. Recall@k shows how much of all relevant documents retriever managed to bring. This is a basic pair of metrics for search and candidate generation.", "blocks": [ { "kind": "code", "label": "formula", "language": "tex", "value": "P@k = relevant_in_top_k / k\nR@k = relevant_in_top_k / total_relevant\n\nP@k -> issue noise\nR@k -> Relevance coverage" }, { "kind": "code", "label": "Diagram", "language": "text", "value": "Total relevant documents = 4\nIn the top 5 found 2 relevant\n\nP@5 = 2 / 5 = 0.40\nR@5 = 2 / 4 = 0.50\n\nYou can have good recall and bad precision and vice versa." }, { "kind": "list", "label": "Where it helps", "items": [ "search quality", "RAG chunk retrieval", "We need relevance markups.", "look up" ] } ] }, { "number": "14", "title": "MRR: Mean Reciprocal Rank", "tag": "Ranking", "description": "The MRR measures how early the first benefit appears. If a user or generator needs one good document, MRR is usually more informative than recalling all relevant items.", "blocks": [ { "kind": "code", "label": "formula", "language": "tex", "value": "RR = 1 / rank_first_relevant\nif the relevant item is not found -> RR = 0\n\nMRR = mean(RR for all requests)" }, { "kind": "code", "label": "Diagram", "language": "text", "value": "3 requests\nFirst relevant in positions: [1, 2, 5]\n\nRR = [1, 1/2, 1/5] = [1.0, 0.5, 0.2]\nMRR = (1.0 + 0.5 + 0.2) / 3 = 0.567\n\nRank 1 is very much appreciated, rank 5 is already noticeably worse." }, { "kind": "list", "label": "when to use", "items": [ "FAQ search", "single-best-doc retrieval", "Ignore second and third useful documents", "Good for QA and support search" ] } ] }, { "number": "15", "title": "AP and MAP", "tag": "Ranking", "description": "Average Precision evaluates the entire ranking at once: the metric increases when relevant documents are higher and appear earlier. MAP averages AP for all queries. This is a convenient metric when not one document is important, but the whole set of relevant elements.", "blocks": [ { "kind": "code", "label": "formula", "language": "sql", "value": "AP = average Precision@r\nat all levels of r, where the relevant document\n\nMAP = mean(AP on request)" }, { "kind": "code", "label": "Diagram", "language": "text", "value": "Relevant documents are ranked 1, 3, 4\n\nP@1 = 1/1 = 1.00\nP@3 = 2/3 = 0.67\nP@4 = 3/4 = 0.75\n\nAP = (1.00 + 0.67 + 0.75) / 3 = 0.806\n\nMAP = average of these APs for all requests" }, { "kind": "list", "label": "Where it helps", "items": [ "document retrieval", "reranker evaluation", "More difficult to explain to a business than Hit@k", "MRR is better when all relevant docs are important" ] } ] }, { "number": "16", "title": "DCG and nDCG@k: when relevance is graded", "tag": "Ranking", "description": "nDCG@k is needed when documents are not just relevant / not relevant, but have levels of utility: for example, 3 = perfect proof, 2 = useful, 1 = partially. The metric rewards important documents at the top and normalizes the result of a relatively perfect sort.", "blocks": [ { "kind": "code", "label": "formula", "language": "tex", "value": "DCG@k = Σ(rel_i / log2(i + 1))\nnDCG@k = DCG@k / IDCG@k\n\nIDCG = best possible DCG for this query\nnDCG is always in the range [0, 1]" }, { "kind": "code", "label": "Diagram", "language": "text", "value": "Graduated relevance:\nideal order = [3, 2, 0]\n\nIDCG = 3/log2(2) + 2/log2(3)\n = 3 + 1.262 = 4.262\n\nActual order = [2, 3, 0]\n\nDCG = 2/log2(2) + 3/log2(3)\n = 2 + 1.893 = 3.893\n\nnDCG = 3.893 / 4.262 = 0.913\n\nAlmost good, but the most valuable document is not the first." }, { "kind": "list", "label": "Where it helps", "items": [ "reranking", "graded search relevance", "human-labeled passage quality", "It is not binary but graded markup." ] } ] } ] }, { "number": "04", "title": "Generation, Judge and Reliability", "intro": "", "cards": [ { "number": "17", "title": "Exact Match and schema accuracy", "tag": "Generation", "description": "If the problem has an exact correct answer or a strict format, the exact metrics work best. For extraction, tool calling, JSON output and structured QA exact match is often more useful than any soft judge score.", "blocks": [ { "kind": "code", "label": "formula", "language": "tex", "value": "Exact Match = exact_matches / total\nSchema Accuracy = valid_schema_outputs / total\n\nEither it matched or it didn't." }, { "kind": "code", "label": "Diagram", "language": "text", "value": "100 answers\n92 valid JSONs\n68 of them completely matched the fields.\n\nschema accuracy = 92 / 100 = 0.92\nexact match = 68 / 100 = 0.68" }, { "kind": "list", "label": "Where it helps", "items": [ "tool calling", "information extraction", "structured QA", "A very strict metric for open-ended chat" ] } ] }, { "number": "18", "title": "ROUGE, BLEU and overlap metrics", "tag": "Generation", "description": "Overlap metrics compare model output to reference text for shared tokens, n-grams, or longest common subsequence. They are useful when there are not too many acceptable formulations, but often miss real utility and actual correctness.", "blocks": [ { "kind": "code", "label": "formula", "language": "tex", "value": "ROUGE-1 recall = overlap_unigrams / reference_unigrams\nBLEU ~= precision in n-grams *brevity penalty\n\nRouge is more likely to watch \"what's covered,\"\nBLEU is more likely to look at “how similar the text is to the reference.”" }, { "kind": "code", "label": "Diagram", "language": "tex", "value": "reference: \"Customer cancels order due to price\"\noutput: \"Customer cancels order due to high price\"\n\nGeneral unigrams: customer, canceled, order, due, price\nROUGE-1 recall = 5 / 6 = 0.83\n\nThe meaning is almost identical, although the text is not identical." }, { "kind": "list", "label": "when to use", "items": [ "summarization baseline", "translation baseline", "Does not measure factuality directly", "bad for creative chat" ] } ] }, { "number": "19", "title": "Pass@k", "tag": "Generation", "description": "Pass@k answers the question: if we allow models to make multiple attempts, what is the probability that at least one of them will be correct. This is the main metric for code generation, unit-test solving and multi-sample decoding.", "blocks": [ { "kind": "code", "label": "formula", "language": "tex", "value": "If n candidates are sampled,\nAnd among them are the correct ones:\n\npass@k = 1 - C(n-c, k) / C(n, k)\n\nIntuitive:\nthe probability that among k attempts\nIt'll be at least one good one." }, { "kind": "code", "label": "Diagram", "language": "text", "value": "n = 10 samples\nc = 3 correct solutions\n\npass@1 = 3 / 10 = 0.30\npass@3 = 1 - C(7,3) / C(10,3)\n = 1 - 35/120\n = 0.708\n\nSeveral attempts dramatically increase the chance of success" }, { "kind": "list", "label": "Where it helps", "items": [ "code generation", "self-consistency decoding", "Increased cost and latency", "It is important to record the sampling policy" ] } ] }, { "number": "20", "title": "Pairwise win rate", "tag": "Generation", "description": "When it is difficult to make an absolute estimate, it is easier to ask which of the two answers is better. Pairwise win rate is convenient for comparison of models, prompts and postprocessors. It can be considered human-raters or LLM-as-judge.", "blocks": [ { "kind": "code", "label": "formula", "language": "tex", "value": "win rate = (wins + 0.5 * ties) / total_comparisons\n\nIf the tie is excluded:\nwin rate = wins / (wins + losses)" }, { "kind": "code", "label": "Diagram", "language": "text", "value": "100 comparisons A vs B\n54 wins A\n26 A defeats\n20 draws\n\nwin rate(A) = (54 + 0.5*20) / 100 = 0.64\n\nThat is, A is preferred in about 64% of comparisons." }, { "kind": "list", "label": "when to use", "items": [ "chat quality comparison", "prompt bake-off", "Position bias control", "stronger for ranking than for calibration" ] } ] }, { "number": "21", "title": "Calibration: Brier Score and ECE", "tag": "Calibration", "description": "Calibration checks whether the stated confidence of the model matches the actual success rate. This is critical for auto-approve, human handoff, self-check and policy-threshold solutions. The model may be accurate but poorly calibrated.", "blocks": [ { "kind": "code", "label": "formula", "language": "sql", "value": "Brier = mean((p_i - y_i)^2)\nwhere y i = 1 for success and 0 for error\n\nECE = Average gap between confidence and accuracy\nconfidence-baskets" }, { "kind": "code", "label": "Diagram", "language": "text", "value": "The model says confidence = 0.9.\nfor 100 answers\nBut only 60 of them are true.\n\naccuracy in this package = 0.60\ncalibration gap = 0.30\n\nFor one erroneous answer with p=0.9:\nBrier = (0.9 - 0)^2 = 0.81\n\nA well-calibrated model has 90% confidence = 90% success." }, { "kind": "list", "label": "Where it helps", "items": [ "auto-approve thresholds", "confidence-based routing", "ECE Depends on Bucketization", "Do not confuse high confidence with good calibration." ] } ] }, { "number": "22", "title": "Coverage, abstain rate and faithfulness", "tag": "Calibration", "description": "In many LLM systems, the model does not always have to respond. It can refuse, request a person or respond only with high confidence. In this case, accuracy is not enough: you need to watch coverage separately. For RAG, faithfulness almost always lives next to this, that is, the fraction of statements that are actually supported by the context.", "blocks": [ { "kind": "code", "label": "formula", "language": "tex", "value": "coverage = answered / total\nabstain rate = abstained / total = 1 - coverage\nselective risk = wrong_answered / answered\n\nfaithfulness = supported_claims / total_claims" }, { "kind": "code", "label": "Diagram", "language": "text", "value": "1,000 requests\nmodel responded to 700\nOf these, 665 are true.\n\ncoverage = 700 / 1000 = 0.70\nselective accuracy = 665 / 700 = 0.95\nselective risk = 35 / 700 = 0.05\n\nIn one answer, 10 factual claims,\n8 are supported by the context:\nfaithfulness = 8 / 10 = 0.80" }, { "kind": "list", "label": "Where it helps", "items": [ "abstention policies", "human handoff", "RAG grounding", "You cannot compare accuracy without coverage." ] } ] } ] }, { "number": "05", "title": "Ops, Product and Pipeline Card", "intro": "", "cards": [ { "number": "23", "title": "Latency, TTFT, p95 and tokens/sec", "tag": "Ops", "description": "For an LLM product, it is important to distinguish between “when the user saw the first token” and “when the answer ended.” That’s why we usually watch TTFT, end-to-end latency and generation rate. SLOs almost always need p95 or p99, not just the average.", "blocks": [ { "kind": "code", "label": "formula", "language": "tex", "value": "TTFT = time_to_first_token\nE2E latency = time_to_last_token\ntokens/sec = generated_tokens / decode_time\n\nFor operational control:\nmean + p95 + p99 + tok/s" }, { "kind": "code", "label": "Diagram", "language": "sql", "value": "One answer:\nTTFT = 0.6 sec\nE2E = 4.6 sec\ngenerated = 120 tokens\n\ndecode_time ~= 4.0 sec\ntokens/sec = 120 / 4.0 = 30 tok/s\n\nThe user feels TTFT.\nInfrastructure suffers from p95 E2E" }, { "kind": "list", "label": "Where it helps", "items": [ "serving SLO", "model / infra comparison", "Mean latency is not enough", "Separate prompt and decode time" ] } ] }, { "number": "24", "title": "Cost per request and cost per success", "tag": "Ops", "description": "A good system should not only be qualitative, but also economically sustainable. The average cost per request is useful for budget planning, and the cost of a successful outcome often better reflects the actual effectiveness of the solution.", "blocks": [ { "kind": "code", "label": "formula", "language": "tex", "value": "cost/request = total_cost / requests\ncost/success = total_cost / successful_requests\n\nIf the success rate falls,\nCost per success increases even at the same API price." }, { "kind": "code", "label": "Diagram", "language": "tex", "value": "$400 for 10,000 requests\nsuccess rate = 80%\n\ncost/request = 400 / 10 000 = $0.04\nsuccesses = 8 000\ncost/success = 400 / 8 000 = $0.05\n\nIt is cost/success that helps to compare complex pipelines." }, { "kind": "list", "label": "Where it helps", "items": [ "model selection", "multi-step pipeline economics", "watch with quality", "Cheaper is not always better than end-to-end." ] } ] }, { "number": "25", "title": "Online funnel: success, fallback, escalation, lift", "tag": "Ops", "description": "After offline eval, it is important to see how the system lives in production. Usually, you need a simple funnel: how many tasks are solved automatically, how many went into a fallback or to a person, how many ended in failure. For A/B comparison, delta and lift are added to this.", "blocks": [ { "kind": "code", "label": "formula", "language": "tex", "value": "success rate = solved / total\nfallback rate = fallback / total\nescalation rate = escalated / total\n\ndelta = metric_B - metric_A\nlift = (metric_B - metric_A) / metric_A" }, { "kind": "code", "label": "Diagram", "language": "tex", "value": "1,000 sessions\n820 solved\n120 fallback\n60 escalated\n\nsuccess rate = 82%\nfallback rate = 12%\nescalation rate = 6%\n\nA/B:\nA = 42% solved\nB = 46% solved\ndelta = +4 pp\nlift = +9.5%" }, { "kind": "list", "label": "Where it helps", "items": [ "assistant funnel", "agent handoff monitoring", "online A/B", "You need guardrails near success" ] } ] }, { "number": "26", "title": "Map of metrics by steps of LLM-pipeline", "tag": "Map.", "description": "The main rule of thumb is that each pipeline step should have its own local metric, and the entire system should have a separate end-to-end success. Don’t expect a single “magical” number to replace the entire diagnosis. A good eval stack is always multi-layered.", "blocks": [ { "kind": "code", "label": "Minimal logic", "language": "sql", "value": "step metric -> tells you exactly where the pipeline breaks\nend-to-end metric -> says whether the system solves the user's problem\nGuardrail metric -> says if you paid for the improvement with hidden risk\n\nPractice:\n1. keep local metrics in step\n2. keep the overall task success metric\n3. keep latency/cost/safety guardrails" }, { "kind": "table", "headers": [ "Step.", "Main metric", "What to keep close to", "What it does failure mean?" ], "rows": [ [ "Classifier / Router", "Recall, Precision, F1", "macro F1, confusion matrix", "The system is going the wrong workflow" ], [ "Safety / Guardrail", "Recall, FNR", "FPR, escalation rate", "Dangerous cases pass or everything is blocked" ], [ "Retriever", "Hit@k, Recall@k", "P@k, coverage", "generator has nothing to rely on" ], [ "Reranker", "MRR, MAP, nDCG@k", "latency, candidate recall", "useful documents, but are too low" ], [ "Extractor / Tool Call", "Exact Match, schema accuracy", "field-level recall, valid JSON rate", "Pipeline breaks down on structural step" ], [ "Generator", "task success, win rate, pass@k", "faithfulness, judge score", "The answer is formally beautiful but useless." ], [ "Confidence / Abstain", "Brier, ECE, coverage", "selective risk", "The model is overconfident or too silent" ], [ "Serving / Product", "p95 latency, cost/success", "TTFT, funnel success, lift", "Quality does not scale in the product" ] ] }, { "kind": "list", "label": "finalist", "items": [ "Local metrics for each step", "One common task success metric", "mandatory guardrails", "Don't argue about the quality of one number" ] } ] } ] } ] }