--- type: "new-runtime-knowledge-guide" stable_id: "knowledge_guide:base-metrics" version: 1 generated_at: "2026-07-23" record_date: "2026-07-23" date_kind: "generated_at" slug: "base-metrics" title: "Core Metrics for LLM Pipelines" description: "One volume about the most useful metrics for LLM systems: from average, variance and standard deviation to accuracy, precision, recall, F1, MRR, MAP, nDCG@k, calibration, pass@k, latency and cost per success. Each card answers four questions: what it measures, how to calculate it, how to read, and where to apply it in the pipeline." retrieval_nugget: "One volume about the most useful metrics for LLM systems: from average, variance and standard deviation to accuracy, precision, recall, F1, MRR, MAP, nDCG@k, calibration, pass@k, latency and cost per success. Each card answers four questions: what it measures, how to calculate it, how to read, and where to apply it in the pipeline." track: "evaluation-and-improvement" volume: 33 --- # Core Metrics for LLM Pipelines ## Retrieval answer One volume about the most useful metrics for LLM systems: from average, variance and standard deviation to accuracy, precision, recall, F1, MRR, MAP, nDCG@k, calibration, pass@k, latency and cost per success. Each card answers four questions: what it measures, how to calculate it, how to read, and where to apply it in the pipeline. ## 01. Basic statistics ### 01.01 Mean and weighted average The average answers the question “how much on average.” This is the most common aggregating metric for judge score, latency, cost, reward and batch quality. A weighted average is needed when observations are unequal: for example, requests differ in the number of tokens or importance. #### formula ```tex mean = Σx_i / n weighted mean = Σ(w_i * x_i) / Σw_i If all observations are equally important -> the usual average. If the weight is different, use weights. ``` #### Diagram ```tex latency = [1.2, 1.6, 3.2] sec mean = (1.2 + 1.6 + 3.2) / 3 = 2.0 sec token-weighted loss: 100 tokens with loss 0. 2. 900 tokens with loss 0.5 weighted mean = (100*0.2 + 900*0.5) / 1000 = 0.47 If you just average 0.2 and 0.5, you get 0.35 -> that's wrong. ``` #### Where it helps - average judge score - cost / request - middle-tail - Don't just look at the mean latency. ### 01.02 Median and percentiles: p50, p95, p99 The median shows a "typical" value that doesn't break down from rare emissions. Percentile p95 means 95% of observations are as good as this. For latency and token count, they are usually more important than average. #### formula ```tex p50 = median p95 = value below which lies 95% of observations p99 = value below which lies 99% of observations Question to the metric: "how bad is the system's tail?" ``` #### Diagram ```tex sorted latency = [100, 120, 140, 180, 900] ms mean = 288 ms p50 = 140 ms p80 = 180 ms p95 ~= 900 ms The average says "like 288 ms." p95 shows real tail pain ``` #### Where it helps - p95 latency - tail token count - median does not show worst-case - p99 for SLO ### 01.03 Dispersion and standard deviation The variance and standard deviation show a spread around the mean. They are needed when it is important not only what the average score is, but also how stable it is from example, for example, from launch to launch and from annotator to annotator. #### formula ```tex variance = Σ(x_i - mean)^2 / n std = sqrt(variance) The more std, the less predictable the outcome. The std is measured in the same units as the reference metric. ``` #### Diagram ```tex System A: [0.7, 0.8, 0.9] mean = 0.8, std ~= 0.08 System B: [0.2, 0.8, 1.4] mean = 0.8, std ~= 0.49 The average is the same, the stability is different. ``` #### Where it helps - run-to-run variance - judge consistency - std without a mean of little information - compare ### 01.04 Share, success rate, error rate A lot of product and eval metrics are actually just fractions: how many queries worked, how many JSONs were valid, how many answers were grounded, how many times the system failed. This is the most universal type of metric. #### formula ```tex rate = successes / total error rate = errors / total = 1 - success rate The main rule: Always indicate the denominator. "87 errors" without "how many" is almost useless. ``` #### Diagram ```tex 100 requests 87 correct answers 13 incorrect success rate = 87 / 100 = 0.87 error rate = 13 / 100 = 0.13 ``` #### Where it helps - task success rate - schema-valid rate - hallucination rate - You need a size sample. ### 01.05 Confidence interval, delta and lift One number without uncertainty often overestimates confidence. The confidence interval shows the range of plausible values. Delta shows an absolute difference, lift shows relative growth relative to baseline. #### formula ```tex For fraction p: 95% CI ~= p +/- 1.96 * sqrt(p(1-p)/n) delta = metric_B - metric_A lift = (metric_B - metric_A) / metric_A ``` #### Diagram ```tex A: success = 78% B: success = 81% delta = 81% - 78% = +3 pp lift = 3 / 78 = +3.8% If p = 0.80 and n = 1000: 95% CI ~= 0.80 +/- 1.96 * sqrt(0.8*0.2/1000) ~= 0.80 +/- 0.025 ~= [0.775, 0.825] ``` #### Where it helps - A/B comparison - report uncertainty - delta in pp!= lift in % - Do not publish a point estimate ## 02. Classification and filters ### 02.06 Confusion Matrix: TP, FP, TN, FN Almost all binary metrics are derived from four numbers. If you do not fix what is a positive class, then it is easy to confuse the meaning of precision, recall and false positive rate. For safety and routing, this is the basic table from which everything starts. #### read ```text TP: Model says 'positive' and it's true FP: Model says 'positive' but it's false alarm TN: Model says "negative" and it's true FN: Model said "negative" but missed positive Accuracy/precision/remember/F1 It's just a different relationship between these four numbers. ``` #### Diagram ```text Truth Positive Negative Pred Positive TP FP Pred Negative FN TN Example of safety-filter: TP = catching a harmful request FP = block the normal request FN = missed harmful request TN = missed a normal request ``` #### Where it helps - toxicity filter - intent classifier - spam / abuse detector - You have to ask positive class. ### 02.07 Accuracy and error rate Accuracy shows a proportion of correct predictions. It is convenient when the classes are more or less balanced. With a strong imbalance, accuracy often looks more beautiful than the model is actually useful. #### formula ```tex accuracy = (TP + TN) / (TP + FP + TN + FN) error rate = 1 - accuracy ``` #### Diagram ```text 990 safe, 10 unsafe Model says "safe" to everyone accuracy = 990 / 1000 = 99% recall unsafe = 0 / 10 = 0% For safety, such accuracy is almost useless. ``` #### when to use - balanced-class - class imbalance makes sense - Do not use it alone for safety ### 02.08 Precision and Recall Precision answers the question, “If a model flagged something, how often is she right?” Recall answers the question, “What proportion of really important cases did the model find?” This is almost always the main pair of metrics for filters, detectors, and routers. #### formula ```tex precision = TP / (TP + FP) recall = TP / (TP + FN) Precision -> Price of false alarm recall -> skip price ``` #### Diagram ```tex There are 10 harmful queries and 90 normal ones. The filter caught 9 harmful and 3 normal hit in vain TP = 9, FP = 3, FN = 1 precision = 9 / (9 + 3) = 0.75 recall = 9 / (9 + 1) = 0.90 ``` #### choose - High recall for safety - High precision for routing - You can't improve one for free. - threshold ### 02.09 Specificity, FPR and FNR These metrics are useful when it is separately important to control false locks and dangerous omissions. For safety-review, it is often more convenient to talk about the false negative rate, and for antifraud and moderation, false positive rate is sometimes more important. #### formula ```tex specificity = TN / (TN + FP) FPR = FP / (FP + TN) = 1 - specificity FNR = FN / (FN + TP) = 1 - recall ``` #### Diagram ```sql From a previous example: TN = 87, FP = 3, FN = 1, TP = 9 specificity = 87 / 90 = 0.967 FPR = 3 / 90 = 0.033 FNR = 1 / 10 = 0.10 FNR is especially important if the pass is expensive. ``` #### Where it helps - guardrails - moderation - abuse detection - It is important to fix the threshold ### 02.10 F1 and F-beta F1 is needed when both precision and recall are important at the same time, and you want one final digit. This is a harmonic average: a metric will not allow one side to mask the failure of the other. F-beta allows you to appreciate recall or precision. #### formula ```tex F1 = 2PR / (P + R) F_beta = (1 + beta^2)PR / (beta^2 P + R) Beta > 1 - More weight recall Beta < 1 - More precision weight ``` #### Diagram ```text If P = 0.75, R = 0.90: F1 = 2 * 0.75 * 0.90 / (0.75 + 0.90) = 0.818 F2 = 5 * 0.75 * 0.90 / (4*0.75 + 0.90) = 0.865 F2 is higher because it rewards recall more ``` #### when to use - single headline metric - model selection - F1 hides which compromise is chosen - F2 for safety/triage ### 02.11 Macro, Micro, Weighted, Balanced Accuracy and MCC Once there are more than two classes or the data is unbalanced, the method of averaging is important. Macro gives all classes the same voice. Weighted gives more weight to frequent classes. Micro counts errors globally. Balanced accuracy and MCC are useful when looking more honestly at imbalanced data. #### formula ```tex macro = mean(metric c by class) weighted = Σ(support_c * metric_c) / Σsupport_c micro = count TP/FP/FN globally, then metric balanced accuracy = (recall + specificity) / 2 MCC = (TP*TN - FP*FN) / sqrt((TP+FP)(TP+FN)(TN+FP)(TN+FN)) ``` #### Diagram ```text Class A: 990 examples, F1 = 0.99 Class B: 10 examples, F1 = 0.20 macro F1 = (0.99 + 0.20) / 2 = 0.595 weighted F1 = (990*0.99 + 10*0.20) / 1000 = 0.982 Weighted almost does not notice a failure in a rare class If the model is always predicting the majority class: balanced accuracy ~= 0.50 MCC = 0 is more accurate than 99% accuracy ``` #### selection rule - macro for fairness by class - micro - MCC for imbalance - Weighted hides minority fail ## 03. Retrieval and Ranking ### 03.12 Hit@k / Success@k Hit@k answers a simple question: is there at least one useful document in the top-k? For the first retrieval stage, this metric is especially convenient when the generator needs one good support, rather than a complete set of all relevant documents. #### formula ```tex Hit@k for one request: 1 if there is at least one relevant item in the top-k 0 if not Hit@k = Average of Requests ``` #### Diagram ```text 10 requests for eval set 8 of them in the top 5 found at least 1 useful passage Hit@5 = 8 / 10 = 0.80 It doesn't matter if it's 1st or 5th. ``` #### when to use - first-stage retrieval - RAG candidate search - Do not see the quality of order inside the top k - handy as a ceiling for a generator ### 03.13 Precision@k and Recall@k Precision@k shows how clean top-k is. Recall@k shows how much of all relevant documents retriever managed to bring. This is a basic pair of metrics for search and candidate generation. #### formula ```tex P@k = relevant_in_top_k / k R@k = relevant_in_top_k / total_relevant P@k -> issue noise R@k -> Relevance coverage ``` #### Diagram ```text Total relevant documents = 4 In the top 5 found 2 relevant P@5 = 2 / 5 = 0.40 R@5 = 2 / 4 = 0.50 You can have good recall and bad precision and vice versa. ``` #### Where it helps - search quality - RAG chunk retrieval - We need relevance markups. - look up ### 03.14 MRR: Mean Reciprocal Rank The MRR measures how early the first benefit appears. If a user or generator needs one good document, MRR is usually more informative than recalling all relevant items. #### formula ```tex RR = 1 / rank_first_relevant if the relevant item is not found -> RR = 0 MRR = mean(RR for all requests) ``` #### Diagram ```text 3 requests First relevant in positions: [1, 2, 5] RR = [1, 1/2, 1/5] = [1.0, 0.5, 0.2] MRR = (1.0 + 0.5 + 0.2) / 3 = 0.567 Rank 1 is very much appreciated, rank 5 is already noticeably worse. ``` #### when to use - FAQ search - single-best-doc retrieval - Ignore second and third useful documents - Good for QA and support search ### 03.15 AP and MAP Average Precision evaluates the entire ranking at once: the metric increases when relevant documents are higher and appear earlier. MAP averages AP for all queries. This is a convenient metric when not one document is important, but the whole set of relevant elements. #### formula ```sql AP = average Precision@r at all levels of r, where the relevant document MAP = mean(AP on request) ``` #### Diagram ```text Relevant documents are ranked 1, 3, 4 P@1 = 1/1 = 1.00 P@3 = 2/3 = 0.67 P@4 = 3/4 = 0.75 AP = (1.00 + 0.67 + 0.75) / 3 = 0.806 MAP = average of these APs for all requests ``` #### Where it helps - document retrieval - reranker evaluation - More difficult to explain to a business than Hit@k - MRR is better when all relevant docs are important ### 03.16 DCG and nDCG@k: when relevance is graded nDCG@k is needed when documents are not just relevant / not relevant, but have levels of utility: for example, 3 = perfect proof, 2 = useful, 1 = partially. The metric rewards important documents at the top and normalizes the result of a relatively perfect sort. #### formula ```tex DCG@k = Σ(rel_i / log2(i + 1)) nDCG@k = DCG@k / IDCG@k IDCG = best possible DCG for this query nDCG is always in the range [0, 1] ``` #### Diagram ```text Graduated relevance: ideal order = [3, 2, 0] IDCG = 3/log2(2) + 2/log2(3) = 3 + 1.262 = 4.262 Actual order = [2, 3, 0] DCG = 2/log2(2) + 3/log2(3) = 2 + 1.893 = 3.893 nDCG = 3.893 / 4.262 = 0.913 Almost good, but the most valuable document is not the first. ``` #### Where it helps - reranking - graded search relevance - human-labeled passage quality - It is not binary but graded markup. ## 04. Generation, Judge and Reliability ### 04.17 Exact Match and schema accuracy If the problem has an exact correct answer or a strict format, the exact metrics work best. For extraction, tool calling, JSON output and structured QA exact match is often more useful than any soft judge score. #### formula ```tex Exact Match = exact_matches / total Schema Accuracy = valid_schema_outputs / total Either it matched or it didn't. ``` #### Diagram ```text 100 answers 92 valid JSONs 68 of them completely matched the fields. schema accuracy = 92 / 100 = 0.92 exact match = 68 / 100 = 0.68 ``` #### Where it helps - tool calling - information extraction - structured QA - A very strict metric for open-ended chat ### 04.18 ROUGE, BLEU and overlap metrics Overlap metrics compare model output to reference text for shared tokens, n-grams, or longest common subsequence. They are useful when there are not too many acceptable formulations, but often miss real utility and actual correctness. #### formula ```tex ROUGE-1 recall = overlap_unigrams / reference_unigrams BLEU ~= precision in n-grams *brevity penalty Rouge is more likely to watch "what's covered," BLEU is more likely to look at “how similar the text is to the reference.” ``` #### Diagram ```tex reference: "Customer cancels order due to price" output: "Customer cancels order due to high price" General unigrams: customer, canceled, order, due, price ROUGE-1 recall = 5 / 6 = 0.83 The meaning is almost identical, although the text is not identical. ``` #### when to use - summarization baseline - translation baseline - Does not measure factuality directly - bad for creative chat ### 04.19 Pass@k Pass@k answers the question: if we allow models to make multiple attempts, what is the probability that at least one of them will be correct. This is the main metric for code generation, unit-test solving and multi-sample decoding. #### formula ```tex If n candidates are sampled, And among them are the correct ones: pass@k = 1 - C(n-c, k) / C(n, k) Intuitive: the probability that among k attempts It'll be at least one good one. ``` #### Diagram ```text n = 10 samples c = 3 correct solutions pass@1 = 3 / 10 = 0.30 pass@3 = 1 - C(7,3) / C(10,3) = 1 - 35/120 = 0.708 Several attempts dramatically increase the chance of success ``` #### Where it helps - code generation - self-consistency decoding - Increased cost and latency - It is important to record the sampling policy ### 04.20 Pairwise win rate When it is difficult to make an absolute estimate, it is easier to ask which of the two answers is better. Pairwise win rate is convenient for comparison of models, prompts and postprocessors. It can be considered human-raters or LLM-as-judge. #### formula ```tex win rate = (wins + 0.5 * ties) / total_comparisons If the tie is excluded: win rate = wins / (wins + losses) ``` #### Diagram ```text 100 comparisons A vs B 54 wins A 26 A defeats 20 draws win rate(A) = (54 + 0.5*20) / 100 = 0.64 That is, A is preferred in about 64% of comparisons. ``` #### when to use - chat quality comparison - prompt bake-off - Position bias control - stronger for ranking than for calibration ### 04.21 Calibration: Brier Score and ECE Calibration checks whether the stated confidence of the model matches the actual success rate. This is critical for auto-approve, human handoff, self-check and policy-threshold solutions. The model may be accurate but poorly calibrated. #### formula ```sql Brier = mean((p_i - y_i)^2) where y i = 1 for success and 0 for error ECE = Average gap between confidence and accuracy confidence-baskets ``` #### Diagram ```text The model says confidence = 0.9. for 100 answers But only 60 of them are true. accuracy in this package = 0.60 calibration gap = 0.30 For one erroneous answer with p=0.9: Brier = (0.9 - 0)^2 = 0.81 A well-calibrated model has 90% confidence = 90% success. ``` #### Where it helps - auto-approve thresholds - confidence-based routing - ECE Depends on Bucketization - Do not confuse high confidence with good calibration. ### 04.22 Coverage, abstain rate and faithfulness In many LLM systems, the model does not always have to respond. It can refuse, request a person or respond only with high confidence. In this case, accuracy is not enough: you need to watch coverage separately. For RAG, faithfulness almost always lives next to this, that is, the fraction of statements that are actually supported by the context. #### formula ```tex coverage = answered / total abstain rate = abstained / total = 1 - coverage selective risk = wrong_answered / answered faithfulness = supported_claims / total_claims ``` #### Diagram ```text 1,000 requests model responded to 700 Of these, 665 are true. coverage = 700 / 1000 = 0.70 selective accuracy = 665 / 700 = 0.95 selective risk = 35 / 700 = 0.05 In one answer, 10 factual claims, 8 are supported by the context: faithfulness = 8 / 10 = 0.80 ``` #### Where it helps - abstention policies - human handoff - RAG grounding - You cannot compare accuracy without coverage. ## 05. Ops, Product and Pipeline Card ### 05.23 Latency, TTFT, p95 and tokens/sec For an LLM product, it is important to distinguish between “when the user saw the first token” and “when the answer ended.” That’s why we usually watch TTFT, end-to-end latency and generation rate. SLOs almost always need p95 or p99, not just the average. #### formula ```tex TTFT = time_to_first_token E2E latency = time_to_last_token tokens/sec = generated_tokens / decode_time For operational control: mean + p95 + p99 + tok/s ``` #### Diagram ```sql One answer: TTFT = 0.6 sec E2E = 4.6 sec generated = 120 tokens decode_time ~= 4.0 sec tokens/sec = 120 / 4.0 = 30 tok/s The user feels TTFT. Infrastructure suffers from p95 E2E ``` #### Where it helps - serving SLO - model / infra comparison - Mean latency is not enough - Separate prompt and decode time ### 05.24 Cost per request and cost per success A good system should not only be qualitative, but also economically sustainable. The average cost per request is useful for budget planning, and the cost of a successful outcome often better reflects the actual effectiveness of the solution. #### formula ```tex cost/request = total_cost / requests cost/success = total_cost / successful_requests If the success rate falls, Cost per success increases even at the same API price. ``` #### Diagram ```tex $400 for 10,000 requests success rate = 80% cost/request = 400 / 10 000 = $0.04 successes = 8 000 cost/success = 400 / 8 000 = $0.05 It is cost/success that helps to compare complex pipelines. ``` #### Where it helps - model selection - multi-step pipeline economics - watch with quality - Cheaper is not always better than end-to-end. ### 05.25 Online funnel: success, fallback, escalation, lift After offline eval, it is important to see how the system lives in production. Usually, you need a simple funnel: how many tasks are solved automatically, how many went into a fallback or to a person, how many ended in failure. For A/B comparison, delta and lift are added to this. #### formula ```tex success rate = solved / total fallback rate = fallback / total escalation rate = escalated / total delta = metric_B - metric_A lift = (metric_B - metric_A) / metric_A ``` #### Diagram ```tex 1,000 sessions 820 solved 120 fallback 60 escalated success rate = 82% fallback rate = 12% escalation rate = 6% A/B: A = 42% solved B = 46% solved delta = +4 pp lift = +9.5% ``` #### Where it helps - assistant funnel - agent handoff monitoring - online A/B - You need guardrails near success ### 05.26 Map of metrics by steps of LLM-pipeline The main rule of thumb is that each pipeline step should have its own local metric, and the entire system should have a separate end-to-end success. Don’t expect a single “magical” number to replace the entire diagnosis. A good eval stack is always multi-layered. #### Minimal logic ```sql step metric -> tells you exactly where the pipeline breaks end-to-end metric -> says whether the system solves the user's problem Guardrail metric -> says if you paid for the improvement with hidden risk Practice: 1. keep local metrics in step 2. keep the overall task success metric 3. keep latency/cost/safety guardrails ``` | Step. | Main metric | What to keep close to | What it does failure mean? | | --- | --- | --- | --- | | Classifier / Router | Recall, Precision, F1 | macro F1, confusion matrix | The system is going the wrong workflow | | Safety / Guardrail | Recall, FNR | FPR, escalation rate | Dangerous cases pass or everything is blocked | | Retriever | Hit@k, Recall@k | P@k, coverage | generator has nothing to rely on | | Reranker | MRR, MAP, nDCG@k | latency, candidate recall | useful documents, but are too low | | Extractor / Tool Call | Exact Match, schema accuracy | field-level recall, valid JSON rate | Pipeline breaks down on structural step | | Generator | task success, win rate, pass@k | faithfulness, judge score | The answer is formally beautiful but useless. | | Confidence / Abstain | Brier, ECE, coverage | selective risk | The model is overconfident or too silent | | Serving / Product | p95 latency, cost/success | TTFT, funnel success, lift | Quality does not scale in the product | #### finalist - Local metrics for each step - One common task success metric - mandatory guardrails - Don't argue about the quality of one number