Vol. 33 · Evaluation & Improvement
Core Metrics for LLM Pipelines
One volume about the most useful metrics for LLM systems: from average, variance and standard deviation to accuracy, precision, recall, F1, MRR, MAP, nDCG@k, calibration, pass@k, latency and cost per success. Each card answers four questions: what it measures, how to calculate it, how to read, and where to apply it in the pipeline.
Basic statistics
01BaseMean and weighted average
The average answers the question “how much on average.” This is the most common aggregating metric for judge score, latency, cost, reward and batch quality. A weighted average is needed when observations are unequal: for example, requests differ in the number of tokens or importance.
texmean = Σx_i / n
weighted mean = Σ(w_i * x_i) / Σw_i
If all observations are equally important -> the usual average.
If the weight is different, use weights.texlatency = [1.2, 1.6, 3.2] sec
mean = (1.2 + 1.6 + 3.2) / 3 = 2.0 sec
token-weighted loss:
100 tokens with loss 0. 2.
900 tokens with loss 0.5
weighted mean = (100*0.2 + 900*0.5) / 1000 = 0.47
If you just average 0.2 and 0.5, you get 0.35 -> that's wrong.Where it helps
- average judge score
- cost / request
- middle-tail
- Don't just look at the mean latency.
02BaseMedian and percentiles: p50, p95, p99
The median shows a "typical" value that doesn't break down from rare emissions. Percentile p95 means 95% of observations are as good as this. For latency and token count, they are usually more important than average.
texp50 = median
p95 = value below which lies 95% of observations
p99 = value below which lies 99% of observations
Question to the metric:
"how bad is the system's tail?"texsorted latency = [100, 120, 140, 180, 900] ms
mean = 288 ms
p50 = 140 ms
p80 = 180 ms
p95 ~= 900 ms
The average says "like 288 ms."
p95 shows real tail painWhere it helps
- p95 latency
- tail token count
- median does not show worst-case
- p99 for SLO
03BaseDispersion and standard deviation
The variance and standard deviation show a spread around the mean. They are needed when it is important not only what the average score is, but also how stable it is from example, for example, from launch to launch and from annotator to annotator.
texvariance = Σ(x_i - mean)^2 / n
std = sqrt(variance)
The more std, the less predictable the outcome.
The std is measured in the same units as the reference metric.texSystem A: [0.7, 0.8, 0.9]
mean = 0.8, std ~= 0.08
System B: [0.2, 0.8, 1.4]
mean = 0.8, std ~= 0.49
The average is the same, the stability is different.Where it helps
- run-to-run variance
- judge consistency
- std without a mean of little information
- compare
04BaseShare, success rate, error rate
A lot of product and eval metrics are actually just fractions: how many queries worked, how many JSONs were valid, how many answers were grounded, how many times the system failed. This is the most universal type of metric.
texrate = successes / total
error rate = errors / total = 1 - success rate
The main rule:
Always indicate the denominator.
"87 errors" without "how many" is almost useless.tex100 requests
87 correct answers
13 incorrect
success rate = 87 / 100 = 0.87
error rate = 13 / 100 = 0.13Where it helps
- task success rate
- schema-valid rate
- hallucination rate
- You need a size sample.
05BaseConfidence interval, delta and lift
One number without uncertainty often overestimates confidence. The confidence interval shows the range of plausible values. Delta shows an absolute difference, lift shows relative growth relative to baseline.
texFor fraction p:
95% CI ~= p +/- 1.96 * sqrt(p(1-p)/n)
delta = metric_B - metric_A
lift = (metric_B - metric_A) / metric_AtexA: success = 78%
B: success = 81%
delta = 81% - 78% = +3 pp
lift = 3 / 78 = +3.8%
If p = 0.80 and n = 1000:
95% CI ~= 0.80 +/- 1.96 * sqrt(0.8*0.2/1000)
~= 0.80 +/- 0.025
~= [0.775, 0.825]Where it helps
- A/B comparison
- report uncertainty
- delta in pp!= lift in %
- Do not publish a point estimate
Classification and filters
06ClassificationConfusion Matrix: TP, FP, TN, FN
Almost all binary metrics are derived from four numbers. If you do not fix what is a positive class, then it is easy to confuse the meaning of precision, recall and false positive rate. For safety and routing, this is the basic table from which everything starts.
textTP: Model says 'positive' and it's true
FP: Model says 'positive' but it's false alarm
TN: Model says "negative" and it's true
FN: Model said "negative" but missed positive
Accuracy/precision/remember/F1
It's just a different relationship between these four numbers.text Truth
Positive Negative
Pred Positive TP FP
Pred Negative FN TN
Example of safety-filter:
TP = catching a harmful request
FP = block the normal request
FN = missed harmful request
TN = missed a normal requestWhere it helps
- toxicity filter
- intent classifier
- spam / abuse detector
- You have to ask positive class.
07ClassificationAccuracy and error rate
Accuracy shows a proportion of correct predictions. It is convenient when the classes are more or less balanced. With a strong imbalance, accuracy often looks more beautiful than the model is actually useful.
texaccuracy = (TP + TN) / (TP + FP + TN + FN)
error rate = 1 - accuracytext990 safe, 10 unsafe
Model says "safe" to everyone
accuracy = 990 / 1000 = 99%
recall unsafe = 0 / 10 = 0%
For safety, such accuracy is almost useless.when to use
- balanced-class
- class imbalance makes sense
- Do not use it alone for safety
08ClassificationPrecision and Recall
Precision answers the question, “If a model flagged something, how often is she right?” Recall answers the question, “What proportion of really important cases did the model find?” This is almost always the main pair of metrics for filters, detectors, and routers.
texprecision = TP / (TP + FP)
recall = TP / (TP + FN)
Precision -> Price of false alarm
recall -> skip pricetexThere are 10 harmful queries and 90 normal ones.
The filter caught 9 harmful and 3 normal hit in vain
TP = 9, FP = 3, FN = 1
precision = 9 / (9 + 3) = 0.75
recall = 9 / (9 + 1) = 0.90choose
- High recall for safety
- High precision for routing
- You can't improve one for free.
- threshold
09ClassificationSpecificity, FPR and FNR
These metrics are useful when it is separately important to control false locks and dangerous omissions. For safety-review, it is often more convenient to talk about the false negative rate, and for antifraud and moderation, false positive rate is sometimes more important.
texspecificity = TN / (TN + FP)
FPR = FP / (FP + TN) = 1 - specificity
FNR = FN / (FN + TP) = 1 - recallsqlFrom a previous example:
TN = 87, FP = 3, FN = 1, TP = 9
specificity = 87 / 90 = 0.967
FPR = 3 / 90 = 0.033
FNR = 1 / 10 = 0.10
FNR is especially important if the pass is expensive.Where it helps
- guardrails
- moderation
- abuse detection
- It is important to fix the threshold
10ClassificationF1 and F-beta
F1 is needed when both precision and recall are important at the same time, and you want one final digit. This is a harmonic average: a metric will not allow one side to mask the failure of the other. F-beta allows you to appreciate recall or precision.
texF1 = 2PR / (P + R)
F_beta = (1 + beta^2)PR / (beta^2 P + R)
Beta > 1 - More weight recall
Beta < 1 - More precision weighttextIf P = 0.75, R = 0.90:
F1 = 2 * 0.75 * 0.90 / (0.75 + 0.90)
= 0.818
F2 = 5 * 0.75 * 0.90 / (4*0.75 + 0.90)
= 0.865
F2 is higher because it rewards recall morewhen to use
- single headline metric
- model selection
- F1 hides which compromise is chosen
- F2 for safety/triage
11ClassificationMacro, Micro, Weighted, Balanced Accuracy and MCC
Once there are more than two classes or the data is unbalanced, the method of averaging is important. Macro gives all classes the same voice. Weighted gives more weight to frequent classes. Micro counts errors globally. Balanced accuracy and MCC are useful when looking more honestly at imbalanced data.
texmacro = mean(metric c by class)
weighted = Σ(support_c * metric_c) / Σsupport_c
micro = count TP/FP/FN globally, then metric
balanced accuracy = (recall + specificity) / 2
MCC = (TP*TN - FP*FN) / sqrt((TP+FP)(TP+FN)(TN+FP)(TN+FN))textClass A: 990 examples, F1 = 0.99
Class B: 10 examples, F1 = 0.20
macro F1 = (0.99 + 0.20) / 2 = 0.595
weighted F1 = (990*0.99 + 10*0.20) / 1000 = 0.982
Weighted almost does not notice a failure in a rare class
If the model is always predicting the majority class:
balanced accuracy ~= 0.50
MCC = 0 is more accurate than 99% accuracyselection rule
- macro for fairness by class
- micro
- MCC for imbalance
- Weighted hides minority fail
Retrieval and Ranking
12RankingHit@k / Success@k
Hit@k answers a simple question: is there at least one useful document in the top-k? For the first retrieval stage, this metric is especially convenient when the generator needs one good support, rather than a complete set of all relevant documents.
texHit@k for one request:
1 if there is at least one relevant item in the top-k
0 if not
Hit@k = Average of Requeststext10 requests for eval set
8 of them in the top 5 found at least 1 useful passage
Hit@5 = 8 / 10 = 0.80
It doesn't matter if it's 1st or 5th. nwhen to use
- first-stage retrieval
- RAG candidate search
- Do not see the quality of order inside the top k
- handy as a ceiling for a generator
13RankingPrecision@k and Recall@k
Precision@k shows how clean top-k is. Recall@k shows how much of all relevant documents retriever managed to bring. This is a basic pair of metrics for search and candidate generation.
texP@k = relevant_in_top_k / k
R@k = relevant_in_top_k / total_relevant
P@k -> issue noise
R@k -> Relevance coveragetextTotal relevant documents = 4
In the top 5 found 2 relevant
P@5 = 2 / 5 = 0.40
R@5 = 2 / 4 = 0.50
You can have good recall and bad precision and vice versa.Where it helps
- search quality
- RAG chunk retrieval
- We need relevance markups.
- look up
14RankingMRR: Mean Reciprocal Rank
The MRR measures how early the first benefit appears. If a user or generator needs one good document, MRR is usually more informative than recalling all relevant items.
texRR = 1 / rank_first_relevant
if the relevant item is not found -> RR = 0
MRR = mean(RR for all requests)text3 requests
First relevant in positions: [1, 2, 5]
RR = [1, 1/2, 1/5] = [1.0, 0.5, 0.2]
MRR = (1.0 + 0.5 + 0.2) / 3 = 0.567
Rank 1 is very much appreciated, rank 5 is already noticeably worse.when to use
- FAQ search
- single-best-doc retrieval
- Ignore second and third useful documents
- Good for QA and support search
15RankingAP and MAP
Average Precision evaluates the entire ranking at once: the metric increases when relevant documents are higher and appear earlier. MAP averages AP for all queries. This is a convenient metric when not one document is important, but the whole set of relevant elements.
sqlAP = average Precision@r
at all levels of r, where the relevant document
MAP = mean(AP on request)textRelevant documents are ranked 1, 3, 4
P@1 = 1/1 = 1.00
P@3 = 2/3 = 0.67
P@4 = 3/4 = 0.75
AP = (1.00 + 0.67 + 0.75) / 3 = 0.806
MAP = average of these APs for all requestsWhere it helps
- document retrieval
- reranker evaluation
- More difficult to explain to a business than Hit@k
- MRR is better when all relevant docs are important
16RankingDCG and nDCG@k: when relevance is graded
nDCG@k is needed when documents are not just relevant / not relevant, but have levels of utility: for example, 3 = perfect proof, 2 = useful, 1 = partially. The metric rewards important documents at the top and normalizes the result of a relatively perfect sort.
texDCG@k = Σ(rel_i / log2(i + 1))
nDCG@k = DCG@k / IDCG@k
IDCG = best possible DCG for this query
nDCG is always in the range [0, 1]textGraduated relevance:
ideal order = [3, 2, 0]
IDCG = 3/log2(2) + 2/log2(3)
= 3 + 1.262 = 4.262
Actual order = [2, 3, 0]
DCG = 2/log2(2) + 3/log2(3)
= 2 + 1.893 = 3.893
nDCG = 3.893 / 4.262 = 0.913
Almost good, but the most valuable document is not the first.Where it helps
- reranking
- graded search relevance
- human-labeled passage quality
- It is not binary but graded markup.
Generation, Judge and Reliability
17GenerationExact Match and schema accuracy
If the problem has an exact correct answer or a strict format, the exact metrics work best. For extraction, tool calling, JSON output and structured QA exact match is often more useful than any soft judge score.
texExact Match = exact_matches / total
Schema Accuracy = valid_schema_outputs / total
Either it matched or it didn't.text100 answers
92 valid JSONs
68 of them completely matched the fields.
schema accuracy = 92 / 100 = 0.92
exact match = 68 / 100 = 0.68Where it helps
- tool calling
- information extraction
- structured QA
- A very strict metric for open-ended chat
18GenerationROUGE, BLEU and overlap metrics
Overlap metrics compare model output to reference text for shared tokens, n-grams, or longest common subsequence. They are useful when there are not too many acceptable formulations, but often miss real utility and actual correctness.
texROUGE-1 recall = overlap_unigrams / reference_unigrams
BLEU ~= precision in n-grams *brevity penalty
Rouge is more likely to watch "what's covered,"
BLEU is more likely to look at “how similar the text is to the reference.”texreference: "Customer cancels order due to price"
output: "Customer cancels order due to high price"
General unigrams: customer, canceled, order, due, price
ROUGE-1 recall = 5 / 6 = 0.83
The meaning is almost identical, although the text is not identical.when to use
- summarization baseline
- translation baseline
- Does not measure factuality directly
- bad for creative chat
19GenerationPass@k
Pass@k answers the question: if we allow models to make multiple attempts, what is the probability that at least one of them will be correct. This is the main metric for code generation, unit-test solving and multi-sample decoding.
texIf n candidates are sampled,
And among them are the correct ones:
pass@k = 1 - C(n-c, k) / C(n, k)
Intuitive:
the probability that among k attempts
It'll be at least one good one.textn = 10 samples
c = 3 correct solutions
pass@1 = 3 / 10 = 0.30
pass@3 = 1 - C(7,3) / C(10,3)
= 1 - 35/120
= 0.708
Several attempts dramatically increase the chance of successWhere it helps
- code generation
- self-consistency decoding
- Increased cost and latency
- It is important to record the sampling policy
20GenerationPairwise win rate
When it is difficult to make an absolute estimate, it is easier to ask which of the two answers is better. Pairwise win rate is convenient for comparison of models, prompts and postprocessors. It can be considered human-raters or LLM-as-judge.
texwin rate = (wins + 0.5 * ties) / total_comparisons
If the tie is excluded:
win rate = wins / (wins + losses)text100 comparisons A vs B
54 wins A
26 A defeats
20 draws
win rate(A) = (54 + 0.5*20) / 100 = 0.64
That is, A is preferred in about 64% of comparisons.when to use
- chat quality comparison
- prompt bake-off
- Position bias control
- stronger for ranking than for calibration
21CalibrationCalibration: Brier Score and ECE
Calibration checks whether the stated confidence of the model matches the actual success rate. This is critical for auto-approve, human handoff, self-check and policy-threshold solutions. The model may be accurate but poorly calibrated.
sqlBrier = mean((p_i - y_i)^2)
where y i = 1 for success and 0 for error
ECE = Average gap between confidence and accuracy
confidence-basketstextThe model says confidence = 0.9.
for 100 answers
But only 60 of them are true.
accuracy in this package = 0.60
calibration gap = 0.30
For one erroneous answer with p=0.9:
Brier = (0.9 - 0)^2 = 0.81
A well-calibrated model has 90% confidence = 90% success.Where it helps
- auto-approve thresholds
- confidence-based routing
- ECE Depends on Bucketization
- Do not confuse high confidence with good calibration.
22CalibrationCoverage, abstain rate and faithfulness
In many LLM systems, the model does not always have to respond. It can refuse, request a person or respond only with high confidence. In this case, accuracy is not enough: you need to watch coverage separately. For RAG, faithfulness almost always lives next to this, that is, the fraction of statements that are actually supported by the context.
texcoverage = answered / total
abstain rate = abstained / total = 1 - coverage
selective risk = wrong_answered / answered
faithfulness = supported_claims / total_claimstext1,000 requests
model responded to 700
Of these, 665 are true.
coverage = 700 / 1000 = 0.70
selective accuracy = 665 / 700 = 0.95
selective risk = 35 / 700 = 0.05
In one answer, 10 factual claims,
8 are supported by the context:
faithfulness = 8 / 10 = 0.80Where it helps
- abstention policies
- human handoff
- RAG grounding
- You cannot compare accuracy without coverage.
Ops, Product and Pipeline Card
23OpsLatency, TTFT, p95 and tokens/sec
For an LLM product, it is important to distinguish between “when the user saw the first token” and “when the answer ended.” That’s why we usually watch TTFT, end-to-end latency and generation rate. SLOs almost always need p95 or p99, not just the average.
texTTFT = time_to_first_token
E2E latency = time_to_last_token
tokens/sec = generated_tokens / decode_time
For operational control:
mean + p95 + p99 + tok/ssqlOne answer:
TTFT = 0.6 sec
E2E = 4.6 sec
generated = 120 tokens
decode_time ~= 4.0 sec
tokens/sec = 120 / 4.0 = 30 tok/s
The user feels TTFT.
Infrastructure suffers from p95 E2EWhere it helps
- serving SLO
- model / infra comparison
- Mean latency is not enough
- Separate prompt and decode time
24OpsCost per request and cost per success
A good system should not only be qualitative, but also economically sustainable. The average cost per request is useful for budget planning, and the cost of a successful outcome often better reflects the actual effectiveness of the solution.
texcost/request = total_cost / requests
cost/success = total_cost / successful_requests
If the success rate falls,
Cost per success increases even at the same API price.tex$400 for 10,000 requests
success rate = 80%
cost/request = 400 / 10 000 = $0.04
successes = 8 000
cost/success = 400 / 8 000 = $0.05
It is cost/success that helps to compare complex pipelines.Where it helps
- model selection
- multi-step pipeline economics
- watch with quality
- Cheaper is not always better than end-to-end.
25OpsOnline funnel: success, fallback, escalation, lift
After offline eval, it is important to see how the system lives in production. Usually, you need a simple funnel: how many tasks are solved automatically, how many went into a fallback or to a person, how many ended in failure. For A/B comparison, delta and lift are added to this.
texsuccess rate = solved / total
fallback rate = fallback / total
escalation rate = escalated / total
delta = metric_B - metric_A
lift = (metric_B - metric_A) / metric_Atex1,000 sessions
820 solved
120 fallback
60 escalated
success rate = 82%
fallback rate = 12%
escalation rate = 6%
A/B:
A = 42% solved
B = 46% solved
delta = +4 pp
lift = +9.5%Where it helps
- assistant funnel
- agent handoff monitoring
- online A/B
- You need guardrails near success
26Map.Map of metrics by steps of LLM-pipeline
The main rule of thumb is that each pipeline step should have its own local metric, and the entire system should have a separate end-to-end success. Don’t expect a single “magical” number to replace the entire diagnosis. A good eval stack is always multi-layered.
sqlstep metric -> tells you exactly where the pipeline breaks
end-to-end metric -> says whether the system solves the user's problem
Guardrail metric -> says if you paid for the improvement with hidden risk
Practice:
1. keep local metrics in step
2. keep the overall task success metric
3. keep latency/cost/safety guardrails| Step. | Main metric | What to keep close to | What it does failure mean? |
|---|---|---|---|
| Classifier / Router | Recall, Precision, F1 | macro F1, confusion matrix | The system is going the wrong workflow |
| Safety / Guardrail | Recall, FNR | FPR, escalation rate | Dangerous cases pass or everything is blocked |
| Retriever | Hit@k, Recall@k | P@k, coverage | generator has nothing to rely on |
| Reranker | MRR, MAP, nDCG@k | latency, candidate recall | useful documents, but are too low |
| Extractor / Tool Call | Exact Match, schema accuracy | field-level recall, valid JSON rate | Pipeline breaks down on structural step |
| Generator | task success, win rate, pass@k | faithfulness, judge score | The answer is formally beautiful but useless. |
| Confidence / Abstain | Brier, ECE, coverage | selective risk | The model is overconfident or too silent |
| Serving / Product | p95 latency, cost/success | TTFT, funnel success, lift | Quality does not scale in the product |
finalist
- Local metrics for each step
- One common task success metric
- mandatory guardrails
- Don't argue about the quality of one number
No dead end
Keep moving through the map.
Continue in sequence, switch to a related guide, or return to the seven-track learning map.