Core Metrics for LLM Pipelines

The average answers the question “how much on average.” This is the most common aggregating metric for judge score, latency, cost, reward and batch quality. A weighted average is needed when observations are unequal: for example, requests differ in the number of tokens or importance.

The median shows a "typical" value that doesn't break down from rare emissions. Percentile p95 means 95% of observations are as good as this. For latency and token count, they are usually more important than average.

The variance and standard deviation show a spread around the mean. They are needed when it is important not only what the average score is, but also how stable it is from example, for example, from launch to launch and from annotator to annotator.

A lot of product and eval metrics are actually just fractions: how many queries worked, how many JSONs were valid, how many answers were grounded, how many times the system failed. This is the most universal type of metric.

One number without uncertainty often overestimates confidence. The confidence interval shows the range of plausible values. Delta shows an absolute difference, lift shows relative growth relative to baseline.

Almost all binary metrics are derived from four numbers. If you do not fix what is a positive class, then it is easy to confuse the meaning of precision, recall and false positive rate. For safety and routing, this is the basic table from which everything starts.

Accuracy shows a proportion of correct predictions. It is convenient when the classes are more or less balanced. With a strong imbalance, accuracy often looks more beautiful than the model is actually useful.

Precision answers the question, “If a model flagged something, how often is she right?” Recall answers the question, “What proportion of really important cases did the model find?” This is almost always the main pair of metrics for filters, detectors, and routers.

These metrics are useful when it is separately important to control false locks and dangerous omissions. For safety-review, it is often more convenient to talk about the false negative rate, and for antifraud and moderation, false positive rate is sometimes more important.

F1 is needed when both precision and recall are important at the same time, and you want one final digit. This is a harmonic average: a metric will not allow one side to mask the failure of the other. F-beta allows you to appreciate recall or precision.

Once there are more than two classes or the data is unbalanced, the method of averaging is important. Macro gives all classes the same voice. Weighted gives more weight to frequent classes. Micro counts errors globally. Balanced accuracy and MCC are useful when looking more honestly at imbalanced data.

Hit@k answers a simple question: is there at least one useful document in the top-k? For the first retrieval stage, this metric is especially convenient when the generator needs one good support, rather than a complete set of all relevant documents.

Precision@k shows how clean top-k is. Recall@k shows how much of all relevant documents retriever managed to bring. This is a basic pair of metrics for search and candidate generation.

The MRR measures how early the first benefit appears. If a user or generator needs one good document, MRR is usually more informative than recalling all relevant items.

Average Precision evaluates the entire ranking at once: the metric increases when relevant documents are higher and appear earlier. MAP averages AP for all queries. This is a convenient metric when not one document is important, but the whole set of relevant elements.

nDCG@k is needed when documents are not just relevant / not relevant, but have levels of utility: for example, 3 = perfect proof, 2 = useful, 1 = partially. The metric rewards important documents at the top and normalizes the result of a relatively perfect sort.

If the problem has an exact correct answer or a strict format, the exact metrics work best. For extraction, tool calling, JSON output and structured QA exact match is often more useful than any soft judge score.

Overlap metrics compare model output to reference text for shared tokens, n-grams, or longest common subsequence. They are useful when there are not too many acceptable formulations, but often miss real utility and actual correctness.

Pass@k answers the question: if we allow models to make multiple attempts, what is the probability that at least one of them will be correct. This is the main metric for code generation, unit-test solving and multi-sample decoding.

When it is difficult to make an absolute estimate, it is easier to ask which of the two answers is better. Pairwise win rate is convenient for comparison of models, prompts and postprocessors. It can be considered human-raters or LLM-as-judge.

Calibration checks whether the stated confidence of the model matches the actual success rate. This is critical for auto-approve, human handoff, self-check and policy-threshold solutions. The model may be accurate but poorly calibrated.

In many LLM systems, the model does not always have to respond. It can refuse, request a person or respond only with high confidence. In this case, accuracy is not enough: you need to watch coverage separately. For RAG, faithfulness almost always lives next to this, that is, the fraction of statements that are actually supported by the context.

For an LLM product, it is important to distinguish between “when the user saw the first token” and “when the answer ended.” That’s why we usually watch TTFT, end-to-end latency and generation rate. SLOs almost always need p95 or p99, not just the average.

A good system should not only be qualitative, but also economically sustainable. The average cost per request is useful for budget planning, and the cost of a successful outcome often better reflects the actual effectiveness of the solution.

After offline eval, it is important to see how the system lives in production. Usually, you need a simple funnel: how many tasks are solved automatically, how many went into a fallback or to a person, how many ended in failure. For A/B comparison, delta and lift are added to this.

The main rule of thumb is that each pipeline step should have its own local metric, and the entire system should have a separate end-to-end success. Don’t expect a single “magical” number to replace the entire diagnosis. A good eval stack is always multi-layered.