Statistics for Evaluating LLM Systems

Predicted + Predicted −
Actual + TP FN
Actual − FP TN

Accuracy = (TP+TN) / (TP+TN+FP+FN)
→ cheats: 99% TN → accuracy=99% effortless

Precision = TP / (TP+FP) ← of all the “+”, how many “+” are true?
Recall = TP / (TP+FN) ← of all the real “+”s, how many did you find?

F1 = 2 · (P R) / (P+R) ← harmonic mean

Macro F1: average F1 for all classes (imbalance not taken into account)
Weighted F1: class frequency weighted

BLEU (precision-oriented):
  BLEU-N = intersection of N-grams(gen, ref) / len(gen)
  ↑ penalizes for output that is too short (BP)
  ↑ usually average BLEU-1..4

ROUGE-N (recall-oriented):
  ROUGE-N = intersection of N-grams(gen, ref) / len(ref)

ROUGE-L: longest common subsequence

Example:
  ref: "the cat was sitting on the rug"
  gen: "the rug is occupied by a cat"
  BLEU-1 = 0.5 (2 words out of 3 are in ref)
  but the meaning is correct → BLEU does not see this

BERTScore: cosine proximity of embeddings
  ↑ better for semantic quality

Retrieval:
  Hit Rate@k - relevant doc in the top-k?
  MRR@k - position of the first relevant
  Precision@k - share of relevant ones in the top k

Generation (RAGAS):
  Faithfulness - Is the answer based on context?
  Answer Relevancy - does the answer answer the question?
  Context Precision - does the context contain the answer?
  Context Recall - has all the required context been found?

Faithfulness Formula (LLM-judge):
  score = (#statements from context) / (#all statements)
  → catches hallucinations

# n runs of the experiment, xᵢ is the result of the i-th

Sample mean:
  x̄ = (1/n) · Σᵢ xᵢ

Sample variance (unbiased):
  s² = 1/(n−1) Σᵢ (xᵢ − x̄)²
  ↑ divide by n-1, not n (Bessel correction)

Standard Deviation:
  s = √s² ← same units as xᵢ

Example: accuracy on 10 test sets:
  [0.82, 0.79, 0.84, 0.81, 0.83,
   0.80, 0.85, 0.78, 0.82, 0.81]
  x = 0.815
  s = 0.021 ← spread ≈ 2%

Standard error of the mean:
  SE = s/√n
  ↑ decreases with increasing sample
  ↑ for n=100: SE is 10 times less than n=1

95% Confidence Interval (normal):
  CI = x̄ ± 1.96 SE = x̄ ± 1.96 s/√n

Example:
  accuracy x̄ = 0.815, s = 0.021, n = 100
  SE = 0.021 / √100 = 0.0021
  95% CI = [0.811, 0.819]

  n = 10 → SE = 0.021/√10 = 0.0066
  95% CI = [0.802, 0.828] ← wide interval!

Conclusion: n=10 is a bad score. Need ≥ 50-100.

results = [0.82, 0.79, 0.84, 0.81, …] # n=100

bootstrap_means = []
for _ in range(10_000):
  sample = resample(results, n=100) # return
  bootstrap_means.append(mean(sample))

#95% CI = 2.5% and 97.5% percentiles
ci_low = percentile(bootstrap_means, 2.5)
ci_high = percentile(bootstrap_means, 97.5)

print(f"95% CI: [{ci_low:.3f}, {ci_high:.3f}]")

from scipy.stats import bootstrap
res = bootstrap((data,), np.mean, n_resamples=10000)

# Both prompts were run on the same 100 questions
scores_A = [0.8, 0.6, 1.0, …] # 100 values
scores_B = [0.9, 0.7, 1.0, …] # 100 values

diff = scores_B - scores_A # pairwise differences
# H₀: mean(diff) = 0

t = mean(diff) / (std(diff) / √n)
# compare with t-distribution (n-1 degrees of freedom)

from scipy.stats import ttest_rel
t_stat, p_value = ttest_rel(scores_B, scores_A)

Why paired is better than independent:
  removes variance from question difficulty
  → one complex issue affects both systems
  → difference is cleaner

# Ratings of two systems (1–5) using 50 examples:
grades_A = [3, 4, 3, 5, 2, …]
grades_B = [4, 4, 4, 5, 3, …]

from scipy.stats import wilcoxon
stat, p_value = wilcoxon(grades_B, grades_A,
                          alternative='greater')

Works through difference ranks:
  diff_i = B_i - A_i
  → rank |diff_i|
  → compare the sums of ranks “+” and “−”

When to use:
  ✓ ratings 1-5, 1-10 (LLM-judge, human eval)
  ✓ ordinal metrics (pass@k ranks)
  ✓ small samples (n < 30)

Family-wise error rate (FWER):
  P(at least 1 false alarm from m tests)
  = 1 − (1−α)^m

  m=45, α=0.05 → 1−(0.95)^45 = 0.90 = 90%!

Bonferroni correction:
  α* = α / m = 0.05 / 45 = 0.0011
  → use this threshold for each test

Benjamini-Hochberg correction (FDR):
  less strict, controls the rate of false discoveries
  → recommended for large numbers of comparisons

from statsmodels.stats.multitest import multipletests
reject, p_adj, _, _ = multipletests(
  p_values, method='bonferroni' # or 'fdr_bh'
)

Four related parameters (know 3 → find 4th):
  α (significance level, usually 0.05)
  power (1−β, typically 0.80)
  d (expected effect size)
  n (← what we are looking for)

Minimum n for paired t-test:
from statsmodels.stats.power import TTestPower
analysis = TTestPower()
n = analysis.solve_power(
  effect_size=0.3, # expected Cohen's d
  alpha=0.05,
  power=0.80
)
# → n ≈ 90 examples per group

Empirical guidelines for LLM:
  d=0.5 → n ≥ 34 (large expected effect)
  d=0.3 → n ≥ 90
  d=0.2 → n ≥ 200 (small effect - you need a lot)

① Formulate a hypothesis BEFORE the experiment
  "Prompt B gives higher F1 on task X"
  → not “let’s see what’s best”

② Fix the metric BEFORE the run
  → you cannot select a metric after, based on the best result

③ Use the SAME test set for A and B
  → paired test, removes variance from the difficulty of questions

④ The test set should not overlap with the dev set
  → otherwise you optimize for a test without knowledge

⑤ Run each system k≥3 times (at T>0)
  → average across runs, estimate σ as a function of temperature

⑥ Calculate n using power analysis IN ADVANCE
  → don't stop when p < 0.05 “for the first time”

# 100 comparisons A vs B
wins_B = 58 # B won 58 times
n = 100 # total comparisons
# H₀: p(B wins) = 0.5 (equal systems)

from scipy.stats import binomtest
result = binomtest(wins_B, n, p=0.5,
                  alternative='greater')
print(result.pvalue) # → 0.044 → significant!

# CI for win-rate (Wilson interval):
from statsmodels.stats.proportion import proportion_confint
ci = proportion_confint(wins_B, n, method='wilson')
# → (0.483, 0.671)

Minimum significant difference (α=0.05):
  n=50 → win-rate > 64% significant
  n=100 → win-rate > 59% significant
  n=200 → win-rate > 56% significant

# Expected probability of A winning over B:
E_A = 1 / (1 + 10^((R_B − R_A) / 400))

# Rating update after A vs B match:
# S_A = 1 (win), 0.5 (draw), 0 (loss)
R_A_new = R_A + K (S_A − E_A)
#K = learning factor (usually 32)

Initial rating: 1000 for everyone
Difference 400 → expected ~10% wins for the weakest

Bootstrap CI for Elo (Chatbot Arena approach):
→ resample paired matches 1000 times
→ recalculate Elo for each bootstrap
→ 95% CI by percentiles

Positional bias:
  judge([A, B]) → A wins 65% of the time
  judge([B, A]) → B wins 62% of the time
  → the first one wins both times!
  Defense: Always run in both orders
  Result: win only if you win in BOTH runs
  or take the average if there is a draw

Verbosity bias:
  judge prefers a longer answer
  even if it's less accurate
  Defense: explicitly in the judge prompt: “length is not a criterion”
  Defense: normalize the score by the length of the answer

Self-enhancement bias:
  Claude judge overestimates Claude answers
  GPT-4 judge overestimates GPT-4 responses
  Protection: use judge from another provider

Antipattern:
  System A: Prompt v7 (carefully written)
  System B: prompt v1 (draft)
  → you compare the quality of prompting, not systems

That's right - fix the prompt template:
  one template for both systems
  vary ONLY the parameter being studied

Or - multi-prompt evaluation:
  k=5 prompt options for each system
  estimate = mean ± std for all options
  → you evaluate robustness, not a point result

Sensitivity measurement:
  CV = std/mean (coefficient of variation)
  CV > 0.1 → system is unstable to formulation