Statistics for Evaluating LLM Systems

For one token y with predicted probability p(y):

  log-loss = −log p(y)
  ↑ if the model is confident: p(y)=0.9 → loss = 0.105
  ↑ if you are not sure: p(y)=0.1 → loss = 2.303

For the entire sequence of N tokens:

  Log-Loss = −(1/N) · Σᵢ log p(yᵢ | y₁…y_{i-1})
  ↑ average for all tokens

  Perplexity = exp(Log-Loss) = exp(−(1/N) · Σ log p(yᵢ))
  ↑ interpretation: average “selection size” of the model

Examples of PPL:
  PPL = 1000 ← bad model or out-of-domain text
  PPL = 20 ← GPT-2 on WikiText-103
  PPL = 6 ← Llama-3 70B in English text
  PPL = 1 ← ideal model (theoretically)

Communication with information bits:
  bits-per-char = log-loss / log(2) ← alternative notation

Predicted + Predicted −
Actual + TP FN
Actual − FP TN

Accuracy = (TP+TN) / (TP+TN+FP+FN)
→ cheats: 99% TN → accuracy=99% effortless

Precision = TP / (TP+FP) ← of all the “+”, how many “+” are true?
Recall = TP / (TP+FN) ← of all the real “+”s, how many did you find?

F1 = 2 · (P R) / (P+R) ← harmonic mean

Macro F1: average F1 for all classes (imbalance not taken into account)
Weighted F1: class frequency weighted

BLEU (precision-oriented):
  BLEU-N = intersection of N-grams(gen, ref) / len(gen)
  ↑ penalizes for output that is too short (BP)
  ↑ usually average BLEU-1..4

ROUGE-N (recall-oriented):
  ROUGE-N = intersection of N-grams(gen, ref) / len(ref)

ROUGE-L: longest common subsequence

Example:
  ref: "the cat was sitting on the rug"
  gen: "the rug is occupied by a cat"
  BLEU-1 = 0.5 (2 words out of 3 are in ref)
  but the meaning is correct → BLEU does not see this

BERTScore: cosine proximity of embeddings
  ↑ better for semantic quality

Retrieval:
  Hit Rate@k - relevant doc in the top-k?
  MRR@k - position of the first relevant
  Precision@k - share of relevant ones in the top k

Generation (RAGAS):
  Faithfulness - Is the answer based on context?
  Answer Relevancy - does the answer answer the question?
  Context Precision - does the context contain the answer?
  Context Recall - has all the required context been found?

Faithfulness Formula (LLM-judge):
  score = (#statements from context) / (#all statements)
  → catches hallucinations

# n runs of the experiment, xᵢ is the result of the i-th

Sample mean:
  x̄ = (1/n) · Σᵢ xᵢ

Sample variance (unbiased):
  s² = 1/(n−1) Σᵢ (xᵢ − x̄)²
  ↑ divide by n-1, not n (Bessel correction)

Standard Deviation:
  s = √s² ← same units as xᵢ

Example: accuracy on 10 test sets:
  [0.82, 0.79, 0.84, 0.81, 0.83,
   0.80, 0.85, 0.78, 0.82, 0.81]
  x = 0.815
  s = 0.021 ← spread ≈ 2%

Standard error of the mean:
  SE = s/√n
  ↑ decreases with increasing sample
  ↑ for n=100: SE is 10 times less than n=1

95% Confidence Interval (normal):
  CI = x̄ ± 1.96 SE = x̄ ± 1.96 s/√n

Example:
  accuracy x̄ = 0.815, s = 0.021, n = 100
  SE = 0.021 / √100 = 0.0021
  95% CI = [0.811, 0.819]

  n = 10 → SE = 0.021/√10 = 0.0066
  95% CI = [0.802, 0.828] ← wide interval!

Conclusion: n=10 is a bad score. Need ≥ 50-100.

results = [0.82, 0.79, 0.84, 0.81, …] # n=100

bootstrap_means = []
for _ in range(10_000):
  sample = resample(results, n=100) # return
  bootstrap_means.append(mean(sample))

#95% CI = 2.5% and 97.5% percentiles
ci_low = percentile(bootstrap_means, 2.5)
ci_high = percentile(bootstrap_means, 97.5)

print(f"95% CI: [{ci_low:.3f}, {ci_high:.3f}]")

from scipy.stats import bootstrap
res = bootstrap((data,), np.mean, n_resamples=10000)

Sources of variance:

① Temperature/sampling
  same prompt → different answers
  → run each prompt k≥3 times, take the average

② Prompt sensitivity
  "Answer briefly" vs "Give a short answer"
  → the difference can be ±5–10% on the metric

③ Test set sampling
  100 questions → another 100 questions → another result
  → bootstrap based on test set examples

④ Dispersion of the judge model
  LLM-as-judge is itself non-deterministic
  → run judge k≥3, consider inter-rater agreement

⑤ Order of few-shot examples
  mix → different result
  → average over several orders

Staging:
  H₀ (null): "prompt A and prompt B give the same result"
  H₁ (alternative): "prompt B is better than prompt A"

p-value = P(get this or more extreme difference | H₀ is correct)

Example:
  Prompt A: accuracy = 0.780 on 200 examples
  Prompt B: accuracy = 0.810 on 200 examples
  p-value = 0.032

Correct interpretation:
  “If H₀ were true, the probability of seeing a difference ≥ 0.03 by chance = 3.2%”
  At threshold α=0.05 → we reject H₀, the difference is statistically significant

WRONG interpretation (common mistakes):
  ✗ “probability that H₀ is correct = 3.2%” ← no
  ✗ “probability that the result is random = 3.2%” ← no
  ✗ “the effect is large and important” ← p does not indicate the size of the effect
  ✗ “p=0.049 and p=0.051 are fundamentally different” ← no, this is a continuous scale

# Both prompts were run on the same 100 questions
scores_A = [0.8, 0.6, 1.0, …] # 100 values
scores_B = [0.9, 0.7, 1.0, …] # 100 values

diff = scores_B - scores_A # pairwise differences
# H₀: mean(diff) = 0

t = mean(diff) / (std(diff) / √n)
# compare with t-distribution (n-1 degrees of freedom)

from scipy.stats import ttest_rel
t_stat, p_value = ttest_rel(scores_B, scores_A)

Why paired is better than independent:
  removes variance from question difficulty
  → one complex issue affects both systems
  → difference is cleaner

# Ratings of two systems (1–5) using 50 examples:
grades_A = [3, 4, 3, 5, 2, …]
grades_B = [4, 4, 4, 5, 3, …]

from scipy.stats import wilcoxon
stat, p_value = wilcoxon(grades_B, grades_A,
                          alternative='greater')

Works through difference ranks:
  diff_i = B_i - A_i
  → rank |diff_i|
  → compare the sums of ranks “+” and “−”

When to use:
  ✓ ratings 1-5, 1-10 (LLM-judge, human eval)
  ✓ ordinal metrics (pass@k ranks)
  ✓ small samples (n < 30)

Family-wise error rate (FWER):
  P(at least 1 false alarm from m tests)
  = 1 − (1−α)^m

  m=45, α=0.05 → 1−(0.95)^45 = 0.90 = 90%!

Bonferroni correction:
  α* = α / m = 0.05 / 45 = 0.0011
  → use this threshold for each test

Benjamini-Hochberg correction (FDR):
  less strict, controls the rate of false discoveries
  → recommended for large numbers of comparisons

from statsmodels.stats.multitest import multipletests
reject, p_adj, _, _ = multipletests(
  p_values, method='bonferroni' # or 'fdr_bh'
)

Cohen's d - standardized mean difference:

  d = (μ_B − μ_A) / s_pooled
  s_pooled = √( (s_A² + s_B²) / 2 )

Cohen's d interpretation:
  d = 0.2 → small effect (almost imperceptible)
  d = 0.5 → average effect (noticeable in use)
  d = 0.8 → large effect (obviously better)

Example:
  Prompt A: x̄=0.780, s=0.12
  Prompt B: x̄=0.810, s=0.11
  s_pooled = √((0.0144+0.0121)/2) = 0.115
  d = (0.810−0.780) / 0.115 = 0.26 ← small effect

  n=200 → p=0.031 → statistically significant
  d=0.26 → but practically a small difference

Conclusion: always report BOTH p-value AND effect size.

Four related parameters (know 3 → find 4th):
  α (significance level, usually 0.05)
  power (1−β, typically 0.80)
  d (expected effect size)
  n (← what we are looking for)

Minimum n for paired t-test:
from statsmodels.stats.power import TTestPower
analysis = TTestPower()
n = analysis.solve_power(
  effect_size=0.3, # expected Cohen's d
  alpha=0.05,
  power=0.80
)
# → n ≈ 90 examples per group

Empirical guidelines for LLM:
  d=0.5 → n ≥ 34 (large expected effect)
  d=0.3 → n ≥ 90
  d=0.2 → n ≥ 200 (small effect - you need a lot)

① Formulate a hypothesis BEFORE the experiment
  "Prompt B gives higher F1 on task X"
  → not “let’s see what’s best”

② Fix the metric BEFORE the run
  → you cannot select a metric after, based on the best result

③ Use the SAME test set for A and B
  → paired test, removes variance from the difficulty of questions

④ The test set should not overlap with the dev set
  → otherwise you optimize for a test without knowledge

⑤ Run each system k≥3 times (at T>0)
  → average across runs, estimate σ as a function of temperature

⑥ Calculate n using power analysis IN ADVANCE
  → don't stop when p < 0.05 “for the first time”

Three questions about quality judge:

① Intra-rater reliability
  Run judge on some examples k=5 times
  σ_judge = std(scores per example)
  → if σ_judge > 0.5 points out of 5 → judge is unstable
  → average 3–5 runs before the final score

② Human correlation
  Take 50–100 examples from human labels
  Spearman ρ = rank correlation(judge_score, human_score)
  ρ > 0.7 → judge is quite reliable
  ρ < 0.5 → judge is not suitable for this task

③ Consistency between judge models
  Claude judge vs GPT-4 judge → correlation?
  Cohen's κ for binary solutions (pass/fail)
  κ > 0.6 → good agreement

Inter-rater reliability (κ for two evaluators):
  κ = (P_observed − P_chance) / (1 − P_chance)
  κ = 0.2–0.4 → weak agreement
  κ = 0.4–0.6 → moderate
  κ = 0.6–0.8 → good → can be trusted

# 100 comparisons A vs B
wins_B = 58 # B won 58 times
n = 100 # total comparisons
# H₀: p(B wins) = 0.5 (equal systems)

from scipy.stats import binomtest
result = binomtest(wins_B, n, p=0.5,
                  alternative='greater')
print(result.pvalue) # → 0.044 → significant!

# CI for win-rate (Wilson interval):
from statsmodels.stats.proportion import proportion_confint
ci = proportion_confint(wins_B, n, method='wilson')
# → (0.483, 0.671)

Minimum significant difference (α=0.05):
  n=50 → win-rate > 64% significant
  n=100 → win-rate > 59% significant
  n=200 → win-rate > 56% significant

# Expected probability of A winning over B:
E_A = 1 / (1 + 10^((R_B − R_A) / 400))

# Rating update after A vs B match:
# S_A = 1 (win), 0.5 (draw), 0 (loss)
R_A_new = R_A + K (S_A − E_A)
#K = learning factor (usually 32)

Initial rating: 1000 for everyone
Difference 400 → expected ~10% wins for the weakest

Bootstrap CI for Elo (Chatbot Arena approach):
→ resample paired matches 1000 times
→ recalculate Elo for each bootstrap
→ 95% CI by percentiles

Example:
             Task1 Task2 Task3 Task4 Task5 Avg
Model A: 0.90 0.85 0.88 0.91 0.87 0.882
Model B: 0.92 0.89 0.90 0.93 0.45 0.818

On average, A is better. But what if Task5 is your task?

See also:
  → minimum tasks (worst-case quality)
  → variance by task (how smooth)
  → profile by task (not only the unit)

Weighted average:
  score = Σᵢ wᵢ score_i
  → weigh the tasks in your use-case by frequency
  → MMLU takes 57 tasks → outweighs others

①Data Contamination:
  The test set accidentally ended up in the pretraining data of the model
  → the model “saw the answers” in the exam
  Symptom: PPL on the test set is abnormally low
  Sign: the model produces accurate verbatim answers
  Protection: use closed/fresh test sets
  Protection: membership inference test (Min-K% Prob)

② Benchmark Overfitting:
  The model was additionally trained on tasks similar to the benchmark
  → MMLU is growing, real tasks are not
  Symptom: gap between benchmark and live quality
  Protection: several different benchmarks
  Defense: human eval as ground truth

③ Goodhart's Law:
  "When a measure becomes a target, it ceases to be a good measure"
  You optimize ROUGE → the model generates n-grams from reference
  You optimize the LLM-judge score → the model learns to “like” judge
  Protection: regularly change metrics and test sets
  Defense: human evaluation as the final arbiter

Positional bias:
  judge([A, B]) → A wins 65% of the time
  judge([B, A]) → B wins 62% of the time
  → the first one wins both times!
  Defense: Always run in both orders
  Result: win only if you win in BOTH runs
  or take the average if there is a draw

Verbosity bias:
  judge prefers a longer answer
  even if it's less accurate
  Defense: explicitly in the judge prompt: “length is not a criterion”
  Defense: normalize the score by the length of the answer

Self-enhancement bias:
  Claude judge overestimates Claude answers
  GPT-4 judge overestimates GPT-4 responses
  Protection: use judge from another provider

Antipattern:
  System A: Prompt v7 (carefully written)
  System B: prompt v1 (draft)
  → you compare the quality of prompting, not systems

That's right - fix the prompt template:
  one template for both systems
  vary ONLY the parameter being studied

Or - multi-prompt evaluation:
  k=5 prompt options for each system
  estimate = mean ± std for all options
  → you evaluate robustness, not a point result

Sensitivity measurement:
  CV = std/mean (coefficient of variation)
  CV > 0.1 → system is unstable to formulation

BEFORE THE EXPERIMENT:
  □ The hypothesis is formulated explicitly (H₀ and H₁)
  □ The metric is selected in advance and corresponds to the task
  □ n calculated using power analysis (power ≥ 0.8)
  □ The test set is representative of prod traffic
  □ The test set does not overlap with dev/train

LAUNCH:
  □ Both systems were run using the SAME examples
  □ For T > 0: each example is run k ≥ 3 times
  □ Judge runs in both orders (A,B) and (B,A)
  □Seed fixed (for reproducibility)
  □ Everything is logged: prompts, temperatures, model versions

ANALYSIS:
  □ CI calculated (bootstrap if the metric is not normal)
  □ Correct test selected (paired vs independent)
  □ If > 1 comparison – Bonferroni/BH correction applied
  □ Effect size calculated (Cohen's d or relative lift)
  □ The consistency of judge (κ or ρ with human) is checked

REPORT:
  □ n is specified explicitly
  □ CI is shown next to the mean: 0.815 ± 0.021 (95% CI)
  □ p-value AND effect size (not just one of the two)
  □ The source of variance is described (temperature? test-set sampling?)
  □ Versions of models and prompts are indicated