{ "type": "new-runtime-knowledge-guide", "version": 1, "generated_at": "2026-07-23", "volume": 8, "slug": "llm-stats-reference", "title": "Statistics for Evaluating LLM Systems", "description": "Basic concepts of statistics as applied to LLM experiments: quality metrics, variance and confidence intervals, hypothesis testing, A/B tests, LLM-as-judge, evaluation pitfalls and a practical checklist for a valid experiment.", "track": { "slug": "ml-and-decision-systems", "title": "ML & Decision Systems", "route": "https://newruntime.com/learn/tracks/ml-and-decision-systems/", "position": 1 }, "counts": { "sections": 7, "cards": 24 }, "routes": { "html": "https://newruntime.com/learn/llm-stats-reference/", "json": "https://newruntime.com/learn/llm-stats-reference.json" }, "sections": [ { "number": "01", "title": "LLM Quality Metrics", "intro": "", "cards": [ { "number": "01", "title": "Log-Loss and Perplexity: one essence, two names", "tag": "Metrics", "description": "Log-loss (cross-entropy) measures how confidently the model predicts the correct token. Perplexity is simply an exponent of log-loss: “how many variants the model loses on average.” PPL=10 means the model “oscillates between 10 equally probable tokens” at each step.", "blocks": [ { "kind": "code", "label": "Diagram", "language": "tex", "value": "For one token y with predicted probability p(y):\n\n log-loss = −log p(y)\n ↑ if the model is confident: p(y)=0.9 → loss = 0.105\n ↑ if you are not sure: p(y)=0.1 → loss = 2.303\n\nFor the entire sequence of N tokens:\n\n Log-Loss = −(1/N) · Σᵢ log p(yᵢ | y₁…y_{i-1})\n ↑ average for all tokens\n\n Perplexity = exp(Log-Loss) = exp(−(1/N) · Σ log p(yᵢ))\n ↑ interpretation: average “selection size” of the model\n\nExamples of PPL:\n PPL = 1000 ← bad model or out-of-domain text\n PPL = 20 ← GPT-2 on WikiText-103\n PPL = 6 ← Llama-3 70B in English text\n PPL = 1 ← ideal model (theoretically)\n\nCommunication with information bits:\n bits-per-char = log-loss / log(2) ← alternative notation" }, { "kind": "list", "label": "when to use", "items": [ "comparison of pretrain quality of models", "assessment of quality degradation during quantization", "PPL on the domain ≠ PPL on your task", "does not reflect Useful framing for downstream tasks" ] } ] }, { "number": "02", "title": "Accuracy, Precision, Recall, F1", "tag": "Metrics", "description": "For classification tasks (IntentRouter, sentiment, RAG relevance), accuracy often deceives when classes are imbalanced. F1 - harmonic mean Precision and Recall - fairer for rare classes.", "blocks": [ { "kind": "code", "label": "formulas and error matrix", "language": "tex", "value": "Predicted + Predicted −\nActual + TP FN\nActual − FP TN\n\nAccuracy = (TP+TN) / (TP+TN+FP+FN)\n→ cheats: 99% TN → accuracy=99% effortless\n\nPrecision = TP / (TP+FP) ← of all the “+”, how many “+” are true?\nRecall = TP / (TP+FN) ← of all the real “+”s, how many did you find?\n\nF1 = 2 · (P R) / (P+R) ← harmonic mean\n\nMacro F1: average F1 for all classes (imbalance not taken into account)\nWeighted F1: class frequency weighted" } ] }, { "number": "03", "title": "BLEU, ROUGE and their limitations", "tag": "Metrics", "description": "BLEU and ROUGE measure the n-gram intersection between a generation and a reference. Historically the standard for translation and summarization, but poorly correlated with human assessment of the quality of free text.", "blocks": [ { "kind": "code", "label": "mechanics", "language": "text", "value": "BLEU (precision-oriented):\n BLEU-N = intersection of N-grams(gen, ref) / len(gen)\n ↑ penalizes for output that is too short (BP)\n ↑ usually average BLEU-1..4\n\nROUGE-N (recall-oriented):\n ROUGE-N = intersection of N-grams(gen, ref) / len(ref)\n\nROUGE-L: longest common subsequence\n\nExample:\n ref: \"the cat was sitting on the rug\"\n gen: \"the rug is occupied by a cat\"\n BLEU-1 = 0.5 (2 words out of 3 are in ref)\n but the meaning is correct → BLEU does not see this\n\nBERTScore: cosine proximity of embeddings\n ↑ better for semantic quality" }, { "kind": "list", "label": "modern practice", "items": [ "BLEU/ROUGE - only as baseline", "BERTScore - if you need automation", "LLM-as-judge - for free generation" ] } ] }, { "number": "04", "title": "Metrics for RAGs and agents", "tag": "Metrics", "description": "The RAG pipeline consists of two stages - retrieval and generation - and both need to be measured separately. The final quality of the answer depends on both, but different types of errors require different corrections.", "blocks": [ { "kind": "code", "label": "RAGAS and components", "language": "sql", "value": "Retrieval:\n Hit Rate@k - relevant doc in the top-k?\n MRR@k - position of the first relevant\n Precision@k - share of relevant ones in the top k\n\nGeneration (RAGAS):\n Faithfulness - Is the answer based on context?\n Answer Relevancy - does the answer answer the question?\n Context Precision - does the context contain the answer?\n Context Recall - has all the required context been found?\n\nFaithfulness Formula (LLM-judge):\n score = (#statements from context) / (#all statements)\n → catches hallucinations" } ] } ] }, { "number": "02", "title": "Scatter, Variance and Confidence Intervals", "intro": "", "cards": [ { "number": "05", "title": "Mean, Variance, Standard Deviation", "tag": "Scatter", "description": "One assessment of the LLM system does not mean anything. What matters is the spread—how unstable the results are. The variance σ² and standard deviation σ measure this spread around the mean.", "blocks": [ { "kind": "code", "label": "formulas", "language": "tex", "value": "# n runs of the experiment, xᵢ is the result of the i-th\n\nSample mean:\n x̄ = (1/n) · Σᵢ xᵢ\n\nSample variance (unbiased):\n s² = 1/(n−1) Σᵢ (xᵢ − x̄)²\n ↑ divide by n-1, not n (Bessel correction)\n\nStandard Deviation:\n s = √s² ← same units as xᵢ\n\nExample: accuracy on 10 test sets:\n [0.82, 0.79, 0.84, 0.81, 0.83,\n 0.80, 0.85, 0.78, 0.82, 0.81]\n x = 0.815\n s = 0.021 ← spread ≈ 2%" }, { "kind": "list", "label": "interpretation", "items": [ "s small → results are stable", "s large → more runs needed", "one point instead of distribution = error" ] } ] }, { "number": "06", "title": "Standard Error and Confidence Interval", "tag": "Scatter", "description": "The average is also a random variable. Standard error (SE) measures how closely the estimated mean reflects the “true.” A confidence interval (CI) gives a range within which there is a 95% probability that the true value of a metric lies.", "blocks": [ { "kind": "code", "label": "formulas", "language": "tex", "value": "Standard error of the mean:\n SE = s/√n\n ↑ decreases with increasing sample\n ↑ for n=100: SE is 10 times less than n=1\n\n95% Confidence Interval (normal):\n CI = x̄ ± 1.96 SE = x̄ ± 1.96 s/√n\n\nExample:\n accuracy x̄ = 0.815, s = 0.021, n = 100\n SE = 0.021 / √100 = 0.0021\n 95% CI = [0.811, 0.819]\n\n n = 10 → SE = 0.021/√10 = 0.0066\n 95% CI = [0.802, 0.828] ← wide interval!\n\nConclusion: n=10 is a bad score. Need ≥ 50-100." } ] }, { "number": "07", "title": "Bootstrap: CI without distribution assumptions", "tag": "Scatter", "description": "The formula 1.96·SE works under normal distribution. LLM metrics (F1, BLEU, win-rate) are often not normal. Bootstrap is a universal method: we sample with return and build an empirical distribution.", "blocks": [ { "kind": "code", "label": "bootstrap CI algorithm", "language": "sql", "value": "results = [0.82, 0.79, 0.84, 0.81, …] # n=100\n\nbootstrap_means = []\nfor _ in range(10_000):\n sample = resample(results, n=100) # return\n bootstrap_means.append(mean(sample))\n\n#95% CI = 2.5% and 97.5% percentiles\nci_low = percentile(bootstrap_means, 2.5)\nci_high = percentile(bootstrap_means, 97.5)\n\nprint(f\"95% CI: [{ci_low:.3f}, {ci_high:.3f}]\")\n\nfrom scipy.stats import bootstrap\nres = bootstrap((data,), np.mean, n_resamples=10000)" }, { "kind": "list", "label": "when to use", "items": [ "the metric is not normally distributed", "small sample (n < 30)", "complex metrics (BLEU, F1, win-rate)", "gold standard for LLM experiments" ] } ] }, { "number": "08", "title": "Sources of variance in the LLM experiment", "tag": "Scatter", "description": "LLM adds its own layer of randomness on top of the normal sample variance. Before calculating the metric, you need to know what noise sources there are and which one dominates.", "blocks": [ { "kind": "code", "label": "Diagram", "language": "text", "value": "Sources of variance:\n\n① Temperature/sampling\n same prompt → different answers\n → run each prompt k≥3 times, take the average\n\n② Prompt sensitivity\n \"Answer briefly\" vs \"Give a short answer\"\n → the difference can be ±5–10% on the metric\n\n③ Test set sampling\n 100 questions → another 100 questions → another result\n → bootstrap based on test set examples\n\n④ Dispersion of the judge model\n LLM-as-judge is itself non-deterministic\n → run judge k≥3, consider inter-rater agreement\n\n⑤ Order of few-shot examples\n mix → different result\n → average over several orders" } ] } ] }, { "number": "03", "title": "Testing Statistical Hypotheses", "intro": "", "cards": [ { "number": "09", "title": "Null hypothesis, p-value and how not to read them", "tag": "Hypotheses", "description": "Hypothesis testing is a formal procedure for deciding whether the improvement is real or random. The p-value is the most commonly misinterpreted value in science. Let's figure out exactly what it means and what it doesn't.", "blocks": [ { "kind": "code", "label": "Diagram", "language": "text", "value": "Staging:\n H₀ (null): \"prompt A and prompt B give the same result\"\n H₁ (alternative): \"prompt B is better than prompt A\"\n\np-value = P(get this or more extreme difference | H₀ is correct)\n\nExample:\n Prompt A: accuracy = 0.780 on 200 examples\n Prompt B: accuracy = 0.810 on 200 examples\n p-value = 0.032\n\nCorrect interpretation:\n “If H₀ were true, the probability of seeing a difference ≥ 0.03 by chance = 3.2%”\n At threshold α=0.05 → we reject H₀, the difference is statistically significant\n\nWRONG interpretation (common mistakes):\n ✗ “probability that H₀ is correct = 3.2%” ← no\n ✗ “probability that the result is random = 3.2%” ← no\n ✗ “the effect is large and important” ← p does not indicate the size of the effect\n ✗ “p=0.049 and p=0.051 are fundamentally different” ← no, this is a continuous scale" }, { "kind": "list", "label": "typical thresholds α", "items": [ "p < 0.05 - standard (5% probability of false alarm)", "p < 0.01 - strict", "p < 0.1 - soft (research)", "p by itself without effect size is an incomplete picture" ] } ] }, { "number": "10", "title": "t-test: comparison of two systems", "tag": "Hypotheses", "description": "The t-test tests whether the observed difference in means could have occurred by chance? Paired t-test - for the same sets of questions (A and B were assessed on the same data). Independent - for different sets.", "blocks": [ { "kind": "code", "label": "paired t-test (recommended for LLM)", "language": "sql", "value": "# Both prompts were run on the same 100 questions\nscores_A = [0.8, 0.6, 1.0, …] # 100 values\nscores_B = [0.9, 0.7, 1.0, …] # 100 values\n\ndiff = scores_B - scores_A # pairwise differences\n# H₀: mean(diff) = 0\n\nt = mean(diff) / (std(diff) / √n)\n# compare with t-distribution (n-1 degrees of freedom)\n\nfrom scipy.stats import ttest_rel\nt_stat, p_value = ttest_rel(scores_B, scores_A)\n\nWhy paired is better than independent:\n removes variance from question difficulty\n → one complex issue affects both systems\n → difference is cleaner" } ] }, { "number": "11", "title": "Wilcoxon: nonparametrics for scores 1–5", "tag": "Hypotheses", "description": "When an LLM-judge or person gives marks 1–5, the data is ordinal, not interval. The t-test assumes normality - this is violated for scores. The Wilcoxon test is a nonparametric analogue of the paired t-test and makes no assumptions about distribution.", "blocks": [ { "kind": "code", "label": "algorithm and application", "language": "sql", "value": "# Ratings of two systems (1–5) using 50 examples:\ngrades_A = [3, 4, 3, 5, 2, …]\ngrades_B = [4, 4, 4, 5, 3, …]\n\nfrom scipy.stats import wilcoxon\nstat, p_value = wilcoxon(grades_B, grades_A,\n alternative='greater')\n\nWorks through difference ranks:\n diff_i = B_i - A_i\n → rank |diff_i|\n → compare the sums of ranks “+” and “−”\n\nWhen to use:\n ✓ ratings 1-5, 1-10 (LLM-judge, human eval)\n ✓ ordinal metrics (pass@k ranks)\n ✓ small samples (n < 30)" } ] }, { "number": "12", "title": "Multiple comparisons and Bonferroni correction", "tag": "Hypotheses", "description": "If you compare 10 prompts in pairs, that’s 45 tests. With α=0.05 and 45 tests, ~2 false significant results are expected simply by chance. The Bonferroni correction adjusts the significance threshold.", "blocks": [ { "kind": "code", "label": "problem and solution", "language": "sql", "value": "Family-wise error rate (FWER):\n P(at least 1 false alarm from m tests)\n = 1 − (1−α)^m\n\n m=45, α=0.05 → 1−(0.95)^45 = 0.90 = 90%!\n\nBonferroni correction:\n α* = α / m = 0.05 / 45 = 0.0011\n → use this threshold for each test\n\nBenjamini-Hochberg correction (FDR):\n less strict, controls the rate of false discoveries\n → recommended for large numbers of comparisons\n\nfrom statsmodels.stats.multitest import multipletests\nreject, p_adj, _, _ = multipletests(\n p_values, method='bonferroni' # or 'fdr_bh'\n)" }, { "kind": "list", "label": "rule", "items": [ "compared 10 prompts, took the best one without correction → error", "Bonferroni - strictly, few comparisons (<20)", "BH / FDR - flexible, many comparisons" ] } ] } ] }, { "number": "04", "title": "A/B Test of LLM systems", "intro": "", "cards": [ { "number": "13", "title": "Effect Size: the difference is significant - but is it big?", "tag": "A/B test", "description": "p-value says “whether the difference is due to chance”, but nothing about its magnitude. With large n, even a difference of 0.1% will be statistically significant. Effect size is a separate value that answers the question “how big is the effect?”", "blocks": [ { "kind": "code", "label": "Diagram", "language": "tex", "value": "Cohen's d - standardized mean difference:\n\n d = (μ_B − μ_A) / s_pooled\n s_pooled = √( (s_A² + s_B²) / 2 )\n\nCohen's d interpretation:\n d = 0.2 → small effect (almost imperceptible)\n d = 0.5 → average effect (noticeable in use)\n d = 0.8 → large effect (obviously better)\n\nExample:\n Prompt A: x̄=0.780, s=0.12\n Prompt B: x̄=0.810, s=0.11\n s_pooled = √((0.0144+0.0121)/2) = 0.115\n d = (0.810−0.780) / 0.115 = 0.26 ← small effect\n\n n=200 → p=0.031 → statistically significant\n d=0.26 → but practically a small difference\n\nConclusion: always report BOTH p-value AND effect size." }, { "kind": "list", "label": "practice", "items": [ "d > 0.5 → worth implementing", "d = 0.2–0.5 → depends on the cost of change", "d < 0.2 → most likely not worth it", "for win-rate: OR / relative lift as effect size" ] } ] }, { "number": "14", "title": "Power Analysis: how many examples do you need?", "tag": "A/B test", "description": "Power (test power) - the probability of detecting a real effect if there is one. Power = 1 − β, where β is the probability of missing an improvement (type II error). Standard: power ≥ 0.80. Power analysis answers the question “how many examples are needed” before the experiment?", "blocks": [ { "kind": "code", "label": "formula and example", "language": "sql", "value": "Four related parameters (know 3 → find 4th):\n α (significance level, usually 0.05)\n power (1−β, typically 0.80)\n d (expected effect size)\n n (← what we are looking for)\n\nMinimum n for paired t-test:\nfrom statsmodels.stats.power import TTestPower\nanalysis = TTestPower()\nn = analysis.solve_power(\n effect_size=0.3, # expected Cohen's d\n alpha=0.05,\n power=0.80\n)\n# → n ≈ 90 examples per group\n\nEmpirical guidelines for LLM:\n d=0.5 → n ≥ 34 (large expected effect)\n d=0.3 → n ≥ 90\n d=0.2 → n ≥ 200 (small effect - you need a lot)" } ] }, { "number": "15", "title": "Correct setup of an A/B experiment", "tag": "A/B test", "description": "Most errors in LLM experiments are not mathematical, but design ones: the test set is chosen incorrectly, systems are compared on different data, or the metric does not correspond to the task.", "blocks": [ { "kind": "code", "label": "design checklist", "language": "sql", "value": "① Formulate a hypothesis BEFORE the experiment\n \"Prompt B gives higher F1 on task X\"\n → not “let’s see what’s best”\n\n② Fix the metric BEFORE the run\n → you cannot select a metric after, based on the best result\n\n③ Use the SAME test set for A and B\n → paired test, removes variance from the difficulty of questions\n\n④ The test set should not overlap with the dev set\n → otherwise you optimize for a test without knowledge\n\n⑤ Run each system k≥3 times (at T>0)\n → average across runs, estimate σ as a function of temperature\n\n⑥ Calculate n using power analysis IN ADVANCE\n → don't stop when p < 0.05 “for the first time”" } ] } ] }, { "number": "05", "title": "LLM-specific assessment", "intro": "", "cards": [ { "number": "16", "title": "LLM-as-Judge: Model Evaluator Statistics", "tag": "LLM Eval", "description": "LLM-as-judge is itself a stochastic process with variance. Before trusting his assessments, you need to measure the consistency of the judge with yourself and with people. Only then can the results be interpreted as a metric.", "blocks": [ { "kind": "code", "label": "Diagram", "language": "sql", "value": "Three questions about quality judge:\n\n① Intra-rater reliability\n Run judge on some examples k=5 times\n σ_judge = std(scores per example)\n → if σ_judge > 0.5 points out of 5 → judge is unstable\n → average 3–5 runs before the final score\n\n② Human correlation\n Take 50–100 examples from human labels\n Spearman ρ = rank correlation(judge_score, human_score)\n ρ > 0.7 → judge is quite reliable\n ρ < 0.5 → judge is not suitable for this task\n\n③ Consistency between judge models\n Claude judge vs GPT-4 judge → correlation?\n Cohen's κ for binary solutions (pass/fail)\n κ > 0.6 → good agreement\n\nInter-rater reliability (κ for two evaluators):\n κ = (P_observed − P_chance) / (1 − P_chance)\n κ = 0.2–0.4 → weak agreement\n κ = 0.4–0.6 → moderate\n κ = 0.6–0.8 → good → can be trusted" }, { "kind": "list", "label": "practice", "items": [ "validate judge on ~100 human-labeled examples", "average 3+ runs of judge at T>0", "set judge T=0 for reproducibility", "one run judge = not a metric, but a random point" ] } ] }, { "number": "17", "title": "Win-Rate and its statistics", "tag": "LLM Eval", "description": "Win-rate (the percentage of victories of A over B in pairwise comparisons) is an intuitive metric. But how do you know if the difference 55% vs 45% is significant? This is a problem for a fraction test - the binomial test or chi-square.", "blocks": [ { "kind": "code", "label": "win-rate significance test", "language": "sql", "value": "# 100 comparisons A vs B\nwins_B = 58 # B won 58 times\nn = 100 # total comparisons\n# H₀: p(B wins) = 0.5 (equal systems)\n\nfrom scipy.stats import binomtest\nresult = binomtest(wins_B, n, p=0.5,\n alternative='greater')\nprint(result.pvalue) # → 0.044 → significant!\n\n# CI for win-rate (Wilson interval):\nfrom statsmodels.stats.proportion import proportion_confint\nci = proportion_confint(wins_B, n, method='wilson')\n# → (0.483, 0.671)\n\nMinimum significant difference (α=0.05):\n n=50 → win-rate > 64% significant\n n=100 → win-rate > 59% significant\n n=200 → win-rate > 56% significant" } ] }, { "number": "18", "title": "Elo Rating for models", "tag": "LLM Eval", "description": "When comparing more than two systems, pairwise win-rates are not transitive. Elo-rating (from chess) solves this problem: each win/loss updates the rating taking into account the expected result. Used in Chatbot Arena (LMSYS).", "blocks": [ { "kind": "code", "label": "Elo mechanics", "language": "text", "value": "# Expected probability of A winning over B:\nE_A = 1 / (1 + 10^((R_B − R_A) / 400))\n\n# Rating update after A vs B match:\n# S_A = 1 (win), 0.5 (draw), 0 (loss)\nR_A_new = R_A + K (S_A − E_A)\n#K = learning factor (usually 32)\n\nInitial rating: 1000 for everyone\nDifference 400 → expected ~10% wins for the weakest\n\nBootstrap CI for Elo (Chatbot Arena approach):\n→ resample paired matches 1000 times\n→ recalculate Elo for each bootstrap\n→ 95% CI by percentiles" } ] }, { "number": "19", "title": "Aggregate Score: when the average deceives", "tag": "LLM Eval", "description": "An average score across multiple tasks/benchmarks masks uneven quality. A model can perform well on 9 out of 10 problems and fail on the 10th - the average score still looks good.", "blocks": [ { "kind": "code", "label": "Diagram", "language": "tex", "value": "Example:\n Task1 Task2 Task3 Task4 Task5 Avg\nModel A: 0.90 0.85 0.88 0.91 0.87 0.882\nModel B: 0.92 0.89 0.90 0.93 0.45 0.818\n\nOn average, A is better. But what if Task5 is your task?\n\nSee also:\n → minimum tasks (worst-case quality)\n → variance by task (how smooth)\n → profile by task (not only the unit)\n\nWeighted average:\n score = Σᵢ wᵢ score_i\n → weigh the tasks in your use-case by frequency\n → MMLU takes 57 tasks → outweighs others" } ] } ] }, { "number": "06", "title": "Pitfalls in Evaluating LLM Systems", "intro": "", "cards": [ { "number": "20", "title": "Data Contamination, Benchmark Overfitting and Goodhart's Law", "tag": "Traps", "description": "The three most dangerous pitfalls when evaluating LLM: test data leaked into training, the metric no longer reflects real quality, or the model is “tailored” for a benchmark. All three make the comparison meaningless.", "blocks": [ { "kind": "code", "label": "Diagram", "language": "sql", "value": "①Data Contamination:\n The test set accidentally ended up in the pretraining data of the model\n → the model “saw the answers” in the exam\n Symptom: PPL on the test set is abnormally low\n Sign: the model produces accurate verbatim answers\n Protection: use closed/fresh test sets\n Protection: membership inference test (Min-K% Prob)\n\n② Benchmark Overfitting:\n The model was additionally trained on tasks similar to the benchmark\n → MMLU is growing, real tasks are not\n Symptom: gap between benchmark and live quality\n Protection: several different benchmarks\n Defense: human eval as ground truth\n\n③ Goodhart's Law:\n \"When a measure becomes a target, it ceases to be a good measure\"\n You optimize ROUGE → the model generates n-grams from reference\n You optimize the LLM-judge score → the model learns to “like” judge\n Protection: regularly change metrics and test sets\n Defense: human evaluation as the final arbiter" } ] }, { "number": "21", "title": "Judge's biases: attitude and verbosity", "tag": "Traps", "description": "LLM-judge systematically distorts scores in favor of the first or second answer (positional bias) and in favor of longer answers (verbosity bias) - regardless of the quality of the content.", "blocks": [ { "kind": "code", "label": "types of displacements and protection", "language": "sql", "value": "Positional bias:\n judge([A, B]) → A wins 65% of the time\n judge([B, A]) → B wins 62% of the time\n → the first one wins both times!\n Defense: Always run in both orders\n Result: win only if you win in BOTH runs\n or take the average if there is a draw\n\nVerbosity bias:\n judge prefers a longer answer\n even if it's less accurate\n Defense: explicitly in the judge prompt: “length is not a criterion”\n Defense: normalize the score by the length of the answer\n\nSelf-enhancement bias:\n Claude judge overestimates Claude answers\n GPT-4 judge overestimates GPT-4 responses\n Protection: use judge from another provider" } ] }, { "number": "22", "title": "Prompt Sensitivity: assessment instability", "tag": "Traps", "description": "A difference in the formulation of a prompt by ±10% of the metric is a common phenomenon. This means that “comparing systems” without prompt control is actually “comparing prompts.” The result depends on who wrote the better prompt for their system.", "blocks": [ { "kind": "code", "label": "how to control", "language": "text", "value": "Antipattern:\n System A: Prompt v7 (carefully written)\n System B: prompt v1 (draft)\n → you compare the quality of prompting, not systems\n\nThat's right - fix the prompt template:\n one template for both systems\n vary ONLY the parameter being studied\n\nOr - multi-prompt evaluation:\n k=5 prompt options for each system\n estimate = mean ± std for all options\n → you evaluate robustness, not a point result\n\nSensitivity measurement:\n CV = std/mean (coefficient of variation)\n CV > 0.1 → system is unstable to formulation" } ] } ] }, { "number": "07", "title": "Practice: Checklist and Rules", "intro": "", "cards": [ { "number": "23", "title": "Checklist for a statistically valid LLM experiment", "tag": "Practice", "description": "Go through this list before and after launch. If at least one point is missed, the conclusions are questionable.", "blocks": [ { "kind": "code", "label": "Diagram", "language": "text", "value": "BEFORE THE EXPERIMENT:\n □ The hypothesis is formulated explicitly (H₀ and H₁)\n □ The metric is selected in advance and corresponds to the task\n □ n calculated using power analysis (power ≥ 0.8)\n □ The test set is representative of prod traffic\n □ The test set does not overlap with dev/train\n\nLAUNCH:\n □ Both systems were run using the SAME examples\n □ For T > 0: each example is run k ≥ 3 times\n □ Judge runs in both orders (A,B) and (B,A)\n □Seed fixed (for reproducibility)\n □ Everything is logged: prompts, temperatures, model versions\n\nANALYSIS:\n □ CI calculated (bootstrap if the metric is not normal)\n □ Correct test selected (paired vs independent)\n □ If > 1 comparison – Bonferroni/BH correction applied\n □ Effect size calculated (Cohen's d or relative lift)\n □ The consistency of judge (κ or ρ with human) is checked\n\nREPORT:\n □ n is specified explicitly\n □ CI is shown next to the mean: 0.815 ± 0.021 (95% CI)\n □ p-value AND effect size (not just one of the two)\n □ The source of variance is described (temperature? test-set sampling?)\n □ Versions of models and prompts are indicated" } ] }, { "number": "24", "title": "Quick Reference: Method Selection", "tag": "Practice", "description": "Quick cheat sheet: which statistical method to use in typical LLM scenarios.", "blocks": [ { "kind": "table", "headers": [ "Task", "Method", "Python" ], "rows": [ [ "Compare 2 prompts", "paired t-test", "ttest_rel(a, b)" ], [ "Ratings 1–5 from judge", "Wilcoxon", "wilcoxon(a, b)" ], [ "Win-rate significance", "Binomial test", "binomtest(w, n)" ], [ "CI for any metric", "Bootstrap", "bootstrap((x,), np.mean)" ], [ "Compare N systems", "Elo + bootstrap CI", "elo_mle() + resample" ], [ "N prompts → choosing the best", "+ Bonferroni", "multipletests(p_vals)" ], [ "Judge vs human agreement", "Spearman ρ / Cohen κ", "spearmanr / cohen_kappa" ], [ "How many examples do you need?", "Power analysis", "TTestPower().solve_power" ] ] }, { "kind": "list", "label": "minimum stack", "items": [ "scipy.stats", "statsmodels", "numpy (bootstrap)", "ragas (for RAG metrics)" ] } ] } ] } ] }