--- type: "new-runtime-knowledge-guide" stable_id: "knowledge_guide:llm-judge-meta-evaluation-reference" version: 1 generated_at: "2026-07-23" record_date: "2026-07-23" date_kind: "generated_at" slug: "llm-judge-meta-evaluation-reference" title: "LLM-as-Judge Meta-Evaluation" description: "A separate volume is not about eval in general, but about the validation of the judge itself: agreement with people, metrics for binary and ordinal verdicts, resistance to repeated runs, confidence calibration, bias audits, pairwise ranking and minimum reporting standard. Agreement → classifier view → stability → calibration → bias → reporting." retrieval_nugget: "A separate volume is not about eval in general, but about the validation of the judge itself: agreement with people, metrics for binary and ordinal verdicts, resistance to repeated runs, confidence calibration, bias audits, pairwise ranking and minimum reporting standard. Agreement → classifier view → stability → calibration → bias → reporting." track: "evaluation-and-improvement" volume: 32 --- # LLM-as-Judge Meta-Evaluation ## Retrieval answer A separate volume is not about eval in general, but about the validation of the judge itself: agreement with people, metrics for binary and ordinal verdicts, resistance to repeated runs, confidence calibration, bias audits, pairwise ranking and minimum reporting standard. Agreement → classifier view → stability → calibration → bias → reporting. ## 01. Frame: Judge should also be evaluated. ### 01.01 Judge is not ground truth, but a separate model with its own errors. LLM-as-judge is convenient to use as a quick proxy for human eval, but that doesn’t make it a true metric. If you start to optimize the system for the judge without checking the judge itself, the model quickly learns to “like the appraiser” rather than really improve. #### Diagram ```text Five Questions to Any Judge 1. Does it match people above chance? 2. Does it give the same verdict on repeated launches? 3. Can you express uncertainty, not just a hard label? 4. Does the verdict depend on the order, length and family of the model? 5. Does the final score have a confidence interval? If the answer to at least one question is no. Judge score = heuristic, not reliable KPI If all five are passed Judge score can be used for regression tracking. A/B comparisons and semi-automatic eval loops ``` #### when it is particularly important - leaderboard - production eval loop - reward shaping for the judge score - consider the single judge score to be true ### 01.02 Minimum meta-eval protocol for new judge Before you connect the judge to the pipeline, you need to go through a short, but mandatory validation loop. The idea is to first understand the ceiling of consent between people, then compare the judge not with one annotator, but with consensus or distribution of human labels. #### minimum ```text 1. 100-200 examples: good/bad/borderline 2. Mark 2-3 people in the same category 3. The Inter-Human Agreement 4. Run the judge on the same set 5. Agreement Judge vs Human 6. Check stability, calibration and bias probes If inter-human κ = 0.45 and judge-human κ = 0.43 The judge is close to the human ceiling. If inter-human κ = 0.75 and judge-human κ = 0.38 Judge is weak, the problem is not the task, but the judge ``` #### practice - compared to the human ceiling - borderline cases - Validate the judge on too easy examples ## 02. Agreement: How much does the judge match people ### 02.03 Cohen's kappa: Basic metric for binary pass/fail If the judge's verdict is binary, 'Cohen's κ' is almost always better than accuracy: it takes into account how much of the coincidences may have come about by chance. This is a good default for tasks like grounded/not grounded, safe/unsafe, pass/fail. #### Diagram ```sql You need two arrays of the same length: human = [1, 0, 1, 1, 0, ...] # consensus or expert label judge = [1, 1, 1, 0, 0, ...] # verdict judge in the same examples First, consider the confusion matrix: TP = human=1 and judge=1 TN = human=0 and judge=0 FP = human=0, judge=1 FN = human=1, judge=0 N = TP + TN + FP + FN Observed agreement: P_observed = (TP + TN) / N Chance agreement from the margins: P_chance = ((TP+FP)/N)·((TP+FN)/N) + ((TN+FN)/N)·((TN+FP)/N) κ = (P_observed − P_chance) / (1 − P_chance) Example: TP=35, TN=51, FP=9, FN=5, N=100 P_observed = (35+51)/100 = 0.86 P_chance = 0.44·0.40 + 0.56·0.60 = 0.512 κ = (0.86−0.512)/(1−0.512) = 0.713 Rough interpretation: κ < 0.40 → judge weak 0.40-0.60 Conditionally acceptable 0.60-0.80 Good working level > 0.80 Very strong agreement But: With a strong class imbalance, κ may look severe. However, raw accuracy seems to be high. Meaning: accuracy = "how often did you match" κ = "how often coincided beyond chance" ``` #### how to count ```sql from sklearn.metrics import cohen_kappa_score human = [1, 0, 1, ...] judge = [1, 1, 1, ...] kappa = cohen_kappa_score(human, judge) #human labels: from 2-3 experts or majority vote # judge labels: from the same eval set, in the same order ``` #### when to use - binary rubric - judge vs consensus label - Look at the confusion matrix - Does not replace calibration and bias probes ### 02.04 Weighted kappa: If the score is on a scale of 1-5 For ordinal labels, the usual 'κ' is too rough: the error '5→4' and the error '5→1' are considered the same. 'Weighted κ' gives a partial penalty and usually better reflects the real quality of the judge on scales. #### formula ```sql We need two ordinal ratings: human = [5, 3, 4, ...] judge = [4, 3, 5, ...] We construct observed matrix O(i,j) and expected matrix E(i,j): O = pair frequencies (human=i, judge=j) E = expected frequencies from marginals κ_w = 1 − (ΣᵢΣⱼ wᵢⱼ Oᵢⱼ) / (ΣᵢΣⱼ wᵢⱼ Eᵢⱼ) Linear weights: wᵢⱼ = |i−j| / (K−1) Quadratic weights: wᵢⱼ = (i−j)² / (K−1)² Where to get the values: K = number of scale levels O and E = from confusion matrix by ratings ``` #### use ```sql from sklearn.metrics import cohen_kappa_score k_linear = cohen_kappa_score(human, judge, weights="linear") k_quad = cohen_kappa_score(human, judge, weights="quadratic") Practically: if the adjacent levels are almost uniform quadratic if long-range misses are particularly painful ``` #### when appropriate - 1-4, 1-5, 1-10 - Only with clear anchor examples - Not to use as a substitute for binary rubric ### 02.05 Fleiss' kappa: if there are more than two people or judges Once in the 3+ evaluator problem, 'Cohen's κ' no longer fits. 'Fleiss' κ' gives a chance-corrected agreement at the level of the entire annotator pool and shows well whether there is consensus at all. #### formula ```tex Say: N = number of examples n = number of appraisers for example k = number of classes nij = how many appraisers gave example i class j Agreement within Example i: 1 Pᵢ = ------- · Σⱼ nᵢⱼ(nᵢⱼ−1) n(n−1) Average observed agreement: P̄ = (1/N) · Σᵢ Pᵢ Share of class j over the pool: 1 pⱼ = -------- · Σᵢ nᵢⱼ N · n Chance agreement: P̄ₑ = Σⱼ pⱼ² Fleiss κ = (P̄ − P̄ₑ) / (1 − P̄ₑ) ``` #### wherein ```text I need a table. rows = examples columns = raters values = class label Example: tc_001 [PASS, PASS, FAIL] tc_002 [FAIL, FAIL, FAIL] tc_003 [PASS, PASS, PASS] It is considered nij for each example. ``` #### scenario - 3+ human raters - Comparison of several judge models - I don't like messy missing labels. ### 02.06 Krippendorff's alpha: the most versatile option for messy eval 'Krippendorff's α' is useful where the actual markup is not ideal: different types of scales, omissions, not all examples have the same number of annotators. In practical eval, it is often a better long-term choice than a zoo of different κ variants. #### formula ```sql α = 1 − D_observed / D_expected Where: D observed = actual disagreement between annotators D expected = Disagreement expected by chance Ordinal labels are usually defined as distance: δ(c, c′) = (c − c′)² Then great discrepancies are fined more. as in quadratic weighted κ If α → 1: almost complete consensus If α → 0: no better than chance ``` #### wherein ```sql We need an item × rater matrix: [ [1, 1, None, 0], [2, 2, 2, 2], [4, 3, 4, None], ] from krippendorff import alpha alpha_value = alpha(reliability_data=data, level_of_measurement="ordinal") ``` #### when - messy annotation pipeline - partial - more difficult to explain to the team from scratch ### 02.07 Pearson r vs Spearman rho vs Kendall tau When a judge returns a score or a rank, there are three types of connection. For most LLM-eval tasks, ‘Spearman ρ’ is better than default. 'Kendall τ' is useful for the stability of order. Pearson r only makes sense if the score is close to the interval and you are interested in linear communication. #### Diagram ```tex You need two arrays of the same length: human_scores = [4, 2, 5, 3, ...] judge_scores = [5, 2, 4, 3, ...] Pearson r is a linear link: Σ (xᵢ−x̄)(yᵢ−ȳ) r = ----------------------------------- sqrt(Σ(xᵢ−x̄)² · Σ(yᵢ−ȳ)²) Spearman rho - Pearson by rank: ρ = Pearson(rank(x), rank(y)) If there are no ties, you can write like this: 6 Σ dᵢ² ρ = 1 − ---------- n(n²−1) Kendall tau is a paired agreement: C − D τ = ------- C + D C = concordant pairs D = discordant pairs ``` #### How to count and what to provide as input ```sql from scipy.stats import pearsonr, spearmanr, kendalltau pearson_r, _ = pearsonr(human_scores, judge_scores) spearman_rho, _ = spearmanr(human_scores, judge_scores) kendall_tau, _ = kendalltau(human_scores, judge_scores) Human scores: average human score or rank for each example Judge scores: judge score for the same examples For the leaderboard: human rank = rank of the system judge rank = rank of the same system ``` | Metrica | What meter | When you're good. | When dangerous | | --- | --- | --- | --- | | Pearson r | Linear communication | interval | ordinal scales 1-5 | | Spearman rho | Monotonic rank | default for judge scores | distanceless | | Kendall tau | Pairwise agreement | rank stability, leaderboard | less familiar | #### conclusion - Spearman as first choice - Kendall for System Order - Pearson only if there is a reason ## 03. Judge as a classifier ### 03.08 Matthews Correlation Coefficient: A binary metric for imbalance MCC is especially useful when positive class is rare: hallucination, unsafe answer, policy violation. Unlike accuracy and even F1, it takes into account all the confusion matrix and gives an honest picture of the quality of the judge in unbalanced tasks. #### Diagram ```tex Through TP, TN, FP, FN: TP·TN − FP·FN MCC = ----------------------------------- sqrt((TP+FP)(TP+FN)(TN+FP)(TN+FN)) Interpretation: -1: systematically wrong No better than chance. +1 - The perfect classifier Practical sense: if unsafe only 5%, 95% accuracy can be applied to any judge. MCC will immediately show the empty ``` #### wherein ```sql First you fix the positive class: 1 = unsafe, 0 = safe Then on the same eval set: y_true = human / gold labels y_pred = hard verdict judge from sklearn.metrics import matthews_corrcoef mcc = matthews_corrcoef(y_true, y_pred) If the judge gives you a probability: y_pred = 1[p_i ≥ threshold] ``` #### particularly useful - safety eval - hallucination detection - rare negative cases - Read with the Reference Dangerous Class ### 03.09 Accuracy cheats if classes are unbalanced A judge who always says ‘PASS’ can look strong on a dataset where 90% of the examples are true pass. Precision can almost never be read alone. #### formulae ```sql accuracy = (TP + TN) / N precision_pos = TP / (TP + FP) recall_pos = TP / (TP + FN) specificity = TN / (TN + FP) F1 = 2 · precision · recall / (precision + recall) balanced_accuracy = (recall_pos + specificity) / 2 Where to get the values: TP, TN, FP, FN = from judge verdict vs gold label An example of accuracy cheating: unsafe = 5%, judge always says SAFE accuracy = 95% recall_unsafe = 0% balanced_accuracy = 50% ``` #### code ```sql from sklearn.metrics import accuracy_score, precision_recall_fscore_support from sklearn.metrics import balanced_accuracy_score acc = accuracy_score(y_true, y_pred) prec, rec, f1, _ = precision_recall_fscore_support(y_true, y_pred, average="binary") bacc = balanced_accuracy_score(y_true, y_pred) ``` #### red flags - Accuracy without class balance - No breakdown by FP/FN errors - classify ### 03.10 ROC-AUC vs PR-AUC: Which Curve to Look at If the judge returns the probability, you can rate it as a ranker. “ROC-AUC” shows overall separability, but in rare positive classes, it is often too optimistic. PR-AUC better reflects how the judge actually finds rare violations. #### Diagram ```text Need: y_true = [0, 1, 0, 0, 1, ...] p_judge = [0.03, 0.91, 0.22, 0.10, 0.78, ...] For any threshold t: TPR(t) = TP / (TP + FN) FPR(t) = FP / (FP + TN) Precision(t) = TP / (TP + FP) Recall(t) = TP / (TP + FN) ROC curve: x = FPR(t), y = TPR(t) ROC-AUC = area under this curve PR curve: x = Recall(t), y = Precision(t) PR-AUC = area under this curve Meaning: ROC-AUC Ranks Positives Above Negatives How accurate is PR-AUC in a rare positive class? ``` #### reckon ```sql from sklearn.metrics import roc_auc_score, average_precision_score roc_auc = roc_auc_score(y_true, p_judge) pr_auc = average_precision_score(y_true, p_judge) Important: Probabilities / confidences are provided here. Not hard labels PASS/FAIL ``` | Metrica | What shows | Best. | | --- | --- | --- | | ROC-AUC | tradeoff TPR/FPR | class | | PR-AUC | precision/recall positive | Rare problems, safety, hallucinations | #### rule - Rarely positive to watch PR-AUC - ROC-AUC as secondary ### 03.11 Confusion Matrix Judge: What Errors Are Really Expensive Judge's not judging in a vacuum. Sometimes the false ‘PASS’ is the most dangerous, sometimes the false ‘FAIL’ turns the system into an over-refusal machine. Therefore, confusion matrix should be associated with the cost of error, not just a beautiful final metric. #### how to build it ```text If the judge gives confidence p: y_pred = 1[p ≥ t] Then you think: judge=1 judge=0 human=1 TP FN human=0 FP TN The threshold t is not chosen for beauty. a for the cost of errors FP and FN ``` #### Example of cost-aware reading ```text Faithfulness eval: FN judge = missed hallucination FP judge = punished with correct answer Moderation eval: FN judge = missed unsafe content FP judge = blocked harmless response Same accuracy. may be acceptable in one case. and unacceptable elsewhere ``` #### do - fix the target operating point - Discuss FP and FN separately - Optimize only the average metric ## 04. Stability & uncertainty ### 04.12 Intra-rater reliability: Judge must match itself Even with ‘temperature=0’ and one model, the judge may be unstable due to long reasoning paths, backend sampling, or subtle prompt shifts. If the same example receives different verdicts, the point score becomes meaningless. #### How to measure binary and score verdicts ```text For each example i run judge K times: vᵢ = [vᵢ₁, vᵢ₂, ..., vᵢₖ] If verdict binary: c pass = number of PASS c fail = number of FAIL self_agreementᵢ = max(c_pass, c_fail) / K If the verdict numeric: μᵢ = mean(vᵢ) σᵢ = std(vᵢ) MADᵢ = mean(|vᵢⱼ − μᵢ|) Outcome on dataset: mean_self_agreement = mean(self_agreementᵢ) mean_sigma = mean(σᵢ) ``` #### wherein ```text Same thing: prompt context answer rubric You run Ks with the same judge. And you only keep your verdict/confidence. dataset-free ``` #### signal - Frequent flips on borderline cases - You need abstain or uncertainty - Average a few runs only if it is conscious ### 04.13 Flip rate and self-inconsistency Convenient Instability Metric: How often a judge changes his mind when repeating the same task. In production, this is easier to communicate than abstract variance: “The judge contradicts itself by 14% of the examples.” #### formula ```tex For example i: unstablei = 1, if unique(vi1 ... vik) > 1 = 0, otherwise 1 flip_rate = ----- · Σᵢ unstableᵢ N If the verdict numeric: First, transfer it to the label. e.g. PASS if score ≥ 4 ``` #### example ```text tc_014 → PASS, PASS, FAIL, PASS, FAIL unstable_014 = 1 tc_015 → FAIL, FAIL, FAIL, FAIL, FAIL unstable_015 = 0 If 14 out of 100 examples are unstable, flip_rate = 0.14 ``` #### useful - Comparison of judge prompts - comparison - It is not a substitute for people. ### 04.14 Paired bootstrap CI: compare systems not by one digit, but by interval If two systems judged on the same set of examples, you need to compare pairs. 'Paired bootstrap' gives a confidence interval for the difference in metrics and helps distinguish real improvements from sample noise. #### Diagram ```python Algorithm for score(A) − score(B): for b in 1..10000: sample_idx = resample(example_ids, replace=True) delta_b = metric(A[sample_idx]) - metric(B[sample_idx]) CI_95 = percentile(delta, 2.5), percentile(delta, 97.5) Interpretation: if 0 within CI Improvement is unconvincing if all CI is > 0 A is statistically better than B on this dataset Keyword: paired Because both systems are evaluated on the same examples. ``` #### wherein ```text Each example requires an outcome for both systems: example_id | score_A | score_B tc_001 | 1 | 0 tc_002 | 1 | 1 tc_003 | 0 | 1 metric may be: accuracy pass_rate mean judge score win_rate It's not tokens or answers that are being reassembled. (a) Example indexes ``` #### where must-have - A/B on same eval set - leaderboard deltas - Compare point estimates without CI ### 04.15 The judge must be able to say unsure. Forced choice gives a beautiful appearance of certainty, but breaks meta-eval on really ambiguous examples. Sometimes the best judge is not the one who always answers, but the one who can send the case to the human review. #### two useful formulae ```tex Let the judge give confidence p . [0.1]. coverage(τ) = #(pᵢ ≥ τ) / N selective_risk(τ) = errors in examples with pi ≥ τ / #(pi ≥ τ) Meaning: increase the threshold τ Coverage is falling Selective risk should also fall. ``` #### three modes ```text High confidence: auto-score Medium confidence: Consider, but mark as weak evidence Low confidence / tie / unsure: send out This is especially important for rating indeterminacy. borderline examples ``` #### whenever - ambiguous rubric - pairwise ties - Downstream Unsure Processing Logic ## 05. Calibration: Can you trust the judge? and ### 05.16 Brier Score: how likely the judge is to be If a judge gives a probability of 'PASS', 'SAFE' or 'A wins', 'Brier Score' is almost always the first. This is a quadratic error of probabilistic prediction: both calibration and sharpness in one digit. #### Diagram ```text For binary event y {0.1} and prediction p: Brier = mean((p − y)^2) Example: Judge says 0.90, event happened → error 0. 01 The judge says 0.90, the event did not happen → error 0. 81 Below. If the judge has only hard labels without confidence, Brier cannot be counted without a separate confidence head. ``` #### where to get p and y ```sql For each example i: pi = confidence judge that label = 1 yi = real binary label from human/gold Example: p = [0.91, 0.72, 0.10, ...] y = [1, 0, 0, ...] from sklearn.metrics import brier_score_loss brier = brier_score_loss(y, p) If the judge gives confidence 0-100, split by 100 first ``` #### Good friend - reliability diagram - ECE - You need the correct probability output ### 05.17 Expected Calibration Error: Is the judge as confident as it is correct? ECE compares confidence to its actual success rate in the basket of probability. If the judge says ‘90% confident’ and is right only 65% of the time, that’s overconfidence. #### Diagram ```tex Divide predictions into M bins by confidence: Bₘ = { i : pᵢ ∈ ((m−1)/M, m/M] } accuracy(Bₘ) = (1 / |Bₘ|) · Σᵢ 1[ŷᵢ = yᵢ] confidence(Bₘ) = (1 / |Bₘ|) · Σᵢ pᵢ ECE = Σₘ (|Bₘ| / N) · |accuracy(Bₘ) − confidence(Bₘ)| Interpretation: ECE ≈ 0 → well calibrated ECE Upgrades the Gap Between Confidence and Reality Careful: ECE is sensitive to the binning scheme Therefore, it is useful to show both a schedule and a number. ``` #### file ```text ŷᵢ = predicted judge label Pi = confidence in this particular label yᵢ = gold / human label Usually: M = 10 bins or equal-width, or equal-mass binning ``` #### crucial - count - point out - Read more about ECE without a reliable plot ### 05.18 Reliability diagram: the most visual calibration audit One number hides the form of the error. Reliability diagram shows where the judge overestimates himself: on high confidence, on medium or only in a narrow range. This is often more useful than any summary metric. #### construct ```text 1. Break predictions by confidence bins 2. For each bin count: mean_confidence empirical_accuracy 3. Draw dots: x = mean_confidence y = empirical_accuracy 4. Add diagonal y=x as ideal ``` #### how to read ```text Perfect line: confidence = empirical accuracy Curve below diagonal: judge overconfident The curve above the diagonal: judge underconfident Separately useful to watch: coverage by bins borderline subset hard cases only ``` #### useful - threshold tuning - abstain / human review policy - show the team without a statistical background ## 06. Bias audits: where the judge is systematically distorted ### 06.19 Position bias: first or second response gets a head start In a pairwise score, the judge may prefer the first or second answer simply because of the position. If you do not do A/B swap, you can get a false leaderboard even with a strong judge model. #### reckon ```text judge([A, B]) → verdict₁ judge([B, A]) → verdict₂ winrate(A first) = wins_A_when_first / pairs_with_A_first winrate(A second) = wins_A_when_second / pairs_with_A_second position_gap = |winrate(A first) − winrate(A second)| swap_robust_winrate(A) = 0.5 · (winrate(A first) + winrate(A second)) ``` #### wherein ```text For each pair of answers, you need two launches: first [A, B] then [B, A] Save: winner confidence explanation Large position gap It depends on order, not just quality. ``` #### protection - Make sure to order swap - Read Delta after swap - One pairwise run without rotation ### 06.20 Verbosity bias: A long answer should not win automatically Judge often confuses length with quality: a detailed but empty answer begins to systematically defeat a short but accurate answer. This is especially dangerous in QA and enterprise copilots, where accuracy is more important than text impression. #### two practical metrics ```tex For pair i: len_deltaᵢ = tokens(longer) − tokens(shorter) pref longi = 1 if the judge chooses a longer answer = 0, otherwise long_win_rate = Σ pref_longᵢ / N_pairs You can still count: corr(length_delta, judge_score_delta) Perfect: at matched-quality pairs long_win_rate ≈ 0.5 ``` #### pairing ```text Best audit set: short and long answers, comparable in human quality Then if the judge consistently chooses a long one, It's a verbosity bias. Not a real difference in quality. ``` #### when it pops up - QA and support - essay-style tasks - consider verbosity bias to be “natural” behavior ### 06.21 Self-enhancement bias: Judge loves the answers of a family of models A single-provider judge may be softer on the responses of models of the same ecosystem: the style, length, reasoning structure, and formulation template seem “naturally good.” This is particularly insidious in intermodel comparisons. #### bias score ```text Let's have a balanced pair: answer_X_family vs answer_Y_family Then for Judge X: pref_X = wins_X_family / decisive_pairs For cross-provider judge: pref_X_cross = wins_X_family_cross / decisive_pairs self_enhancement_gap = pref_X − pref_X_cross Big positive gap Judge X overstates his family of models ``` #### How to experiment ```sql Need: matched prompts A balanced set of responses from different families at least two judges from different providers It's not just absolute wins. Change of leader when changing judge ``` #### practice - cross-provider judging - ensemble of judges - Evaluate the closed model only its answers ### 06.22 Prompt Sensitivity: A good judge should not be broken by paraphrasing the rubric If you rewrite the rubric or output format slightly enough, and the final scores are noticeably floating, then the judge is too sensitive to the prompt surface form. Such fragility then turns into erratic experiments and false regressions. #### reckon ```sql There are P equivalent prompt variants: prompt_1 ... prompt_P score_drift = std(metric(prompt_1), ..., metric(prompt_P)) rank_drift = max_rank_shift across prompts prompt_flip_rate = Examples where the verdict changes when changing the wording of rubric ``` #### wherein ```text Same dataset, The same judge model, Only word rubric changes If the score drift is big, Regression after prompt editing Could be an artifact judge- and ``` #### helping - fix the prompt version - rubric examples - Use robust prompt templates ## 07. Pairwise ranking & reporting ### 07.23 Win rate, ties and pairwise preference When two systems are compared on a single prompt, a pairwise verdict is often more stable than an absolute score on a scale. But it is important not to throw away draws: ties and "unsure" carry information about the indistinguishability of systems and the complexity of the case. #### formulas ```tex Say: W_A = wins(A) W_B = wins(B) T = ties U = unsure N = W_A + W_B + T + U overall_win_share(A) = W_A / N decisive_win_rate(A) = W_A / (W_A + W_B) tie_rate = T / N unsure_rate = U / N If you want a significance test, H₀: decisive_win_rate(A) = 0.5 binomtest(W_A, W_A + W_B, 0.5) ``` #### wherein ```text Each prompt judge returns one of: A_WINS B_WINS TIE UNSURE Good practice: Keeping raw verdict and confidence Not just the final win rate. ``` #### practice - swap order for each pair - tie off - Keep an explanation for hard cases ### 07.24 Bradley-Terry / Elo + CI: if there are more than two systems As soon as there are many models, it is more convenient to switch from raw pairwise wins to a general power model. “Bradley-Terry” and “Elo” allow you to aggregate a grid of pairwise results, but without confidence intervals, such a rating is easy to interpret. #### two working formulas ```tex Bradley-Terry: exp(β_A) P(A>B) = ---------------------- exp(β_A) + exp(β_B) β A, β B = hidden forces of systems Which are matched by pairwise wins Elo: E_A = 1 / (1 + 10^((R_B−R_A)/400)) R′_A = R_A + K · (S_A − E_A) S A = 1 on win, 0.5 on tie, 0 on loss ``` #### What to show with the rating ```text ranking score bootstrap CI Number of pairwise matches played tie rate position-bias audit If the intervals overlap strongly, The leaderboard is visually accurate. what he really is ``` #### useful - arena-style eval - Many systems and few pairs per system - Publish Elo without uncertainty ### 07.25 Minimum reporting standard for LLM-as-Judge A properly documented judge-report should allow the other person to understand exactly what was measured, on what data, on what prompt, against what human ceiling, and with what limitations. Everything else quickly turns into an unreplicable number. #### Diagram ```sql CHECLIST PUBLICATION OR INTERNAL REPORT 1. Task and rubric 2. Judge model, prompt version, decoding params 3. Eval dataset size, composition, class balance 4. How many human rates and what kind of human agreement 5. Judge-human agreement: κ / α / ρ / τ 6. Binary metrics: precision / recall / MCC / confusion matrix 7. Stability: repeat runs, flip rate, paired bootstrap CI 8. Calibration: Brier / ECE / reliability plot 9. Bias probes: position / verbosity / self-enhancement / prompt sensitivity 10. Ambiguous cases, ties, domains where judge fails If half of those items are missing, The result is better understood as exploratory. Not as a production-read metric. ``` #### result - replicability - fair comparison of judges - Update the report when changing the judge model - Post only "correlates with humans"