{
  "type": "new-runtime-knowledge-guide",
  "version": 1,
  "generated_at": "2026-07-23",
  "volume": 32,
  "slug": "llm-judge-meta-evaluation-reference",
  "title": "LLM-as-Judge Meta-Evaluation",
  "description": "A separate volume is not about eval in general, but about the validation of the judge itself: agreement with people, metrics for binary and ordinal verdicts, resistance to repeated runs, confidence calibration, bias audits, pairwise ranking and minimum reporting standard. Agreement → classifier view → stability → calibration → bias → reporting.",
  "track": {
    "slug": "evaluation-and-improvement",
    "title": "Evaluation & Improvement",
    "route": "https://newruntime.com/learn/tracks/evaluation-and-improvement/",
    "position": 5
  },
  "counts": {
    "sections": 7,
    "cards": 25
  },
  "routes": {
    "html": "https://newruntime.com/learn/llm-judge-meta-evaluation-reference/",
    "json": "https://newruntime.com/learn/llm-judge-meta-evaluation-reference.json"
  },
  "sections": [
    {
      "number": "01",
      "title": "Frame: Judge should also be evaluated.",
      "intro": "",
      "cards": [
        {
          "number": "01",
          "title": "Judge is not ground truth, but a separate model with its own errors.",
          "tag": "Frame",
          "description": "LLM-as-judge is convenient to use as a quick proxy for human eval, but that doesn’t make it a true metric. If you start to optimize the system for the judge without checking the judge itself, the model quickly learns to “like the appraiser” rather than really improve.",
          "blocks": [
            {
              "kind": "code",
              "label": "Diagram",
              "language": "text",
              "value": "Five Questions to Any Judge\n\n1. Does it match people above chance?\n2. Does it give the same verdict on repeated launches?\n3. Can you express uncertainty, not just a hard label?\n4. Does the verdict depend on the order, length and family of the model?\n5. Does the final score have a confidence interval?\n\nIf the answer to at least one question is no.\nJudge score = heuristic, not reliable KPI\n\nIf all five are passed\nJudge score can be used for regression tracking.\n  A/B comparisons and semi-automatic eval loops"
            },
            {
              "kind": "list",
              "label": "when it is particularly important",
              "items": [
                "leaderboard",
                "production eval loop",
                "reward shaping for the judge score",
                "consider the single judge score to be true"
              ]
            }
          ]
        },
        {
          "number": "02",
          "title": "Minimum meta-eval protocol for new judge",
          "tag": "Frame",
          "description": "Before you connect the judge to the pipeline, you need to go through a short, but mandatory validation loop. The idea is to first understand the ceiling of consent between people, then compare the judge not with one annotator, but with consensus or distribution of human labels.",
          "blocks": [
            {
              "kind": "code",
              "label": "minimum",
              "language": "text",
              "value": "1. 100-200 examples: good/bad/borderline\n2. Mark 2-3 people in the same category\n3. The Inter-Human Agreement\n4. Run the judge on the same set\n5. Agreement Judge vs Human\n6. Check stability, calibration and bias probes\n\nIf inter-human κ = 0.45 and judge-human κ = 0.43\nThe judge is close to the human ceiling.\n\nIf inter-human κ = 0.75 and judge-human κ = 0.38\nJudge is weak, the problem is not the task, but the judge"
            },
            {
              "kind": "list",
              "label": "practice",
              "items": [
                "compared to the human ceiling",
                "borderline cases",
                "Validate the judge on too easy examples"
              ]
            }
          ]
        }
      ]
    },
    {
      "number": "02",
      "title": "Agreement: How much does the judge match people",
      "intro": "",
      "cards": [
        {
          "number": "03",
          "title": "Cohen's kappa: Basic metric for binary pass/fail",
          "tag": "Agreement",
          "description": "If the judge's verdict is binary, 'Cohen's κ' is almost always better than accuracy: it takes into account how much of the coincidences may have come about by chance. This is a good default for tasks like grounded/not grounded, safe/unsafe, pass/fail.",
          "blocks": [
            {
              "kind": "code",
              "label": "Diagram",
              "language": "sql",
              "value": "You need two arrays of the same length:\n\n  human = [1, 0, 1, 1, 0, ...] # consensus or expert label\n  judge = [1, 1, 1, 0, 0, ...] # verdict judge in the same examples\n\nFirst, consider the confusion matrix:\n  TP = human=1 and judge=1\n  TN = human=0 and judge=0\n  FP = human=0, judge=1\n  FN = human=1, judge=0\n  N  = TP + TN + FP + FN\n\nObserved agreement:\n  P_observed = (TP + TN) / N\n\nChance agreement from the margins:\n  P_chance =\n    ((TP+FP)/N)·((TP+FN)/N) +\n    ((TN+FN)/N)·((TN+FP)/N)\n\n  κ = (P_observed − P_chance) / (1 − P_chance)\n\nExample:\n  TP=35, TN=51, FP=9, FN=5, N=100\n  P_observed = (35+51)/100 = 0.86\n  P_chance   = 0.44·0.40 + 0.56·0.60 = 0.512\n  κ          = (0.86−0.512)/(1−0.512) = 0.713\n\nRough interpretation:\n  κ < 0.40 → judge weak\n  0.40-0.60 Conditionally acceptable\n  0.60-0.80 Good working level\n  > 0.80 Very strong agreement\n\nBut:\n  With a strong class imbalance, κ may look severe.\n  However, raw accuracy seems to be high.\n\nMeaning:\n  accuracy = \"how often did you match\"\n  κ = \"how often coincided beyond chance\""
            },
            {
              "kind": "code",
              "label": "how to count",
              "language": "sql",
              "value": "from sklearn.metrics import cohen_kappa_score\n\nhuman = [1, 0, 1, ...]\njudge = [1, 1, 1, ...]\n\nkappa = cohen_kappa_score(human, judge)\n#human labels: from 2-3 experts or majority vote\n# judge labels: from the same eval set, in the same order"
            },
            {
              "kind": "list",
              "label": "when to use",
              "items": [
                "binary rubric",
                "judge vs consensus label",
                "Look at the confusion matrix",
                "Does not replace calibration and bias probes"
              ]
            }
          ]
        },
        {
          "number": "04",
          "title": "Weighted kappa: If the score is on a scale of 1-5",
          "tag": "Agreement",
          "description": "For ordinal labels, the usual 'κ' is too rough: the error '5→4' and the error '5→1' are considered the same. 'Weighted κ' gives a partial penalty and usually better reflects the real quality of the judge on scales.",
          "blocks": [
            {
              "kind": "code",
              "label": "formula",
              "language": "sql",
              "value": "We need two ordinal ratings:\nhuman = [5, 3, 4, ...]\njudge = [4, 3, 5, ...]\n\nWe construct observed matrix O(i,j) and expected matrix E(i,j):\n  O = pair frequencies (human=i, judge=j)\n  E = expected frequencies from marginals\n\n  κ_w = 1 − (ΣᵢΣⱼ wᵢⱼ Oᵢⱼ) / (ΣᵢΣⱼ wᵢⱼ Eᵢⱼ)\n\nLinear weights:\n  wᵢⱼ = |i−j| / (K−1)\n\nQuadratic weights:\n  wᵢⱼ = (i−j)² / (K−1)²\n\nWhere to get the values:\n  K = number of scale levels\n  O and E = from confusion matrix by ratings"
            },
            {
              "kind": "code",
              "label": "use",
              "language": "sql",
              "value": "from sklearn.metrics import cohen_kappa_score\n\nk_linear = cohen_kappa_score(human, judge, weights=\"linear\")\nk_quad   = cohen_kappa_score(human, judge, weights=\"quadratic\")\n\nPractically:\n  if the adjacent levels are almost uniform\n  quadratic if long-range misses are particularly painful"
            },
            {
              "kind": "list",
              "label": "when appropriate",
              "items": [
                "1-4, 1-5, 1-10",
                "Only with clear anchor examples",
                "Not to use as a substitute for binary rubric"
              ]
            }
          ]
        },
        {
          "number": "05",
          "title": "Fleiss' kappa: if there are more than two people or judges",
          "tag": "Agreement",
          "description": "Once in the 3+ evaluator problem, 'Cohen's κ' no longer fits. 'Fleiss' κ' gives a chance-corrected agreement at the level of the entire annotator pool and shows well whether there is consensus at all.",
          "blocks": [
            {
              "kind": "code",
              "label": "formula",
              "language": "tex",
              "value": "Say:\n  N = number of examples\n  n = number of appraisers for example\n  k = number of classes\n  nij = how many appraisers gave example i class j\n\nAgreement within Example i:\n         1\n  Pᵢ = ------- · Σⱼ nᵢⱼ(nᵢⱼ−1)\n       n(n−1)\n\nAverage observed agreement:\n  P̄ = (1/N) · Σᵢ Pᵢ\n\nShare of class j over the pool:\n           1\n  pⱼ = -------- · Σᵢ nᵢⱼ\n        N · n\n\nChance agreement:\n  P̄ₑ = Σⱼ pⱼ²\n\n  Fleiss κ = (P̄ − P̄ₑ) / (1 − P̄ₑ)"
            },
            {
              "kind": "code",
              "label": "wherein",
              "language": "text",
              "value": "I need a table.\n  rows = examples\n  columns = raters\n  values  = class label\n\nExample:\n  tc_001  [PASS, PASS, FAIL]\n  tc_002  [FAIL, FAIL, FAIL]\n  tc_003  [PASS, PASS, PASS]\n\nIt is considered nij for each example."
            },
            {
              "kind": "list",
              "label": "scenario",
              "items": [
                "3+ human raters",
                "Comparison of several judge models",
                "I don't like messy missing labels."
              ]
            }
          ]
        },
        {
          "number": "06",
          "title": "Krippendorff's alpha: the most versatile option for messy eval",
          "tag": "Agreement",
          "description": "'Krippendorff's α' is useful where the actual markup is not ideal: different types of scales, omissions, not all examples have the same number of annotators. In practical eval, it is often a better long-term choice than a zoo of different κ variants.",
          "blocks": [
            {
              "kind": "code",
              "label": "formula",
              "language": "sql",
              "value": "α = 1 − D_observed / D_expected\n\nWhere:\n  D observed = actual disagreement between annotators\n  D expected = Disagreement expected by chance\n\nOrdinal labels are usually defined as distance:\n  δ(c, c′) = (c − c′)²\n\nThen great discrepancies are fined more.\nas in quadratic weighted κ\n\nIf α → 1: almost complete consensus\nIf α → 0: no better than chance"
            },
            {
              "kind": "code",
              "label": "wherein",
              "language": "sql",
              "value": "We need an item × rater matrix:\n\n[\n  [1, 1, None, 0],\n  [2, 2, 2,    2],\n  [4, 3, 4,    None],\n]\n\nfrom krippendorff import alpha\nalpha_value = alpha(reliability_data=data, level_of_measurement=\"ordinal\")"
            },
            {
              "kind": "list",
              "label": "when",
              "items": [
                "messy annotation pipeline",
                "partial",
                "more difficult to explain to the team from scratch"
              ]
            }
          ]
        },
        {
          "number": "07",
          "title": "Pearson r vs Spearman rho vs Kendall tau",
          "tag": "Agreement",
          "description": "When a judge returns a score or a rank, there are three types of connection. For most LLM-eval tasks, ‘Spearman ρ’ is better than default. 'Kendall τ' is useful for the stability of order. Pearson r only makes sense if the score is close to the interval and you are interested in linear communication.",
          "blocks": [
            {
              "kind": "code",
              "label": "Diagram",
              "language": "tex",
              "value": "You need two arrays of the same length:\n\n  human_scores = [4, 2, 5, 3, ...]\n  judge_scores = [5, 2, 4, 3, ...]\n\nPearson r is a linear link:\n                 Σ (xᵢ−x̄)(yᵢ−ȳ)\n  r = -----------------------------------\n      sqrt(Σ(xᵢ−x̄)² · Σ(yᵢ−ȳ)²)\n\nSpearman rho - Pearson by rank:\n  ρ = Pearson(rank(x), rank(y))\n\nIf there are no ties, you can write like this:\n           6 Σ dᵢ²\n  ρ = 1 − ----------\n         n(n²−1)\n\nKendall tau is a paired agreement:\n      C − D\n  τ = -------\n      C + D\n\n  C = concordant pairs\n  D = discordant pairs"
            },
            {
              "kind": "code",
              "label": "How to count and what to provide as input",
              "language": "sql",
              "value": "from scipy.stats import pearsonr, spearmanr, kendalltau\n\npearson_r,  _ = pearsonr(human_scores, judge_scores)\nspearman_rho, _ = spearmanr(human_scores, judge_scores)\nkendall_tau, _ = kendalltau(human_scores, judge_scores)\n\nHuman scores: Average human score or rank in each example\njudge scores: score judge in the same examples\n\nFor the leaderboard:\n  human rank = rank of the system n\n  judge rank = rank of the same system n"
            },
            {
              "kind": "table",
              "headers": [
                "Metrica",
                "What meter",
                "When you're good.",
                "When dangerous"
              ],
              "rows": [
                [
                  "Pearson r",
                  "Linear communication",
                  "interval",
                  "ordinal scales 1-5"
                ],
                [
                  "Spearman rho",
                  "Monotonic rank",
                  "default for judge scores",
                  "distanceless"
                ],
                [
                  "Kendall tau",
                  "Pair consent n",
                  "rank stability, leaderboard",
                  "less familiar"
                ]
              ]
            },
            {
              "kind": "list",
              "label": "conclusion",
              "items": [
                "Spearman as first choice",
                "Kendall for System Order",
                "Pearson only if there is a reason"
              ]
            }
          ]
        }
      ]
    },
    {
      "number": "03",
      "title": "Judge as a classifier",
      "intro": "",
      "cards": [
        {
          "number": "08",
          "title": "Matthews Correlation Coefficient: A binary metric for imbalance",
          "tag": "Classifier",
          "description": "MCC is especially useful when positive class is rare: hallucination, unsafe answer, policy violation. Unlike accuracy and even F1, it takes into account all the confusion matrix and gives an honest picture of the quality of the judge in unbalanced tasks.",
          "blocks": [
            {
              "kind": "code",
              "label": "Diagram",
              "language": "tex",
              "value": "Through TP, TN, FP, FN:\n\n                 TP·TN − FP·FN\n  MCC = -----------------------------------\n        sqrt((TP+FP)(TP+FN)(TN+FP)(TN+FN))\n\nInterpretation:\n  -1: systematically wrong\n   No better than chance.\n  +1 - The perfect classifier\n\nPractical sense:\n  if unsafe only 5%,\n  95% accuracy can be applied to any judge.\n  MCC will immediately show the empty"
            },
            {
              "kind": "code",
              "label": "wherein",
              "language": "sql",
              "value": "First you fix the positive class:\n  1 = unsafe, 0 = safe\n\nThen on the same eval set:\n  y_true = human / gold labels\n  y_pred = hard verdict judge\n\nfrom sklearn.metrics import matthews_corrcoef\nmcc = matthews_corrcoef(y_true, y_pred)\n\nIf the judge gives you a probability:\n  y_pred = 1[p_i ≥ threshold]"
            },
            {
              "kind": "list",
              "label": "particularly useful",
              "items": [
                "safety eval",
                "hallucination detection",
                "rare negative cases",
                "Read with the Reference Dangerous Class"
              ]
            }
          ]
        },
        {
          "number": "09",
          "title": "Accuracy cheats if classes are unbalanced",
          "tag": "Classifier",
          "description": "A judge who always says ‘PASS’ can look strong on a dataset where 90% of the examples are true pass. Precision can almost never be read alone.",
          "blocks": [
            {
              "kind": "code",
              "label": "formulae",
              "language": "sql",
              "value": "accuracy          = (TP + TN) / N\nprecision_pos     = TP / (TP + FP)\nrecall_pos        = TP / (TP + FN)\nspecificity       = TN / (TN + FP)\nF1                = 2 · precision · recall / (precision + recall)\nbalanced_accuracy = (recall_pos + specificity) / 2\n\nWhere to get the values:\n  TP, TN, FP, FN = from judge verdict vs gold label\n\nAn example of accuracy cheating:\n  unsafe = 5%, judge always says SAFE\n  accuracy = 95%\n  recall_unsafe = 0%\n  balanced_accuracy = 50%"
            },
            {
              "kind": "code",
              "label": "code",
              "language": "sql",
              "value": "from sklearn.metrics import accuracy_score, precision_recall_fscore_support\nfrom sklearn.metrics import balanced_accuracy_score\n\nacc  = accuracy_score(y_true, y_pred)\nprec, rec, f1, _ = precision_recall_fscore_support(y_true, y_pred, average=\"binary\")\nbacc = balanced_accuracy_score(y_true, y_pred)"
            },
            {
              "kind": "list",
              "label": "red flags",
              "items": [
                "Accuracy without class balance",
                "No breakdown by FP/FN errors",
                "classify"
              ]
            }
          ]
        },
        {
          "number": "10",
          "title": "ROC-AUC vs PR-AUC: Which Curve to Look at",
          "tag": "Classifier",
          "description": "If the judge returns the probability, you can rate it as a ranker. “ROC-AUC” shows overall separability, but in rare positive classes, it is often too optimistic. PR-AUC better reflects how the judge actually finds rare violations.",
          "blocks": [
            {
              "kind": "code",
              "label": "Diagram",
              "language": "text",
              "value": "Need:\n\n  y_true  = [0, 1, 0, 0, 1, ...]\n  p_judge = [0.03, 0.91, 0.22, 0.10, 0.78, ...]\n\nFor any threshold t:\n  TPR(t) = TP / (TP + FN)\n  FPR(t) = FP / (FP + TN)\n  Precision(t) = TP / (TP + FP)\n  Recall(t)    = TP / (TP + FN)\n\nROC curve:\n  x = FPR(t), y = TPR(t)\n  ROC-AUC = area under this curve\n\nPR curve:\n  x = Recall(t), y = Precision(t)\n  PR-AUC = area under this curve\n\nMeaning:\n  ROC-AUC Ranks Positives Above Negatives\n  How accurate is PR-AUC in a rare positive class?"
            },
            {
              "kind": "code",
              "label": "reckon",
              "language": "sql",
              "value": "from sklearn.metrics import roc_auc_score, average_precision_score\n\nroc_auc = roc_auc_score(y_true, p_judge)\npr_auc  = average_precision_score(y_true, p_judge)\n\nImportant:\n  Probabilities / confidences are provided here.\n  Not hard labels PASS/FAIL"
            },
            {
              "kind": "table",
              "headers": [
                "Metrica",
                "What shows",
                "Best."
              ],
              "rows": [
                [
                  "ROC-AUC",
                  "tradeoff TPR/FPR",
                  "class"
                ],
                [
                  "PR-AUC",
                  "precision/recall positive",
                  "Rare problems, safety, hallucinations"
                ]
              ]
            },
            {
              "kind": "list",
              "label": "rule",
              "items": [
                "Rarely positive to watch PR-AUC",
                "ROC-AUC as secondary"
              ]
            }
          ]
        },
        {
          "number": "11",
          "title": "Confusion Matrix Judge: What Errors Are Really Expensive",
          "tag": "Classifier",
          "description": "Judge's not judging in a vacuum. Sometimes the false ‘PASS’ is the most dangerous, sometimes the false ‘FAIL’ turns the system into an over-refusal machine. Therefore, confusion matrix should be associated with the cost of error, not just a beautiful final metric.",
          "blocks": [
            {
              "kind": "code",
              "label": "how to build it",
              "language": "text",
              "value": "If the judge gives confidence p:\n  y_pred = 1[p ≥ t]\n\nThen you think:\n            judge=1   judge=0\nhuman=1       TP        FN\nhuman=0       FP        TN\n\nThe threshold t is not chosen for beauty.\na for the cost of errors FP and FN"
            },
            {
              "kind": "code",
              "label": "Example of cost-aware reading",
              "language": "text",
              "value": "Faithfulness eval:\n  FN judge = missed hallucination\n  FP judge = punished with correct answer\n\nModeration eval:\n  FN judge = missed unsafe content\n  FP judge = blocked harmless response\n\nSame accuracy.\nmay be acceptable in one case.\nand unacceptable elsewhere"
            },
            {
              "kind": "list",
              "label": "do",
              "items": [
                "fix the target operating point",
                "Discuss FP and FN separately",
                "Optimize only the average metric"
              ]
            }
          ]
        }
      ]
    },
    {
      "number": "04",
      "title": "Stability & uncertainty",
      "intro": "",
      "cards": [
        {
          "number": "12",
          "title": "Intra-rater reliability: Judge must match itself",
          "tag": "Stability",
          "description": "Even with ‘temperature=0’ and one model, the judge may be unstable due to long reasoning paths, backend sampling, or subtle prompt shifts. If the same example receives different verdicts, the point score becomes meaningless.",
          "blocks": [
            {
              "kind": "code",
              "label": "How to measure binary and score verdicts",
              "language": "text",
              "value": "For each example i run judge K times:\n  vᵢ = [vᵢ₁, vᵢ₂, ..., vᵢₖ]\n\nIf verdict binary:\n  c pass = number of PASS\n  c fail = number of FAIL\n  self_agreementᵢ = max(c_pass, c_fail) / K\n\nIf the verdict numeric:\n  μᵢ = mean(vᵢ)\n  σᵢ = std(vᵢ)\n  MADᵢ = mean(|vᵢⱼ − μᵢ|)\n\nOutcome on dataset:\n  mean_self_agreement = mean(self_agreementᵢ)\n  mean_sigma          = mean(σᵢ)"
            },
            {
              "kind": "code",
              "label": "wherein",
              "language": "text",
              "value": "Same thing:\n  prompt\n  context\n  answer\n  rubric\n\nYou run Ks with the same judge.\nAnd you only keep your verdict/confidence.\ndataset-free"
            },
            {
              "kind": "list",
              "label": "signal",
              "items": [
                "Frequent flips on borderline cases",
                "You need abstain or uncertainty",
                "Average a few runs only if it is conscious"
              ]
            }
          ]
        },
        {
          "number": "13",
          "title": "Flip rate and self-inconsistency",
          "tag": "Stability",
          "description": "Convenient Instability Metric: How often a judge changes his mind when repeating the same task. In production, this is easier to communicate than abstract variance: “The judge contradicts itself by 14% of the examples.”",
          "blocks": [
            {
              "kind": "code",
              "label": "formula",
              "language": "tex",
              "value": "For example i:\n  unstablei = 1, if unique(vi1 ... vik) > 1\n             = 0, otherwise\n\n                1\nflip_rate = ----- · Σᵢ unstableᵢ\n                N\n\nIf the verdict numeric:\n  First, transfer it to the label.\n  e.g. PASS if score ≥ 4"
            },
            {
              "kind": "code",
              "label": "example",
              "language": "text",
              "value": "tc_014 → PASS, PASS, FAIL, PASS, FAIL\nunstable_014 = 1\n\ntc_015 → FAIL, FAIL, FAIL, FAIL, FAIL\nunstable_015 = 0\n\nIf 14 out of 100 examples are unstable,\nflip_rate = 0.14"
            },
            {
              "kind": "list",
              "label": "useful",
              "items": [
                "Comparison of judge prompts",
                "comparison",
                "It is not a substitute for people."
              ]
            }
          ]
        },
        {
          "number": "14",
          "title": "Paired bootstrap CI: compare systems not by one digit, but by interval",
          "tag": "Stability",
          "description": "If two systems judged on the same set of examples, you need to compare pairs. 'Paired bootstrap' gives a confidence interval for the difference in metrics and helps distinguish real improvements from sample noise.",
          "blocks": [
            {
              "kind": "code",
              "label": "Diagram",
              "language": "python",
              "value": "Algorithm for score(A) − score(B):\n\nfor b in 1..10000:\n  sample_idx = resample(example_ids, replace=True)\n  delta_b = metric(A[sample_idx]) - metric(B[sample_idx])\n\nCI_95 = percentile(delta, 2.5), percentile(delta, 97.5)\n\nInterpretation:\n  if 0 within CI\n  Improvement is unconvincing\n\n  if all CI is > 0\n  A is statistically better than B on this dataset\n\nKeyword: paired\nBecause both systems are evaluated on the same examples."
            },
            {
              "kind": "code",
              "label": "wherein",
              "language": "text",
              "value": "Each example requires an outcome for both systems:\n\nexample_id | score_A | score_B\ntc_001     | 1       | 0\ntc_002     | 1       | 1\ntc_003     | 0       | 1\n\nmetric may be:\n  accuracy\n  pass_rate\n  mean judge score\n  win_rate\n\nIt's not tokens or answers that are being reassembled.\n(a) Example indexes"
            },
            {
              "kind": "list",
              "label": "where must-have",
              "items": [
                "A/B on same eval set",
                "leaderboard deltas",
                "Compare point estimates without CI"
              ]
            }
          ]
        },
        {
          "number": "15",
          "title": "The judge must be able to say unsure.",
          "tag": "Stability",
          "description": "Forced choice gives a beautiful appearance of certainty, but breaks meta-eval on really ambiguous examples. Sometimes the best judge is not the one who always answers, but the one who can send the case to the human review.",
          "blocks": [
            {
              "kind": "code",
              "label": "two useful formulae",
              "language": "tex",
              "value": "Let the judge give confidence p . [0.1].\n\ncoverage(τ) =\n  #(pᵢ ≥ τ) / N\n\nselective_risk(τ) =\n  errors in examples with pi ≥ τ / #(pi ≥ τ)\n\nMeaning:\n  increase the threshold τ\n  Coverage is falling\n  Selective risk should also fall."
            },
            {
              "kind": "code",
              "label": "three modes",
              "language": "text",
              "value": "High confidence:\n  auto-score\n\nMedium confidence:\n  Consider, but mark as weak evidence\n\nLow confidence / tie / unsure:\n  send out\n\nThis is especially important for rating indeterminacy.\nborderline examples"
            },
            {
              "kind": "list",
              "label": "whenever",
              "items": [
                "ambiguous rubric",
                "pairwise ties",
                "Downstream Unsure Processing Logic"
              ]
            }
          ]
        }
      ]
    },
    {
      "number": "05",
      "title": "Calibration: Can you trust the judge? and",
      "intro": "",
      "cards": [
        {
          "number": "16",
          "title": "Brier Score: how likely the judge is to be",
          "tag": "Calibration",
          "description": "If a judge gives a probability of 'PASS', 'SAFE' or 'A wins', 'Brier Score' is almost always the first. This is a quadratic error of probabilistic prediction: both calibration and sharpness in one digit.",
          "blocks": [
            {
              "kind": "code",
              "label": "Diagram",
              "language": "text",
              "value": "For binary event y   {0.1} and prediction p:\n\n  Brier = mean((p − y)^2)\n\nExample:\n  Judge says 0.90, event happened → error 0. 01\n  The judge says 0.90, the event did not happen → error 0. 81\n\nBelow.\n\nIf the judge has only hard labels without confidence,\nBrier cannot be counted without a separate confidence head."
            },
            {
              "kind": "code",
              "label": "where to get p and y",
              "language": "sql",
              "value": "For each example i:\n  pi = confidence judge that label = 1\n  yi = real binary label from human/gold\n\nExample:\n  p = [0.91, 0.72, 0.10, ...]\n  y = [1,    0,    0,    ...]\n\nfrom sklearn.metrics import brier_score_loss\nbrier = brier_score_loss(y, p)\n\nIf the judge gives confidence 0-100,\nsplit by 100 first"
            },
            {
              "kind": "list",
              "label": "Good friend",
              "items": [
                "reliability diagram",
                "ECE",
                "You need the correct probability output"
              ]
            }
          ]
        },
        {
          "number": "17",
          "title": "Previous PostPrevious Expected Calibration Error: Is the judge as sure as right?",
          "tag": "Calibration",
          "description": "ECE compares confidence to its actual success rate in the basket of probability. If the judge says ‘90% confident’ and is right only 65% of the time, that’s overconfidence.",
          "blocks": [
            {
              "kind": "code",
              "label": "Diagram",
              "language": "tex",
              "value": "Divide predictions into M bins by confidence:\n\n  Bₘ = { i : pᵢ ∈ ((m−1)/M, m/M] }\n\naccuracy(Bₘ)   = (1 / |Bₘ|) · Σᵢ 1[ŷᵢ = yᵢ]\nconfidence(Bₘ) = (1 / |Bₘ|) · Σᵢ pᵢ\n\nECE = Σₘ (|Bₘ| / N) · |accuracy(Bₘ) − confidence(Bₘ)|\n\nInterpretation:\n  ECE ≈ 0 → well calibrated\n  ECE Upgrades the Gap Between Confidence and Reality\n\nCareful:\n  ECE is sensitive to the binning scheme\n  Therefore, it is useful to show both a schedule and a number."
            },
            {
              "kind": "code",
              "label": "file",
              "language": "text",
              "value": "ŷᵢ = predicted judge label\nPi = confidence in this particular label\nyᵢ = gold / human label\n\nUsually:\n  M = 10 bins\n  or equal-width,\n  or equal-mass binning"
            },
            {
              "kind": "list",
              "label": "crucial",
              "items": [
                "count",
                "point out",
                "Read more about ECE without a reliable plot"
              ]
            }
          ]
        },
        {
          "number": "18",
          "title": "Reliability diagram: the most visual calibration audit",
          "tag": "Calibration",
          "description": "One number hides the form of the error. Reliability diagram shows where the judge overestimates himself: on high confidence, on medium or only in a narrow range. This is often more useful than any summary metric.",
          "blocks": [
            {
              "kind": "code",
              "label": "construct",
              "language": "text",
              "value": "1. Break predictions by confidence bins\n2. For each bin count:\n   mean_confidence\n   empirical_accuracy\n3. Draw dots:\n   x = mean_confidence\n   y = empirical_accuracy\n4. Add diagonal y=x as ideal"
            },
            {
              "kind": "code",
              "label": "how to read",
              "language": "text",
              "value": "Perfect line:\n  confidence = empirical accuracy\n\nCurve below diagonal:\n  judge overconfident\n\nThe curve above the diagonal:\n  judge underconfident\n\nSeparately useful to watch:\n  coverage by bins\n  borderline subset\n  hard cases only"
            },
            {
              "kind": "list",
              "label": "useful",
              "items": [
                "threshold tuning",
                "abstain / human review policy",
                "show the team without a statistical background"
              ]
            }
          ]
        }
      ]
    },
    {
      "number": "06",
      "title": "Bias audits: where the judge is systematically distorted",
      "intro": "",
      "cards": [
        {
          "number": "19",
          "title": "Position bias: first or second response gets a head start",
          "tag": "Bias",
          "description": "In a pairwise score, the judge may prefer the first or second answer simply because of the position. If you do not do A/B swap, you can get a false leaderboard even with a strong judge model.",
          "blocks": [
            {
              "kind": "code",
              "label": "reckon",
              "language": "text",
              "value": "judge([A, B]) → verdict₁\njudge([B, A]) → verdict₂\n\nwinrate(A first)  = wins_A_when_first / pairs_with_A_first\nwinrate(A second) = wins_A_when_second / pairs_with_A_second\n\nposition_gap = |winrate(A first) − winrate(A second)|\n\nswap_robust_winrate(A) =\n  0.5 · (winrate(A first) + winrate(A second))"
            },
            {
              "kind": "code",
              "label": "wherein",
              "language": "text",
              "value": "For each pair of answers, you need two launches:\n  first [A, B]\n  then [B, A]\n\nSave:\n  winner\n  confidence\n  explanation\n\nLarge position gap\nIt depends on order, not just quality."
            },
            {
              "kind": "list",
              "label": "protection",
              "items": [
                "Make sure to order swap",
                "Read Delta after swap",
                "One pairwise run without rotation"
              ]
            }
          ]
        },
        {
          "number": "20",
          "title": "Verbosity bias: A long answer should not win automatically",
          "tag": "Bias",
          "description": "Judge often confuses length with quality: a detailed but empty answer begins to systematically defeat a short but accurate answer. This is especially dangerous in QA and enterprise copilots, where accuracy is more important than text impression.",
          "blocks": [
            {
              "kind": "code",
              "label": "two practical metrics",
              "language": "tex",
              "value": "For pair i:\n  len_deltaᵢ = tokens(longer) − tokens(shorter)\n  pref longi = 1 if the judge chooses a longer answer\n             = 0, otherwise\n\nlong_win_rate =\n  Σ pref_longᵢ / N_pairs\n\nYou can still count:\n  corr(length_delta, judge_score_delta)\n\nPerfect:\n  at matched-quality pairs\n  long_win_rate ≈ 0.5"
            },
            {
              "kind": "code",
              "label": "pairing",
              "language": "text",
              "value": "Best audit set:\n  short and long answers,\n  comparable in human quality\n\nThen if the judge consistently chooses a long one,\nIt's a verbosity bias.\nNot a real difference in quality."
            },
            {
              "kind": "list",
              "label": "when it pops up",
              "items": [
                "QA and support",
                "essay-style tasks",
                "consider verbosity bias to be “natural” behavior"
              ]
            }
          ]
        },
        {
          "number": "21",
          "title": "Self-enhancement bias: Judge loves the answers of a family of models",
          "tag": "Bias",
          "description": "A single-provider judge may be softer on the responses of models of the same ecosystem: the style, length, reasoning structure, and formulation template seem “naturally good.” This is particularly insidious in intermodel comparisons.",
          "blocks": [
            {
              "kind": "code",
              "label": "bias score",
              "language": "text",
              "value": "Let's have a balanced pair:\n  answer_X_family vs answer_Y_family\n\nThen for Judge X:\n  pref_X = wins_X_family / decisive_pairs\n\nFor cross-provider judge:\n  pref_X_cross = wins_X_family_cross / decisive_pairs\n\nself_enhancement_gap =\n  pref_X − pref_X_cross\n\nBig positive gap\nJudge X overstates his family of models"
            },
            {
              "kind": "code",
              "label": "How to experiment",
              "language": "sql",
              "value": "Need:\n  matched prompts\n  A balanced set of responses from different families\n  at least two judges from different providers\n\nIt's not just absolute wins.\nChange of leader when changing judge"
            },
            {
              "kind": "list",
              "label": "practice",
              "items": [
                "cross-provider judging",
                "ensemble of judges",
                "Evaluate the closed model only its answers"
              ]
            }
          ]
        },
        {
          "number": "22",
          "title": "Prompt Sensitivity: A good judge should not be broken by paraphrasing the rubric",
          "tag": "Bias",
          "description": "If you rewrite the rubric or output format slightly enough, and the final scores are noticeably floating, then the judge is too sensitive to the prompt surface form. Such fragility then turns into erratic experiments and false regressions.",
          "blocks": [
            {
              "kind": "code",
              "label": "reckon",
              "language": "sql",
              "value": "There are P equivalent prompt variants:\n  prompt_1 ... prompt_P\n\nscore_drift =\n  std(metric(prompt_1), ..., metric(prompt_P))\n\nrank_drift =\n  max_rank_shift across prompts\n\nprompt_flip_rate =\n  Examples where the verdict changes\n  when changing the wording of rubric"
            },
            {
              "kind": "code",
              "label": "wherein",
              "language": "text",
              "value": "Same dataset,\nThe same judge model,\nOnly word rubric changes\n\nIf the score drift is big,\nRegression after prompt editing\nCould be an artifact judge- and"
            },
            {
              "kind": "list",
              "label": "helping",
              "items": [
                "fix the prompt version",
                "rubric examples",
                "Use robust prompt templates"
              ]
            }
          ]
        }
      ]
    },
    {
      "number": "07",
      "title": "Pairwise ranking & reporting",
      "intro": "",
      "cards": [
        {
          "number": "23",
          "title": "Win rate, ties and pairwise preference",
          "tag": "Pairwise",
          "description": "When two systems are compared on a single prompt, a pairwise verdict is often more stable than an absolute score on a scale. But it is important not to throw away draws: ties and \"unsure\" carry information about the indistinguishability of systems and the complexity of the case.",
          "blocks": [
            {
              "kind": "code",
              "label": "formulas",
              "language": "tex",
              "value": "Say:\n  W_A = wins(A)\n  W_B = wins(B)\n  T   = ties\n  U   = unsure\n  N   = W_A + W_B + T + U\n\noverall_win_share(A) = W_A / N\ndecisive_win_rate(A) = W_A / (W_A + W_B)\ntie_rate             = T / N\nunsure_rate          = U / N\n\nIf you want a significance test,\n  H₀: decisive_win_rate(A) = 0.5\n  binomtest(W_A, W_A + W_B, 0.5)"
            },
            {
              "kind": "code",
              "label": "wherein",
              "language": "text",
              "value": "Each prompt judge returns one of:\n  A_WINS\n  B_WINS\n  TIE\n  UNSURE\n\nGood practice:\n  Keeping raw verdict and confidence\n  Not just the final win rate."
            },
            {
              "kind": "list",
              "label": "practice",
              "items": [
                "swap order for each pair",
                "tie off",
                "Keep an explanation for hard cases"
              ]
            }
          ]
        },
        {
          "number": "24",
          "title": "Bradley-Terry / Elo + CI: if there are more than two systems",
          "tag": "Pairwise",
          "description": "As soon as there are many models, it is more convenient to switch from raw pairwise wins to a general power model. “Bradley-Terry” and “Elo” allow you to aggregate a grid of pairwise results, but without confidence intervals, such a rating is easy to interpret.",
          "blocks": [
            {
              "kind": "code",
              "label": "two working formulas",
              "language": "tex",
              "value": "Bradley-Terry:\n           exp(β_A)\nP(A>B) = ----------------------\n         exp(β_A) + exp(β_B)\n\nβ A, β B = hidden forces of systems\nWhich are matched by pairwise wins\n\nElo:\n  E_A = 1 / (1 + 10^((R_B−R_A)/400))\n  R′_A = R_A + K · (S_A − E_A)\n\n  S A = 1 on win, 0.5 on tie, 0 on loss"
            },
            {
              "kind": "code",
              "label": "What to show with the rating",
              "language": "text",
              "value": "ranking score\nbootstrap CI\nNumber of pairwise matches played\ntie rate\nposition-bias audit\n\nIf the intervals overlap strongly,\nThe leaderboard is visually accurate.\nwhat he really is"
            },
            {
              "kind": "list",
              "label": "useful",
              "items": [
                "arena-style eval",
                "Many systems and few pairs per system",
                "Publish Elo without uncertainty"
              ]
            }
          ]
        },
        {
          "number": "25",
          "title": "Minimum reporting standard for LLM-as-Judge",
          "tag": "Reporting",
          "description": "A properly documented judge-report should allow the other person to understand exactly what was measured, on what data, on what prompt, against what human ceiling, and with what limitations. Everything else quickly turns into an unreplicable number.",
          "blocks": [
            {
              "kind": "code",
              "label": "Diagram",
              "language": "sql",
              "value": "CHECLIST PUBLICATION OR INTERNAL REPORT\n\n1. Task and rubric\n2. Judge model, prompt version, decoding params\n3. Eval dataset size, composition, class balance\n4. How many human rates and what kind of human agreement\n5. Judge-human agreement: κ / α / ρ / τ\n6. Binary metrics: precision / recall / MCC / confusion matrix\n7. Stability: repeat runs, flip rate, paired bootstrap CI\n8. Calibration: Brier / ECE / reliability plot\n9. Bias probes: position / verbosity / self-enhancement / prompt sensitivity\n10. Ambiguous cases, ties, domains where judge fails\n\nIf half of those items are missing,\nThe result is better understood as exploratory.\nNot as a production-read metric."
            },
            {
              "kind": "list",
              "label": "result",
              "items": [
                "replicability",
                "fair comparison of judges",
                "Update the report when changing the judge model",
                "Post only \"correlates with humans\""
              ]
            }
          ]
        }
      ]
    }
  ]
}