LLM-as-Judge Meta-Evaluation

Five Questions to Any Judge

1. Does it match people above chance?
2. Does it give the same verdict on repeated launches?
3. Can you express uncertainty, not just a hard label?
4. Does the verdict depend on the order, length and family of the model?
5. Does the final score have a confidence interval?

If the answer to at least one question is no.
Judge score = heuristic, not reliable KPI

If all five are passed
Judge score can be used for regression tracking.
  A/B comparisons and semi-automatic eval loops

1. 100-200 examples: good/bad/borderline
2. Mark 2-3 people in the same category
3. The Inter-Human Agreement
4. Run the judge on the same set
5. Agreement Judge vs Human
6. Check stability, calibration and bias probes

If inter-human κ = 0.45 and judge-human κ = 0.43
The judge is close to the human ceiling.

If inter-human κ = 0.75 and judge-human κ = 0.38
Judge is weak, the problem is not the task, but the judge

You need two arrays of the same length:

  human = [1, 0, 1, 1, 0, ...] # consensus or expert label
  judge = [1, 1, 1, 0, 0, ...] # verdict judge in the same examples

First, consider the confusion matrix:
  TP = human=1 and judge=1
  TN = human=0 and judge=0
  FP = human=0, judge=1
  FN = human=1, judge=0
  N  = TP + TN + FP + FN

Observed agreement:
  P_observed = (TP + TN) / N

Chance agreement from the margins:
  P_chance =
    ((TP+FP)/N)·((TP+FN)/N) +
    ((TN+FN)/N)·((TN+FP)/N)

  κ = (P_observed − P_chance) / (1 − P_chance)

Example:
  TP=35, TN=51, FP=9, FN=5, N=100
  P_observed = (35+51)/100 = 0.86
  P_chance   = 0.44·0.40 + 0.56·0.60 = 0.512
  κ          = (0.86−0.512)/(1−0.512) = 0.713

Rough interpretation:
  κ < 0.40 → judge weak
  0.40-0.60 Conditionally acceptable
  0.60-0.80 Good working level
  > 0.80 Very strong agreement

But:
  With a strong class imbalance, κ may look severe.
  However, raw accuracy seems to be high.

Meaning:
  accuracy = "how often did you match"
  κ = "how often coincided beyond chance"

from sklearn.metrics import cohen_kappa_score

human = [1, 0, 1, ...]
judge = [1, 1, 1, ...]

kappa = cohen_kappa_score(human, judge)
#human labels: from 2-3 experts or majority vote
# judge labels: from the same eval set, in the same order

We need two ordinal ratings:
human = [5, 3, 4, ...]
judge = [4, 3, 5, ...]

We construct observed matrix O(i,j) and expected matrix E(i,j):
  O = pair frequencies (human=i, judge=j)
  E = expected frequencies from marginals

  κ_w = 1 − (ΣᵢΣⱼ wᵢⱼ Oᵢⱼ) / (ΣᵢΣⱼ wᵢⱼ Eᵢⱼ)

Linear weights:
  wᵢⱼ = |i−j| / (K−1)

Quadratic weights:
  wᵢⱼ = (i−j)² / (K−1)²

Where to get the values:
  K = number of scale levels
  O and E = from confusion matrix by ratings

from sklearn.metrics import cohen_kappa_score

k_linear = cohen_kappa_score(human, judge, weights="linear")
k_quad   = cohen_kappa_score(human, judge, weights="quadratic")

Practically:
  if the adjacent levels are almost uniform
  quadratic if long-range misses are particularly painful

Say:
  N = number of examples
  n = number of appraisers for example
  k = number of classes
  nij = how many appraisers gave example i class j

Agreement within Example i:
         1
  Pᵢ = ------- · Σⱼ nᵢⱼ(nᵢⱼ−1)
       n(n−1)

Average observed agreement:
  P̄ = (1/N) · Σᵢ Pᵢ

Share of class j over the pool:
           1
  pⱼ = -------- · Σᵢ nᵢⱼ
        N · n

Chance agreement:
  P̄ₑ = Σⱼ pⱼ²

  Fleiss κ = (P̄ − P̄ₑ) / (1 − P̄ₑ)

I need a table.
  rows = examples
  columns = raters
  values  = class label

Example:
  tc_001  [PASS, PASS, FAIL]
  tc_002  [FAIL, FAIL, FAIL]
  tc_003  [PASS, PASS, PASS]

It is considered nij for each example.

α = 1 − D_observed / D_expected

Where:
  D observed = actual disagreement between annotators
  D expected = Disagreement expected by chance

Ordinal labels are usually defined as distance:
  δ(c, c′) = (c − c′)²

Then great discrepancies are fined more.
as in quadratic weighted κ

If α → 1: almost complete consensus
If α → 0: no better than chance

We need an item × rater matrix:

[
  [1, 1, None, 0],
  [2, 2, 2,    2],
  [4, 3, 4,    None],
]

from krippendorff import alpha
alpha_value = alpha(reliability_data=data, level_of_measurement="ordinal")

You need two arrays of the same length:

  human_scores = [4, 2, 5, 3, ...]
  judge_scores = [5, 2, 4, 3, ...]

Pearson r is a linear link:
                 Σ (xᵢ−x̄)(yᵢ−ȳ)
  r = -----------------------------------
      sqrt(Σ(xᵢ−x̄)² · Σ(yᵢ−ȳ)²)

Spearman rho - Pearson by rank:
  ρ = Pearson(rank(x), rank(y))

If there are no ties, you can write like this:
           6 Σ dᵢ²
  ρ = 1 − ----------
         n(n²−1)

Kendall tau is a paired agreement:
      C − D
  τ = -------
      C + D

  C = concordant pairs
  D = discordant pairs

from scipy.stats import pearsonr, spearmanr, kendalltau

pearson_r,  _ = pearsonr(human_scores, judge_scores)
spearman_rho, _ = spearmanr(human_scores, judge_scores)
kendall_tau, _ = kendalltau(human_scores, judge_scores)

Human scores: Average human score or rank in each example
judge scores: score judge in the same examples

For the leaderboard:
  human rank = rank of the system n
  judge rank = rank of the same system n

Through TP, TN, FP, FN:

                 TP·TN − FP·FN
  MCC = -----------------------------------
        sqrt((TP+FP)(TP+FN)(TN+FP)(TN+FN))

Interpretation:
  -1: systematically wrong
   No better than chance.
  +1 - The perfect classifier

Practical sense:
  if unsafe only 5%,
  95% accuracy can be applied to any judge.
  MCC will immediately show the empty

First you fix the positive class:
  1 = unsafe, 0 = safe

Then on the same eval set:
  y_true = human / gold labels
  y_pred = hard verdict judge

from sklearn.metrics import matthews_corrcoef
mcc = matthews_corrcoef(y_true, y_pred)

If the judge gives you a probability:
  y_pred = 1[p_i ≥ threshold]

accuracy          = (TP + TN) / N
precision_pos     = TP / (TP + FP)
recall_pos        = TP / (TP + FN)
specificity       = TN / (TN + FP)
F1                = 2 · precision · recall / (precision + recall)
balanced_accuracy = (recall_pos + specificity) / 2

Where to get the values:
  TP, TN, FP, FN = from judge verdict vs gold label

An example of accuracy cheating:
  unsafe = 5%, judge always says SAFE
  accuracy = 95%
  recall_unsafe = 0%
  balanced_accuracy = 50%

from sklearn.metrics import accuracy_score, precision_recall_fscore_support
from sklearn.metrics import balanced_accuracy_score

acc  = accuracy_score(y_true, y_pred)
prec, rec, f1, _ = precision_recall_fscore_support(y_true, y_pred, average="binary")
bacc = balanced_accuracy_score(y_true, y_pred)

Need:

  y_true  = [0, 1, 0, 0, 1, ...]
  p_judge = [0.03, 0.91, 0.22, 0.10, 0.78, ...]

For any threshold t:
  TPR(t) = TP / (TP + FN)
  FPR(t) = FP / (FP + TN)
  Precision(t) = TP / (TP + FP)
  Recall(t)    = TP / (TP + FN)

ROC curve:
  x = FPR(t), y = TPR(t)
  ROC-AUC = area under this curve

PR curve:
  x = Recall(t), y = Precision(t)
  PR-AUC = area under this curve

Meaning:
  ROC-AUC Ranks Positives Above Negatives
  How accurate is PR-AUC in a rare positive class?

from sklearn.metrics import roc_auc_score, average_precision_score

roc_auc = roc_auc_score(y_true, p_judge)
pr_auc  = average_precision_score(y_true, p_judge)

Important:
  Probabilities / confidences are provided here.
  Not hard labels PASS/FAIL

If the judge gives confidence p:
  y_pred = 1[p ≥ t]

Then you think:
            judge=1   judge=0
human=1       TP        FN
human=0       FP        TN

The threshold t is not chosen for beauty.
a for the cost of errors FP and FN

Faithfulness eval:
  FN judge = missed hallucination
  FP judge = punished with correct answer

Moderation eval:
  FN judge = missed unsafe content
  FP judge = blocked harmless response

Same accuracy.
may be acceptable in one case.
and unacceptable elsewhere

For each example i run judge K times:
  vᵢ = [vᵢ₁, vᵢ₂, ..., vᵢₖ]

If verdict binary:
  c pass = number of PASS
  c fail = number of FAIL
  self_agreementᵢ = max(c_pass, c_fail) / K

If the verdict numeric:
  μᵢ = mean(vᵢ)
  σᵢ = std(vᵢ)
  MADᵢ = mean(|vᵢⱼ − μᵢ|)

Outcome on dataset:
  mean_self_agreement = mean(self_agreementᵢ)
  mean_sigma          = mean(σᵢ)

Same thing:
  prompt
  context
  answer
  rubric

You run Ks with the same judge.
And you only keep your verdict/confidence.
dataset-free

For example i:
  unstablei = 1, if unique(vi1 ... vik) > 1
             = 0, otherwise

                1
flip_rate = ----- · Σᵢ unstableᵢ
                N

If the verdict numeric:
  First, transfer it to the label.
  e.g. PASS if score ≥ 4

tc_014 → PASS, PASS, FAIL, PASS, FAIL
unstable_014 = 1

tc_015 → FAIL, FAIL, FAIL, FAIL, FAIL
unstable_015 = 0

If 14 out of 100 examples are unstable,
flip_rate = 0.14

Algorithm for score(A) − score(B):

for b in 1..10000:
  sample_idx = resample(example_ids, replace=True)
  delta_b = metric(A[sample_idx]) - metric(B[sample_idx])

CI_95 = percentile(delta, 2.5), percentile(delta, 97.5)

Interpretation:
  if 0 within CI
  Improvement is unconvincing

  if all CI is > 0
  A is statistically better than B on this dataset

Keyword: paired
Because both systems are evaluated on the same examples.

Each example requires an outcome for both systems:

example_id | score_A | score_B
tc_001     | 1       | 0
tc_002     | 1       | 1
tc_003     | 0       | 1

metric may be:
  accuracy
  pass_rate
  mean judge score
  win_rate

It's not tokens or answers that are being reassembled.
(a) Example indexes

Let the judge give confidence p . [0.1].

coverage(τ) =
  #(pᵢ ≥ τ) / N

selective_risk(τ) =
  errors in examples with pi ≥ τ / #(pi ≥ τ)

Meaning:
  increase the threshold τ
  Coverage is falling
  Selective risk should also fall.

High confidence:
  auto-score

Medium confidence:
  Consider, but mark as weak evidence

Low confidence / tie / unsure:
  send out

This is especially important for rating indeterminacy.
borderline examples

For binary event y   {0.1} and prediction p:

  Brier = mean((p − y)^2)

Example:
  Judge says 0.90, event happened → error 0. 01
  The judge says 0.90, the event did not happen → error 0. 81

Below.

If the judge has only hard labels without confidence,
Brier cannot be counted without a separate confidence head.

For each example i:
  pi = confidence judge that label = 1
  yi = real binary label from human/gold

Example:
  p = [0.91, 0.72, 0.10, ...]
  y = [1,    0,    0,    ...]

from sklearn.metrics import brier_score_loss
brier = brier_score_loss(y, p)

If the judge gives confidence 0-100,
split by 100 first

Divide predictions into M bins by confidence:

  Bₘ = { i : pᵢ ∈ ((m−1)/M, m/M] }

accuracy(Bₘ)   = (1 / |Bₘ|) · Σᵢ 1[ŷᵢ = yᵢ]
confidence(Bₘ) = (1 / |Bₘ|) · Σᵢ pᵢ

ECE = Σₘ (|Bₘ| / N) · |accuracy(Bₘ) − confidence(Bₘ)|

Interpretation:
  ECE ≈ 0 → well calibrated
  ECE Upgrades the Gap Between Confidence and Reality

Careful:
  ECE is sensitive to the binning scheme
  Therefore, it is useful to show both a schedule and a number.

ŷᵢ = predicted judge label
Pi = confidence in this particular label
yᵢ = gold / human label

Usually:
  M = 10 bins
  or equal-width,
  or equal-mass binning

1. Break predictions by confidence bins
2. For each bin count:
   mean_confidence
   empirical_accuracy
3. Draw dots:
   x = mean_confidence
   y = empirical_accuracy
4. Add diagonal y=x as ideal

Perfect line:
  confidence = empirical accuracy

Curve below diagonal:
  judge overconfident

The curve above the diagonal:
  judge underconfident

Separately useful to watch:
  coverage by bins
  borderline subset
  hard cases only

judge([A, B]) → verdict₁
judge([B, A]) → verdict₂

winrate(A first)  = wins_A_when_first / pairs_with_A_first
winrate(A second) = wins_A_when_second / pairs_with_A_second

position_gap = |winrate(A first) − winrate(A second)|

swap_robust_winrate(A) =
  0.5 · (winrate(A first) + winrate(A second))

For each pair of answers, you need two launches:
  first [A, B]
  then [B, A]

Save:
  winner
  confidence
  explanation

Large position gap
It depends on order, not just quality.

For pair i:
  len_deltaᵢ = tokens(longer) − tokens(shorter)
  pref longi = 1 if the judge chooses a longer answer
             = 0, otherwise

long_win_rate =
  Σ pref_longᵢ / N_pairs

You can still count:
  corr(length_delta, judge_score_delta)

Perfect:
  at matched-quality pairs
  long_win_rate ≈ 0.5

Best audit set:
  short and long answers,
  comparable in human quality

Then if the judge consistently chooses a long one,
It's a verbosity bias.
Not a real difference in quality.

Let's have a balanced pair:
  answer_X_family vs answer_Y_family

Then for Judge X:
  pref_X = wins_X_family / decisive_pairs

For cross-provider judge:
  pref_X_cross = wins_X_family_cross / decisive_pairs

self_enhancement_gap =
  pref_X − pref_X_cross

Big positive gap
Judge X overstates his family of models

Need:
  matched prompts
  A balanced set of responses from different families
  at least two judges from different providers

It's not just absolute wins.
Change of leader when changing judge

There are P equivalent prompt variants:
  prompt_1 ... prompt_P

score_drift =
  std(metric(prompt_1), ..., metric(prompt_P))

rank_drift =
  max_rank_shift across prompts

prompt_flip_rate =
  Examples where the verdict changes
  when changing the wording of rubric

Same dataset,
The same judge model,
Only word rubric changes

If the score drift is big,
Regression after prompt editing
Could be an artifact judge- and

Say:
  W_A = wins(A)
  W_B = wins(B)
  T   = ties
  U   = unsure
  N   = W_A + W_B + T + U

overall_win_share(A) = W_A / N
decisive_win_rate(A) = W_A / (W_A + W_B)
tie_rate             = T / N
unsure_rate          = U / N

If you want a significance test,
  H₀: decisive_win_rate(A) = 0.5
  binomtest(W_A, W_A + W_B, 0.5)

Each prompt judge returns one of:
  A_WINS
  B_WINS
  TIE
  UNSURE

Good practice:
  Keeping raw verdict and confidence
  Not just the final win rate.

Bradley-Terry:
           exp(β_A)
P(A>B) = ----------------------
         exp(β_A) + exp(β_B)

β A, β B = hidden forces of systems
Which are matched by pairwise wins

Elo:
  E_A = 1 / (1 + 10^((R_B−R_A)/400))
  R′_A = R_A + K · (S_A − E_A)

  S A = 1 on win, 0.5 on tie, 0 on loss

ranking score
bootstrap CI
Number of pairwise matches played
tie rate
position-bias audit

If the intervals overlap strongly,
The leaderboard is visually accurate.
what he really is

CHECLIST PUBLICATION OR INTERNAL REPORT

1. Task and rubric
2. Judge model, prompt version, decoding params
3. Eval dataset size, composition, class balance
4. How many human rates and what kind of human agreement
5. Judge-human agreement: κ / α / ρ / τ
6. Binary metrics: precision / recall / MCC / confusion matrix
7. Stability: repeat runs, flip rate, paired bootstrap CI
8. Calibration: Brier / ECE / reliability plot
9. Bias probes: position / verbosity / self-enhancement / prompt sensitivity
10. Ambiguous cases, ties, domains where judge fails

If half of those items are missing,
The result is better understood as exploratory.
Not as a production-read metric.