LLM-as-Judge Meta-Evaluation

1. 100-200 examples: good/bad/borderline
2. Mark 2-3 people in the same category
3. The Inter-Human Agreement
4. Run the judge on the same set
5. Agreement Judge vs Human
6. Check stability, calibration and bias probes

If inter-human κ = 0.45 and judge-human κ = 0.43
The judge is close to the human ceiling.

If inter-human κ = 0.75 and judge-human κ = 0.38
Judge is weak, the problem is not the task, but the judge

from sklearn.metrics import cohen_kappa_score

human = [1, 0, 1, ...]
judge = [1, 1, 1, ...]

kappa = cohen_kappa_score(human, judge)
#human labels: from 2-3 experts or majority vote
# judge labels: from the same eval set, in the same order

We need two ordinal ratings:
human = [5, 3, 4, ...]
judge = [4, 3, 5, ...]

We construct observed matrix O(i,j) and expected matrix E(i,j):
  O = pair frequencies (human=i, judge=j)
  E = expected frequencies from marginals

  κ_w = 1 − (ΣᵢΣⱼ wᵢⱼ Oᵢⱼ) / (ΣᵢΣⱼ wᵢⱼ Eᵢⱼ)

Linear weights:
  wᵢⱼ = |i−j| / (K−1)

Quadratic weights:
  wᵢⱼ = (i−j)² / (K−1)²

Where to get the values:
  K = number of scale levels
  O and E = from confusion matrix by ratings

from sklearn.metrics import cohen_kappa_score

k_linear = cohen_kappa_score(human, judge, weights="linear")
k_quad   = cohen_kappa_score(human, judge, weights="quadratic")

Practically:
  if the adjacent levels are almost uniform
  quadratic if long-range misses are particularly painful

Say:
  N = number of examples
  n = number of appraisers for example
  k = number of classes
  nij = how many appraisers gave example i class j

Agreement within Example i:
         1
  Pᵢ = ------- · Σⱼ nᵢⱼ(nᵢⱼ−1)
       n(n−1)

Average observed agreement:
  P̄ = (1/N) · Σᵢ Pᵢ

Share of class j over the pool:
           1
  pⱼ = -------- · Σᵢ nᵢⱼ
        N · n

Chance agreement:
  P̄ₑ = Σⱼ pⱼ²

  Fleiss κ = (P̄ − P̄ₑ) / (1 − P̄ₑ)

I need a table.
  rows = examples
  columns = raters
  values  = class label

Example:
  tc_001  [PASS, PASS, FAIL]
  tc_002  [FAIL, FAIL, FAIL]
  tc_003  [PASS, PASS, PASS]

It is considered nij for each example.

α = 1 − D_observed / D_expected

Where:
  D observed = actual disagreement between annotators
  D expected = Disagreement expected by chance

Ordinal labels are usually defined as distance:
  δ(c, c′) = (c − c′)²

Then great discrepancies are fined more.
as in quadratic weighted κ

If α → 1: almost complete consensus
If α → 0: no better than chance

We need an item × rater matrix:

[
  [1, 1, None, 0],
  [2, 2, 2,    2],
  [4, 3, 4,    None],
]

from krippendorff import alpha
alpha_value = alpha(reliability_data=data, level_of_measurement="ordinal")

from scipy.stats import pearsonr, spearmanr, kendalltau

pearson_r,  _ = pearsonr(human_scores, judge_scores)
spearman_rho, _ = spearmanr(human_scores, judge_scores)
kendall_tau, _ = kendalltau(human_scores, judge_scores)

Human scores: average human score or rank for each example
Judge scores: judge score for the same examples

For the leaderboard:
  human rank = rank of the system
  judge rank = rank of the same system

First you fix the positive class:
  1 = unsafe, 0 = safe

Then on the same eval set:
  y_true = human / gold labels
  y_pred = hard verdict judge

from sklearn.metrics import matthews_corrcoef
mcc = matthews_corrcoef(y_true, y_pred)

If the judge gives you a probability:
  y_pred = 1[p_i ≥ threshold]

accuracy          = (TP + TN) / N
precision_pos     = TP / (TP + FP)
recall_pos        = TP / (TP + FN)
specificity       = TN / (TN + FP)
F1                = 2 · precision · recall / (precision + recall)
balanced_accuracy = (recall_pos + specificity) / 2

Where to get the values:
  TP, TN, FP, FN = from judge verdict vs gold label

An example of accuracy cheating:
  unsafe = 5%, judge always says SAFE
  accuracy = 95%
  recall_unsafe = 0%
  balanced_accuracy = 50%

from sklearn.metrics import accuracy_score, precision_recall_fscore_support
from sklearn.metrics import balanced_accuracy_score

acc  = accuracy_score(y_true, y_pred)
prec, rec, f1, _ = precision_recall_fscore_support(y_true, y_pred, average="binary")
bacc = balanced_accuracy_score(y_true, y_pred)

from sklearn.metrics import roc_auc_score, average_precision_score

roc_auc = roc_auc_score(y_true, p_judge)
pr_auc  = average_precision_score(y_true, p_judge)

Important:
  Probabilities / confidences are provided here.
  Not hard labels PASS/FAIL

If the judge gives confidence p:
  y_pred = 1[p ≥ t]

Then you think:
            judge=1   judge=0
human=1       TP        FN
human=0       FP        TN

The threshold t is not chosen for beauty.
a for the cost of errors FP and FN

Faithfulness eval:
  FN judge = missed hallucination
  FP judge = punished with correct answer

Moderation eval:
  FN judge = missed unsafe content
  FP judge = blocked harmless response

Same accuracy.
may be acceptable in one case.
and unacceptable elsewhere

For each example i run judge K times:
  vᵢ = [vᵢ₁, vᵢ₂, ..., vᵢₖ]

If verdict binary:
  c pass = number of PASS
  c fail = number of FAIL
  self_agreementᵢ = max(c_pass, c_fail) / K

If the verdict numeric:
  μᵢ = mean(vᵢ)
  σᵢ = std(vᵢ)
  MADᵢ = mean(|vᵢⱼ − μᵢ|)

Outcome on dataset:
  mean_self_agreement = mean(self_agreementᵢ)
  mean_sigma          = mean(σᵢ)

Same thing:
  prompt
  context
  answer
  rubric

You run Ks with the same judge.
And you only keep your verdict/confidence.
dataset-free

For example i:
  unstablei = 1, if unique(vi1 ... vik) > 1
             = 0, otherwise

                1
flip_rate = ----- · Σᵢ unstableᵢ
                N

If the verdict numeric:
  First, transfer it to the label.
  e.g. PASS if score ≥ 4

tc_014 → PASS, PASS, FAIL, PASS, FAIL
unstable_014 = 1

tc_015 → FAIL, FAIL, FAIL, FAIL, FAIL
unstable_015 = 0

If 14 out of 100 examples are unstable,
flip_rate = 0.14

Each example requires an outcome for both systems:

example_id | score_A | score_B
tc_001     | 1       | 0
tc_002     | 1       | 1
tc_003     | 0       | 1

metric may be:
  accuracy
  pass_rate
  mean judge score
  win_rate

It's not tokens or answers that are being reassembled.
(a) Example indexes

Let the judge give confidence p . [0.1].

coverage(τ) =
  #(pᵢ ≥ τ) / N

selective_risk(τ) =
  errors in examples with pi ≥ τ / #(pi ≥ τ)

Meaning:
  increase the threshold τ
  Coverage is falling
  Selective risk should also fall.

High confidence:
  auto-score

Medium confidence:
  Consider, but mark as weak evidence

Low confidence / tie / unsure:
  send out

This is especially important for rating indeterminacy.
borderline examples

For each example i:
  pi = confidence judge that label = 1
  yi = real binary label from human/gold

Example:
  p = [0.91, 0.72, 0.10, ...]
  y = [1,    0,    0,    ...]

from sklearn.metrics import brier_score_loss
brier = brier_score_loss(y, p)

If the judge gives confidence 0-100,
split by 100 first

ŷᵢ = predicted judge label
Pi = confidence in this particular label
yᵢ = gold / human label

Usually:
  M = 10 bins
  or equal-width,
  or equal-mass binning

1. Break predictions by confidence bins
2. For each bin count:
   mean_confidence
   empirical_accuracy
3. Draw dots:
   x = mean_confidence
   y = empirical_accuracy
4. Add diagonal y=x as ideal

Perfect line:
  confidence = empirical accuracy

Curve below diagonal:
  judge overconfident

The curve above the diagonal:
  judge underconfident

Separately useful to watch:
  coverage by bins
  borderline subset
  hard cases only

judge([A, B]) → verdict₁
judge([B, A]) → verdict₂

winrate(A first)  = wins_A_when_first / pairs_with_A_first
winrate(A second) = wins_A_when_second / pairs_with_A_second

position_gap = |winrate(A first) − winrate(A second)|

swap_robust_winrate(A) =
  0.5 · (winrate(A first) + winrate(A second))

For each pair of answers, you need two launches:
  first [A, B]
  then [B, A]

Save:
  winner
  confidence
  explanation

Large position gap
It depends on order, not just quality.

For pair i:
  len_deltaᵢ = tokens(longer) − tokens(shorter)
  pref longi = 1 if the judge chooses a longer answer
             = 0, otherwise

long_win_rate =
  Σ pref_longᵢ / N_pairs

You can still count:
  corr(length_delta, judge_score_delta)

Perfect:
  at matched-quality pairs
  long_win_rate ≈ 0.5

Best audit set:
  short and long answers,
  comparable in human quality

Then if the judge consistently chooses a long one,
It's a verbosity bias.
Not a real difference in quality.

Let's have a balanced pair:
  answer_X_family vs answer_Y_family

Then for Judge X:
  pref_X = wins_X_family / decisive_pairs

For cross-provider judge:
  pref_X_cross = wins_X_family_cross / decisive_pairs

self_enhancement_gap =
  pref_X − pref_X_cross

Big positive gap
Judge X overstates his family of models

Need:
  matched prompts
  A balanced set of responses from different families
  at least two judges from different providers

It's not just absolute wins.
Change of leader when changing judge

There are P equivalent prompt variants:
  prompt_1 ... prompt_P

score_drift =
  std(metric(prompt_1), ..., metric(prompt_P))

rank_drift =
  max_rank_shift across prompts

prompt_flip_rate =
  Examples where the verdict changes
  when changing the wording of rubric

Same dataset,
The same judge model,
Only word rubric changes

If the score drift is big,
Regression after prompt editing
Could be an artifact judge- and

Say:
  W_A = wins(A)
  W_B = wins(B)
  T   = ties
  U   = unsure
  N   = W_A + W_B + T + U

overall_win_share(A) = W_A / N
decisive_win_rate(A) = W_A / (W_A + W_B)
tie_rate             = T / N
unsure_rate          = U / N

If you want a significance test,
  H₀: decisive_win_rate(A) = 0.5
  binomtest(W_A, W_A + W_B, 0.5)

Each prompt judge returns one of:
  A_WINS
  B_WINS
  TIE
  UNSURE

Good practice:
  Keeping raw verdict and confidence
  Not just the final win rate.

Bradley-Terry:
           exp(β_A)
P(A>B) = ----------------------
         exp(β_A) + exp(β_B)

β A, β B = hidden forces of systems
Which are matched by pairwise wins

Elo:
  E_A = 1 / (1 + 10^((R_B−R_A)/400))
  R′_A = R_A + K · (S_A − E_A)

  S A = 1 on win, 0.5 on tie, 0 on loss

ranking score
bootstrap CI
Number of pairwise matches played
tie rate
position-bias audit

If the intervals overlap strongly,
The leaderboard is visually accurate.
what he really is