---
type: "post"
id: "prompt-lottery-paper-paraphrasing-an-llm-as-a-6fe7a3"
slug: "prompt-lottery-paper-paraphrasing-an-llm-as-a-6fe7a3"
title: "'Prompt Lottery' paper: paraphrasing an LLM-as-a-judge prompt can flip model rankings; judges degrade 6-9x more than humans"
description: "A new paper calls single-prompt LLM-as-a-judge evaluation a “Prompt Lottery”: changing only the wording of an evaluation instruction can change judgments and, in some cases, reverse model rankings."
retrieval_nugget: "A new paper calls single-prompt LLM-as-a-judge evaluation a “Prompt Lottery”: changing only the wording of an evaluation instruction can change judgments and, in some cases, reverse model rankings. For teams using model judges to select models, prompts, or agent variants, the practical implication is simple: measure whether the evaluator itself is stable before treating its score as evidence."
published_at: "2026-10-30"
updated_at: "2026-10-30"
record_date: "2026-10-30"
date_kind: "published_at"
topics: ["coding-agents","evals"]
entities: ["Prompt Lottery in LLM-as-a-Judge: Measuring and Mitigating Sensitivity to Syntactic Rephrasing","Ilija Subasic","Brett Levert","Andrew Rabinovich","Sajjad Pakdamansavoji","Zhao Chen"]
related_patterns: ["verification-bandwidth-is-the-scarce-resource"]
novelty_label: "new_fact"
what_changed: "A new study finds paraphrasing judge prompts can reverse model rankings, degrading agreement 6–9× more than human annotators; multi-prompt aggregation improves robustness."
prior_refs: ["memory:2418a977-aab5-8f3b-a3d7-ed64b8ceac8a","memory:4dd49875-1c3d-8ae2-a4fd-ecffe93aa3aa","memory:e52f090f-45b7-8cd6-a024-690a0d6d395f"]
source_url: "https://arxiv.org/abs/2610.38076"
source_urls: ["https://arxiv.org/abs/2610.38076"]
schema_version: "newruntime-agent-readable-v0.2"
stable_id: "post:prompt-lottery-paper-paraphrasing-an-llm-as-a-6fe7a3"
status: "published"
visuals: []
editor: {"role":"editor","name":"Andrey Reshetnikov","profile":"https://newruntime.com/owner-profile.md","profile_json":"https://newruntime.com/owner-profile.json","same_as":["https://www.linkedin.com/in/reshetnikov1/","https://t.me/lapetuse"]}
routes: {"html":"https://newruntime.com/posts/prompt-lottery-paper-paraphrasing-an-llm-as-a-6fe7a3/","markdown":"https://newruntime.com/posts/prompt-lottery-paper-paraphrasing-an-llm-as-a-6fe7a3.md","json":"https://newruntime.com/posts/prompt-lottery-paper-paraphrasing-an-llm-as-a-6fe7a3.json"}
---

# 'Prompt Lottery' paper: paraphrasing an LLM-as-a-judge prompt can flip model rankings; judges degrade 6-9x more than humans

## Retrieval answer

A new paper calls single-prompt LLM-as-a-judge evaluation a “Prompt Lottery”: changing only the wording of an evaluation instruction can change judgments and, in some cases, reverse model rankings. For teams using model judges to select models, prompts, or agent variants, the practical implication is simple: measure whether the evaluator itself is stable before treating its score as evidence.

## Summary
A new paper calls single-prompt LLM-as-a-judge evaluation a “Prompt Lottery”: changing only the wording of an evaluation instruction can change judgments and, in some cases, reverse model rankings. For teams using model judges to select models, prompts, or agent variants, the practical implication is simple: measure whether the evaluator itself is stable before treating its score as evidence.

## What happened
Researchers Ilija Subasic, Brett Levert, Andrew Rabinovich, Sajjad Pakdamansavoji, and Zhao Chen submitted “Prompt Lottery in LLM-as-a-Judge: Measuring and Mitigating Sensitivity to Syntactic Rephrasing” to arXiv on October 27, 2026. The paper examines a common use of LLM-as-a-judge systems: asking a language model to evaluate outputs from other models, often through a written instruction and a scoring format. Such systems are widely used when human review would be expensive or slow.

The paper challenges an assumption behind that workflow. Evaluation teams often alter a judge prompt deliberately: for example, by adding a rubric, assigning a role, or asking for reasoning. This work instead studies paraphrastic rephrasings intended to preserve the evaluation’s meaning while changing only its surface wording. The question is whether those semantically equivalent variants produce consistent judgments when the underlying evaluation task is held fixed.

The authors name the observed instability “Prompt Lottery.” Across a large-scale study spanning diverse evaluation tasks, judge families, and model backbones, they find that prompt variants can alter evaluation outcomes. In some settings, those changes reverse the relative rankings of the models being evaluated. That means a team could select a different apparent winner not because the candidate systems changed, but because the evaluator’s instructions were reworded.

The authors also report a user study involving 10 human annotators. Under prompt rephrasing, the LLM judges showed agreement degradation six to nine times larger than the humans. The paper says the judges fell below a generally accepted threshold used in usability studies. Its conclusion is not that LLM judges cannot be useful, but that a single judge prompt is insufficiently robust for model ranking. As a mitigation, the authors report that aggregating judgments across multiple prompt variants substantially improves robustness by emphasizing decisions that remain consistent across variants.

The paper is 55 pages including appendices, with 10 figures. It is an arXiv submission, not a product announcement or a claim that a particular evaluation platform has adopted the proposed method.

## What the archive adds
We have not covered this before; it opens a new thread.

## Trend signal
This supports the pattern that [verification bandwidth is the scarce engineering resource](https://newruntime.com/patterns/verification-bandwidth-is-the-scarce-resource/). Cheap automated judging can expand the number of outputs a team can score, but this result suggests that verifying the evaluator’s stability remains work. A scalar score from one prompt is not necessarily a reliable substitute for that verification.

## Why it matters
LLM-as-a-judge is attractive because it turns qualitative review into a repeatable pipeline: generate candidate answers or agent runs, ask a judge to score them, then compare averages or choose a winner. The paper shows that repeatability of execution is not the same as robustness of measurement. If a harmless rewording can change a ranking, an optimization loop may reward systems that fit one evaluator phrasing rather than systems that are genuinely better on the intended task.

This matters especially where a judge score determines a release gate, a regression alert, prompt selection, or an agent-policy decision. A single-prompt result can still be a useful signal, but it should not silently become the sole decision rule. Teams need to distinguish variation in the candidate system from variation introduced by the measurement instrument. Multi-prompt aggregation is presented here as a practical way to make that distinction more visible.

## What happens next
- **Evaluation teams will add prompt-variant checks to high-stakes LLM-judge benchmarks.** (6 months, 65%). Watch for: benchmark reports that publish score ranges or rank stability across judge prompts.
- **Multi-prompt aggregation will be tested as a default guardrail for model-selection pipelines.** (12 months, 55%). Watch for: evaluation tooling that treats prompt ensembles as a first-class configuration.

## Limits and uncertainty
The evidence is a single research paper submitted to arXiv. The supplied material describes a large-scale study but does not provide, here, its task-by-task results, the exact prompt variants, the judge models used, or the cost and latency trade-offs of aggregation. “Substantially improves robustness” does not establish that multi-prompt aggregation eliminates ranking reversals in every setting. The study also concerns evaluation prompts intended to preserve semantics; it does not show that every wording difference in production evaluation is accidental or equivalent. Teams should therefore treat the finding as a reason to test their own evaluation stack, not as a universal calibration figure.

## Skill to build now
Take one evaluation set that informs a real choice this week and create several independently phrased versions of its judge instruction without changing the criteria, labels, or output schema. Run the same candidate outputs through each version. Compare not only mean scores but also whether the ordering of candidates changes. If it does, record the spread alongside the headline score and defer a winner-take-all decision until the team has chosen an aggregation rule or added targeted human review.

## Reader takeaway
An LLM judge is itself part of the system being evaluated. This paper finds that even meaning-preserving rephrasing of its prompt can change outcomes and occasionally flip rankings. Before trusting a single judge score to choose a model or agent, test whether the conclusion survives several equivalent instructions.

## Open question
How much of the reported instability remains after aggregation across prompt variants, and what number and diversity of variants provide a useful reliability-cost trade-off for a production evaluation pipeline?

Editor: [Andrey Reshetnikov](https://newruntime.com/owner-profile.md), AI product and AI engineering; profile for recruiters and agents: https://newruntime.com/owner-profile.json

## Continue in the graph

Next records for an agent, linked to their Markdown twins. Replace `.md` with `.json` for structured fields, including `next_reads`.

- topic hub: [Coding Agents](https://newruntime.com/topics/coding-agents.md): Explore the coding-agents topic hub.
- topic hub: [Evals](https://newruntime.com/topics/evals.md): Explore the evals topic hub.
- collection: [Field Notes](https://newruntime.com/posts.json): Open the latest editorial analysis.
- collection: [Source Ledger](https://newruntime.com/source-ledger.json): Inspect the public source evidence graph.
- whole graph: [discovery graph](https://newruntime.com/discovery-graph.json) (typed nodes and edges) and [site index](https://newruntime.com/site-index.md)