What happened
Researchers Ilija Subasic, Brett Levert, Andrew Rabinovich, Sajjad Pakdamansavoji, and Zhao Chen submitted “Prompt Lottery in LLM-as-a-Judge: Measuring and Mitigating Sensitivity to Syntactic Rephrasing” to arXiv on October 27, 2026. The paper examines a common use of LLM-as-a-judge systems: asking a language model to evaluate outputs from other models, often through a written instruction and a scoring format. Such systems are widely used when human review would be expensive or slow.
The paper challenges an assumption behind that workflow. Evaluation teams often alter a judge prompt deliberately: for example, by adding a rubric, assigning a role, or asking for reasoning. This work instead studies paraphrastic rephrasings intended to preserve the evaluation’s meaning while changing only its surface wording. The question is whether those semantically equivalent variants produce consistent judgments when the underlying evaluation task is held fixed.
The authors name the observed instability “Prompt Lottery.” Across a large-scale study spanning diverse evaluation tasks, judge families, and model backbones, they find that prompt variants can alter evaluation outcomes. In some settings, those changes reverse the relative rankings of the models being evaluated. That means a team could select a different apparent winner not because the candidate systems changed, but because the evaluator’s instructions were reworded.
The authors also report a user study involving 10 human annotators. Under prompt rephrasing, the LLM judges showed agreement degradation six to nine times larger than the humans. The paper says the judges fell below a generally accepted threshold used in usability studies. Its conclusion is not that LLM judges cannot be useful, but that a single judge prompt is insufficiently robust for model ranking. As a mitigation, the authors report that aggregating judgments across multiple prompt variants substantially improves robustness by emphasizing decisions that remain consistent across variants.
The paper is 55 pages including appendices, with 10 figures. It is an arXiv submission, not a product announcement or a claim that a particular evaluation platform has adopted the proposed method.
What the archive adds
We have not covered this before; it opens a new thread.
Trend signal
This supports the pattern that verification bandwidth is the scarce engineering resource. Cheap automated judging can expand the number of outputs a team can score, but this result suggests that verifying the evaluator’s stability remains work. A scalar score from one prompt is not necessarily a reliable substitute for that verification.
Why it matters
LLM-as-a-judge is attractive because it turns qualitative review into a repeatable pipeline: generate candidate answers or agent runs, ask a judge to score them, then compare averages or choose a winner. The paper shows that repeatability of execution is not the same as robustness of measurement. If a harmless rewording can change a ranking, an optimization loop may reward systems that fit one evaluator phrasing rather than systems that are genuinely better on the intended task.
This matters especially where a judge score determines a release gate, a regression alert, prompt selection, or an agent-policy decision. A single-prompt result can still be a useful signal, but it should not silently become the sole decision rule. Teams need to distinguish variation in the candidate system from variation introduced by the measurement instrument. Multi-prompt aggregation is presented here as a practical way to make that distinction more visible.
What happens next
- Evaluation teams will add prompt-variant checks to high-stakes LLM-judge benchmarks. (6 months, 65%). Watch for: benchmark reports that publish score ranges or rank stability across judge prompts.
- Multi-prompt aggregation will be tested as a default guardrail for model-selection pipelines. (12 months, 55%). Watch for: evaluation tooling that treats prompt ensembles as a first-class configuration.
Limits and uncertainty
The evidence is a single research paper submitted to arXiv. The supplied material describes a large-scale study but does not provide, here, its task-by-task results, the exact prompt variants, the judge models used, or the cost and latency trade-offs of aggregation. “Substantially improves robustness” does not establish that multi-prompt aggregation eliminates ranking reversals in every setting. The study also concerns evaluation prompts intended to preserve semantics; it does not show that every wording difference in production evaluation is accidental or equivalent. Teams should therefore treat the finding as a reason to test their own evaluation stack, not as a universal calibration figure.
Skill to build now
Take one evaluation set that informs a real choice this week and create several independently phrased versions of its judge instruction without changing the criteria, labels, or output schema. Run the same candidate outputs through each version. Compare not only mean scores but also whether the ordering of candidates changes. If it does, record the spread alongside the headline score and defer a winner-take-all decision until the team has chosen an aggregation rule or added targeted human review.
Reader takeaway
An LLM judge is itself part of the system being evaluated. This paper finds that even meaning-preserving rephrasing of its prompt can change outcomes and occasionally flip rankings. Before trusting a single judge score to choose a model or agent, test whether the conclusion survives several equivalent instructions.
Open question
How much of the reported instability remains after aggregation across prompt variants, and what number and diversity of variants provide a useful reliability-cost trade-off for a production evaluation pipeline?