{"type":"post","id":"prompt-lottery-paper-paraphrasing-an-llm-as-a-6fe7a3","slug":"prompt-lottery-paper-paraphrasing-an-llm-as-a-6fe7a3","title":"'Prompt Lottery' paper: paraphrasing an LLM-as-a-judge prompt can flip model rankings; judges degrade 6-9x more than humans","description":"A new paper calls single-prompt LLM-as-a-judge evaluation a “Prompt Lottery”: changing only the wording of an evaluation instruction can change judgments and, in some cases, reverse model rankings.","retrieval_nugget":"A new paper calls single-prompt LLM-as-a-judge evaluation a “Prompt Lottery”: changing only the wording of an evaluation instruction can change judgments and, in some cases, reverse model rankings. For teams using model judges to select models, prompts, or agent variants, the practical implication is simple: measure whether the evaluator itself is stable before treating its score as evidence.","published_at":"2026-10-30","updated_at":"2026-10-30","record_date":"2026-10-30","date_kind":"published_at","topics":["coding-agents","evals"],"entities":["Prompt Lottery in LLM-as-a-Judge: Measuring and Mitigating Sensitivity to Syntactic Rephrasing","Ilija Subasic","Brett Levert","Andrew Rabinovich","Sajjad Pakdamansavoji","Zhao Chen"],"related_patterns":["verification-bandwidth-is-the-scarce-resource"],"novelty_label":"new_fact","what_changed":"A new study finds paraphrasing judge prompts can reverse model rankings, degrading agreement 6–9× more than human annotators; multi-prompt aggregation improves robustness.","prior_refs":["memory:2418a977-aab5-8f3b-a3d7-ed64b8ceac8a","memory:4dd49875-1c3d-8ae2-a4fd-ecffe93aa3aa","memory:e52f090f-45b7-8cd6-a024-690a0d6d395f"],"source_url":"https://arxiv.org/abs/2610.38076","source_urls":["https://arxiv.org/abs/2610.38076"],"schema_version":"newruntime-agent-readable-v0.2","stable_id":"post:prompt-lottery-paper-paraphrasing-an-llm-as-a-6fe7a3","status":"published","visuals":[],"editor":{"role":"editor","name":"Andrey Reshetnikov","profile":"https://newruntime.com/owner-profile.md","profile_json":"https://newruntime.com/owner-profile.json","same_as":["https://www.linkedin.com/in/reshetnikov1/","https://t.me/lapetuse"]},"routes":{"html":"https://newruntime.com/posts/prompt-lottery-paper-paraphrasing-an-llm-as-a-6fe7a3/","markdown":"https://newruntime.com/posts/prompt-lottery-paper-paraphrasing-an-llm-as-a-6fe7a3.md","json":"https://newruntime.com/posts/prompt-lottery-paper-paraphrasing-an-llm-as-a-6fe7a3.json"},"next_reads":[{"type":"topic","path":"/topics/coding-agents/","url":"https://newruntime.com/topics/coding-agents/","title":"Coding Agents","reason":"Explore the coding-agents topic hub.","media_type":"text/html","markdown_url":"https://newruntime.com/topics/coding-agents.md","json_url":"https://newruntime.com/topics/coding-agents.json"},{"type":"topic","path":"/topics/evals/","url":"https://newruntime.com/topics/evals/","title":"Evals","reason":"Explore the evals topic hub.","media_type":"text/html","markdown_url":"https://newruntime.com/topics/evals.md","json_url":"https://newruntime.com/topics/evals.json"},{"type":"collection","path":"/posts/","url":"https://newruntime.com/posts/","title":"Field Notes","reason":"Open the latest editorial analysis.","media_type":"text/html","json_url":"https://newruntime.com/posts.json"},{"type":"collection","path":"/source-ledger.json","url":"https://newruntime.com/source-ledger.json","title":"Source Ledger","reason":"Inspect the public source evidence graph.","media_type":"application/json"}]}