---
type: "post"
id: "building-llm-judges-to-judge-llms-2999c9"
slug: "building-llm-judges-to-judge-llms-2999c9"
title: "Building LLM Judges to Judge LLMs"
description: "Ramp says it replaced test-based grading in Ramp SWE-Bench with a pipeline of LLM judges that compare candidate patches with engineer-reviewed reference patches."
retrieval_nugget: "Ramp says it replaced test-based grading in Ramp SWE-Bench with a pipeline of LLM judges that compare candidate patches with engineer-reviewed reference patches."
published_at: "2026-11-21"
updated_at: "2026-11-21"
record_date: "2026-11-21"
date_kind: "published_at"
topics: ["evals","models"]
entities: ["Ramp","Ramp SWE-Bench","Inspect","Router"]
related_patterns: ["verification-bandwidth-is-the-scarce-resource","ai-engineering-evaluations-are-splitting-into-task-specific"]
novelty_label: "new_fact"
what_changed: null
prior_refs: []
source_url: "https://builders.ramp.com/post/building-llm-judges-to-judge-llms"
source_urls: ["https://builders.ramp.com/post/building-llm-judges-to-judge-llms"]
schema_version: "newruntime-agent-readable-v0.2"
stable_id: "post:building-llm-judges-to-judge-llms-2999c9"
status: "published"
visuals: []
editor: {"role":"editor","name":"Andrey Reshetnikov","profile":"https://newruntime.com/owner-profile.md","profile_json":"https://newruntime.com/owner-profile.json","same_as":["https://www.linkedin.com/in/reshetnikov1/","https://t.me/lapetuse"]}
routes: {"html":"https://newruntime.com/posts/building-llm-judges-to-judge-llms-2999c9/","markdown":"https://newruntime.com/posts/building-llm-judges-to-judge-llms-2999c9.md","json":"https://newruntime.com/posts/building-llm-judges-to-judge-llms-2999c9.json"}
markdown_url: "https://newruntime.com/posts/building-llm-judges-to-judge-llms-2999c9.md"
related_shifts: ["verification-ownership"]
---

# Building LLM Judges to Judge LLMs

## Retrieval answer

Ramp says it replaced test-based grading in Ramp SWE-Bench with a pipeline of LLM judges that compare candidate patches with engineer-reviewed reference patches.

## Summary
Ramp says it replaced test-based grading in Ramp SWE-Bench with a pipeline of LLM judges that compare candidate patches with engineer-reviewed reference patches. In its reported sample, deliberation raised agreement between two judges from 54% to 94%, while engineers agreed with nearly 90% of task-level verdicts. The account matters to teams evaluating AI-written code because it shows both a scalable approach and the need for human validation.

## What happened
Ramp’s Builders Blog describes the grading change in Ramp SWE-Bench V2, which it launched alongside a benchmark built from real engineering tasks. The tasks come from a subset of Inspect sessions that ended in merged pull requests. Ramp uses the code that passed engineer review and reached production as a reference, or “gold patch.” Its earlier approach graded new solutions against tests associated with that patch. Ramp says those tests could be overspecified, rejecting valid solutions that used a different implementation, or underspecified, allowing solutions that missed some requested behavior to pass. Ramp says V1 required labor-intensive auditing and repairs of test suites, and that filtering restrictive tests left more room for incomplete solutions to slip through.

The new approach asks whether a candidate patch implements the same requested behavior as the gold patch. The judge receives the task prompt, relevant repository context, and both patches. Ramp says a one-shot question about whether a solution is correct was not reliable enough: judge models disagreed, and their overall pass rates varied substantially. In a test using seven judge models on solutions from 20 solver models across 72 V1 tasks, the judges generally agreed with tests about which solvers were stronger, with a reported correlation of 0.85 using a majority vote across judges. But individual judges assigned markedly different pass rates: Opus 4.8 passed nearly 60% of comparisons, while Sol 5.6 and Luna 5.6 xhigh passed about 25%.

Ramp’s pipeline separates finding differences from deciding whether they matter. It extracts differences between the gold and candidate patches, deduplicates and consolidates them, has judges assess each difference, resolves disagreements through deliberation, and produces a verdict. Ramp says clarifying prompts and adding task-specific guidance did little to reduce disagreement; the latter also risked overfitting and did not scale. In its pipeline test, the company reports 360 comparisons across 72 tasks and 20 solver models. For Sol and Opus, agreement rose from 195 of 360 judgments (54.2%) in the original panel to 275 (76.4%) on the new pipeline’s first vote, then to 337 (93.6%) after deliberation.

Agreement is not the same as correctness, and Ramp reports two checks on that distinction. Engineers behind the original pull requests reviewed task-level verdicts and agreed with nearly 90% of them. Ramp also says it found no evidence that a candidate model was rewarded or penalized for sharing a provider with the model that produced the gold patch. The company presents the resulting leaderboard as Ramp SWE-Bench V2, with 500 high-effort pass@1 runs using native harnesses in the November 17, 2026 snapshot. Its post says V2 has six times as many tasks as V1 and evaluates models on native harnesses as well as a common harness.

## What the archive adds
New Runtime’s earlier coverage described Ramp’s Laminar agent harness and a separate credit-analysis case study: [Building Laminar: Ramp’s Managed Agents Platform](https://newruntime.com/posts/building-laminar-ramp-s-managed-agents-platform-c45176/) (October 24, 2026) and [Agentic Credit Analysis: Ramp Builders](https://newruntime.com/posts/agentic-credit-analysis-ramp-builders-651933/) (October 23, 2026). This is a new event from the same company, not a continuation of either story: it adds a detailed account of how Ramp evaluates code produced by AI agents.

## Trend signal
This does not yet fit a pattern New Runtime tracks. Ramp’s account nevertheless makes a narrow, useful claim: evaluation capacity has to be designed when AI systems produce more code than people can inspect through conventional test suites alone. Its method turns review into discrete work—identify differences, assess their relevance, and check verdicts against engineers—rather than treating a judge score as self-validating.

## Why now
Ramp ties the work to its launch of SWE-Bench V2 and says it spends millions of dollars on tokens to produce and review code internally and through Router. It frames reliable grading as necessary to understand how models perform on coding tasks its engineers delegate and to guide that spending. The timing is therefore linked to scaling its own evaluation and code production, not to a general change in access to LLM judges.

## Why it matters
Teams using coding agents need evaluations that distinguish an implementation that merely passes a test suite from one that meets the request. Ramp’s account shows why reference tests can fail in both directions: they can encode one implementation too tightly, or omit requirements that matter. Comparing patches against an engineer-reviewed reference may surface behavioral gaps, while splitting comparison into explicit differences gives reviewers and judges something more inspectable than a single correctness score.

The numbers also argue against treating an LLM judge’s score as an objective measurement. Different models can produce sharply different pass rates on the same work, and even a process that reaches 94% inter-judge agreement still needs an external check. For teams, the useful lesson is to treat judge agreement, alignment with human review, and the underlying task quality as separate questions. A benchmark can inform model and harness choices only to the extent that its grading process measures the intended work rather than the quirks of its tests or judges.

## Limits and uncertainty
Ramp’s figures come from its own tasks, judges, and benchmark setup; the post does not establish that the same agreement or engineer-alignment rates will hold on other repositories, programming languages, task types, or teams. The human check is described as small-scale, and the post does not give a broader sample size or uncertainty intervals for the nearly 90% agreement figure. Inter-judge agreement measures consistency, not truth, while agreement with engineers who authored the original work is not an independent external audit. The source also does not provide enough detail to reproduce the full prompts, deliberation protocol, or operating cost of the pipeline. Its claims about provider bias are limited to the tests Ramp reports.

## Skill to build now
For an internal coding-agent evaluation, take a small set of reviewed changes and compare each agent patch with the accepted patch. Ask reviewers to enumerate behavioral differences before rating whether each one changes fulfillment of the task. Keep a sample of disagreements for human adjudication, then compare judge agreement with those human decisions rather than relying on agreement alone. This adapts the structure Ramp describes without assuming its reported rates will transfer to your codebase.

## Reader takeaway
A judge pipeline can make code evaluation more flexible than brittle pass/fail tests, but agreement is only a consistency signal. Separate patch differences from their task relevance, and retain human checks so a repeatable verdict is not mistaken for a correct one.

Editor: [Andrey Reshetnikov](https://newruntime.com/owner-profile.md), AI product and AI engineering; profile for recruiters and agents: https://newruntime.com/owner-profile.json

## Continue in the graph

Next records for an agent, linked to their Markdown twins. Replace `.md` with `.json` for structured fields, including `next_reads`.

- topic hub: [Evals](https://newruntime.com/topics/evals.md): Explore the evals topic hub.
- topic hub: [Models](https://newruntime.com/topics/models.md): Explore the models topic hub.
- collection: [Field Notes](https://newruntime.com/posts.json): Open the latest editorial analysis.
- collection: [Sources](https://newruntime.com/sources/): See the original sources this archive cites.
- whole graph: [discovery graph](https://newruntime.com/discovery-graph.json) (typed nodes and edges) and [site index](https://newruntime.com/site-index.md)
