---
schema_version: "newruntime-agent-readable-v0.2"
type: "post"
stable_id: "post:langchain-reviewbench-review-agent-evals"
slug: "langchain-reviewbench-review-agent-evals"
title: "ReviewBench Turns Code Review Into An Agent Eval"
description: "LangChain's ReviewBench uses real PR review history to test whether code-review agents can recover substantive reviewer findings without flooding humans with noise."
retrieval_nugget: "LangChain's ReviewBench uses real PR review history to test whether code-review agents can recover substantive reviewer findings without flooding humans with noise. LangChain's ReviewBench is interesting because it treats code review as a concrete agent task, not as a vibe around \"AI catches bugs.\" The benchmark starts from real PR feedback in the LangSmith monorepo."
status: "published"
published_at: "2026-08-01"
updated_at: "2026-08-01"
record_date: "2026-08-01"
date_kind: "published_at"
topics: ["coding-agents","evals","verification","developer-tools"]
source_urls: ["https://x.com/LangChain/status/2083236117839499511","https://www.langchain.com/blog/towards-automating-eval-engineering","https://www.langchain.com/blog/unified-stack-for-evaluating-agents"]
visuals: [{"id":"langchain-reviewbench-review-agent-evals","kind":"editorial-diagram","role":"hero","src":"https://newruntime.com/images/posts/langchain-reviewbench-review-agent-evals.webp","alt":"Hand-drawn evaluation pipeline where real pull request review comments become curated issue cards, reproducible benchmark tasks, agent findings, and coverage plus precision scores.","caption":"ReviewBench moves code-review agents away from generic bug hunting and toward the exact failures trusted reviewers catch in real pull requests.","credit":"New Runtime synthesis from LangChain ReviewBench public article","source_url":"https://x.com/LangChain/status/2083236117839499511","generated_with":"gemini-3.1-flash-image","width":1600,"height":900,"legend":[{"label":"Real reviews","description":"Merged PR comments become candidate findings instead of synthetic bugs written from scratch."},{"label":"Curated tasks","description":"Substantive review comments are converted into reproducible Harbor tasks with frozen PR context."},{"label":"Review quality","description":"Agents are judged on coverage of baseline issues and precision of submitted findings."}]}]
routes: {"html":"https://newruntime.com/posts/langchain-reviewbench-review-agent-evals/","markdown":"https://newruntime.com/posts/langchain-reviewbench-review-agent-evals.md","json":"https://newruntime.com/posts/langchain-reviewbench-review-agent-evals.json"}
source_format: "markdown"
---

# ReviewBench Turns Code Review Into An Agent Eval

## Retrieval answer

LangChain's ReviewBench uses real PR review history to test whether code-review agents can recover substantive reviewer findings without flooding humans with noise. LangChain's ReviewBench is interesting because it treats code review as a concrete agent task, not as a vibe around "AI catches bugs." The benchmark starts from real PR feedback in the LangSmith monorepo.

LangChain's ReviewBench is interesting because it treats code review as a concrete agent task, not as a vibe around "AI catches bugs."

The benchmark starts from real PR feedback in the LangSmith monorepo. Review comments from trusted reviewers become candidate findings. Weak comments, nits, and questions are filtered out. The remaining issues are curated into concrete review defects, then packaged as Harbor tasks with frozen PR metadata, a seeded repository, and a verifier.

That makes the benchmark closer to the real review problem. One task can require noticing that a query deleted a resource by ID without checking the tenant. Another can require comparing an endpoint migration against the old API and catching a dropped filter. Those are not changed-line lint errors. They are implicit system contracts.

The scoring also matters. ReviewBench measures coverage and precision. Coverage asks whether the agent recovered the curated baseline issue in the same code path. Precision asks whether the submitted findings are actually supported by the code, including correct extra findings that were not in the baseline.

The reported baseline is humbling: with the same minimal Deep Agents harness, current models still miss most curated reviewer findings. The strongest runs recover about 30% of the baseline issues. The useful follow-up is that a structured review prompt substantially changes Luna's behavior on a 20-task slice. The same model does better when it is pushed to map the PR, trace dependent behavior, and validate findings against callers, tests, and related implementations.

For New Runtime, this connects directly to the existing review-bottleneck thread. Faster generation creates more code to own. ReviewBench gives that bottleneck an eval shape: not "did the agent comment," but "did it recover the kind of issue a trusted reviewer would have stopped."