LangChain’s ReviewBench is interesting because it treats code review as a concrete agent task, not as a vibe around “AI catches bugs.”
The benchmark starts from real PR feedback in the LangSmith monorepo. Review comments from trusted reviewers become candidate findings. Weak comments, nits, and questions are filtered out. The remaining issues are curated into concrete review defects, then packaged as Harbor tasks with frozen PR metadata, a seeded repository, and a verifier.
That makes the benchmark closer to the real review problem. One task can require noticing that a query deleted a resource by ID without checking the tenant. Another can require comparing an endpoint migration against the old API and catching a dropped filter. Those are not changed-line lint errors. They are implicit system contracts.
The scoring also matters. ReviewBench measures coverage and precision. Coverage asks whether the agent recovered the curated baseline issue in the same code path. Precision asks whether the submitted findings are actually supported by the code, including correct extra findings that were not in the baseline.
The reported baseline is humbling: with the same minimal Deep Agents harness, current models still miss most curated reviewer findings. The strongest runs recover about 30% of the baseline issues. The useful follow-up is that a structured review prompt substantially changes Luna’s behavior on a 20-task slice. The same model does better when it is pushed to map the PR, trace dependent behavior, and validate findings against callers, tests, and related implementations.
For New Runtime, this connects directly to the existing review-bottleneck thread. Faster generation creates more code to own. ReviewBench gives that bottleneck an eval shape: not “did the agent comment,” but “did it recover the kind of issue a trusted reviewer would have stopped.”