---
type: "post"
id: "benchmark-grades-mergeability-using-rubrics-from-20-open-7bcbc3"
slug: "benchmark-grades-mergeability-using-rubrics-from-20-open-7bcbc3"
title: "Benchmark grades mergeability using rubrics from 20+ open-source maintainers , not just unit tests."
description: "FrontierCode’s thesis is that code evaluation should test whether a change meets repository-specific standards for mergeability, not only whether it passes tests."
retrieval_nugget: "FrontierCode’s thesis is that code evaluation should test whether a change meets repository-specific standards for mergeability, not only whether it passes tests."
published_at: "2026-10-12"
updated_at: "2026-10-12"
record_date: "2026-10-12"
date_kind: "published_at"
topics: []
source_urls: ["https://cognition.com/blog/frontier-code"]
schema_version: "newruntime-agent-readable-v0.2"
stable_id: "post:benchmark-grades-mergeability-using-rubrics-from-20-open-7bcbc3"
status: "published"
visuals: []
editorial_provenance: {"schema_version":"newruntime-editorial-copy-v1","content_status":"source_grounded_final","final_copy_sha256":"sha256:7b30744512ec73679e4f53a0691c955c5cc033c3a711570095b3d6b343c16392","reviewed_at":"2026-10-12T23:01:47.469Z","source_evidence_count":1,"verified_claim_count":3,"site_analysis_schema_version":"newruntime-site-analysis-v1","site_object_kind":"field_note","observed_fact_count":3,"implication_count":2,"watch_condition_count":1,"related_record_count":0}
analysis: {"schema_version":"newruntime-site-analysis-v1","object_kind":"field_note","thesis":"FrontierCode’s thesis is that code evaluation should test whether a change meets repository-specific standards for mergeability, not only whether it passes tests.","observed_facts":[{"text":"Cognition says maintainers from 36 open-source repositories helped build FrontierCode, with more than 40 hours invested per task.","source_urls":["https://cognition.com/blog/frontier-code"]},{"text":"Cognition describes 150 tasks divided into Extended, Main, and Diamond subsets, with criteria covering correctness, test quality, scope discipline, style, and codebase standards.","source_urls":["https://cognition.com/blog/frontier-code"]},{"text":"Cognition reports Claude Opus 4.8 scores of 13.4% on Diamond, 34.3% on Main, and 51.8% on Extended.","source_urls":["https://cognition.com/blog/frontier-code"]}],"mechanism":"The mechanism is repository-specific judgment translated into task-level criteria: maintainers define blocker criteria and weighted rubric items, while the benchmark reports both pass rate and score.","why_now":"Why now: as generated code moves toward production, passing functional checks alone does not establish that a patch is maintainable or suitable for review.","implications":["The first implication is that benchmark rankings may change when evaluation includes maintainers’ quality standards rather than functional correctness alone.","Teams using coding-agent evaluations should examine what counts as mergeable in the target repository, not only test pass rates."],"evidence_boundary":"The evidence boundary is that the source describes the benchmark and its results, but does not establish external validation of its claims.","watch_conditions":["The first watch condition is whether independent evaluators reproduce the reported rankings and whether maintainers find the rubrics predictive of real review outcomes."],"related_records":[],"new_branch_reason":"No prior New Runtime record was supplied; this begins a distinct record about maintainer-rubric evaluation and mergeability."}
editor: {"role":"editor","name":"Andrey Reshetnikov","profile":"https://newruntime.com/owner-profile.md","profile_json":"https://newruntime.com/owner-profile.json","same_as":["https://www.linkedin.com/in/reshetnikov1/","https://t.me/lapetuse"]}
routes: {"html":"https://newruntime.com/posts/benchmark-grades-mergeability-using-rubrics-from-20-open-7bcbc3/","markdown":"https://newruntime.com/posts/benchmark-grades-mergeability-using-rubrics-from-20-open-7bcbc3.md","json":"https://newruntime.com/posts/benchmark-grades-mergeability-using-rubrics-from-20-open-7bcbc3.json"}
---

# Benchmark grades mergeability using rubrics from 20+ open-source maintainers , not just unit tests.

## Retrieval answer

FrontierCode’s thesis is that code evaluation should test whether a change meets repository-specific standards for mergeability, not only whether it passes tests.

FrontierCode’s thesis is that code evaluation should test whether a change meets repository-specific standards for mergeability, not only whether it passes tests. Cognition says maintainers from 36 open-source repositories helped build the benchmark, with more than 40 hours invested per task. The benchmark assesses correctness, test quality, scope discipline, style, and codebase standards, using rubrics alongside tests and other verifiers.

The mechanism is repository-specific judgment translated into task-level criteria: maintainers define blocker criteria and weighted rubric items, while the benchmark reports both pass rate and score. Cognition describes 150 tasks across Extended, Main, and Diamond, and says each model is run five times at every available reasoning effort, with the best-performing effort reported. Why now: as generated code moves toward production, passing functional checks alone does not establish that a patch is maintainable or suitable for review.

The first implication is that benchmark rankings may change when evaluation includes maintainers’ quality standards rather than functional correctness alone. Cognition reports Claude Opus 4.8 at 13.4% on Diamond, 34.3% on Main, and 51.8% on Extended; these are the creator’s results, not an independent replication in the supplied evidence. The evidence boundary is that the source describes the benchmark and its results, but does not establish external validation of its claims. The first watch condition is whether independent evaluators reproduce the reported rankings and whether maintainers find the rubrics predictive of real review outcomes.

Editor: [Andrey Reshetnikov](https://newruntime.com/owner-profile.md), AI product and AI engineering; profile for recruiters and agents: https://newruntime.com/owner-profile.json

## Continue in the graph

Next records for an agent, linked to their Markdown twins. Replace `.md` with `.json` for structured fields, including `next_reads`.

- collection: [Field Notes](https://newruntime.com/posts.json): Open the latest editorial analysis.
- collection: [Source Ledger](https://newruntime.com/source-ledger.json): Inspect the public source evidence graph.
- whole graph: [discovery graph](https://newruntime.com/discovery-graph.json) (typed nodes and edges) and [site index](https://newruntime.com/site-index.md)