{"type":"post","id":"benchmark-grades-mergeability-using-rubrics-from-20-open-7bcbc3","slug":"benchmark-grades-mergeability-using-rubrics-from-20-open-7bcbc3","title":"Benchmark grades mergeability using rubrics from 20+ open-source maintainers , not just unit tests.","description":"FrontierCode’s thesis is that code evaluation should test whether a change meets repository-specific standards for mergeability, not only whether it passes tests.","retrieval_nugget":"FrontierCode’s thesis is that code evaluation should test whether a change meets repository-specific standards for mergeability, not only whether it passes tests.","published_at":"2026-10-12","updated_at":"2026-10-12","record_date":"2026-10-12","date_kind":"published_at","topics":[],"source_urls":["https://cognition.com/blog/frontier-code"],"schema_version":"newruntime-agent-readable-v0.2","stable_id":"post:benchmark-grades-mergeability-using-rubrics-from-20-open-7bcbc3","status":"published","visuals":[],"editorial_provenance":{"schema_version":"newruntime-editorial-copy-v1","content_status":"source_grounded_final","final_copy_sha256":"sha256:7b30744512ec73679e4f53a0691c955c5cc033c3a711570095b3d6b343c16392","reviewed_at":"2026-10-12T23:01:47.469Z","source_evidence_count":1,"verified_claim_count":3,"site_analysis_schema_version":"newruntime-site-analysis-v1","site_object_kind":"field_note","observed_fact_count":3,"implication_count":2,"watch_condition_count":1,"related_record_count":0},"analysis":{"schema_version":"newruntime-site-analysis-v1","object_kind":"field_note","thesis":"FrontierCode’s thesis is that code evaluation should test whether a change meets repository-specific standards for mergeability, not only whether it passes tests.","observed_facts":[{"text":"Cognition says maintainers from 36 open-source repositories helped build FrontierCode, with more than 40 hours invested per task.","source_urls":["https://cognition.com/blog/frontier-code"]},{"text":"Cognition describes 150 tasks divided into Extended, Main, and Diamond subsets, with criteria covering correctness, test quality, scope discipline, style, and codebase standards.","source_urls":["https://cognition.com/blog/frontier-code"]},{"text":"Cognition reports Claude Opus 4.8 scores of 13.4% on Diamond, 34.3% on Main, and 51.8% on Extended.","source_urls":["https://cognition.com/blog/frontier-code"]}],"mechanism":"The mechanism is repository-specific judgment translated into task-level criteria: maintainers define blocker criteria and weighted rubric items, while the benchmark reports both pass rate and score.","why_now":"Why now: as generated code moves toward production, passing functional checks alone does not establish that a patch is maintainable or suitable for review.","implications":["The first implication is that benchmark rankings may change when evaluation includes maintainers’ quality standards rather than functional correctness alone.","Teams using coding-agent evaluations should examine what counts as mergeable in the target repository, not only test pass rates."],"evidence_boundary":"The evidence boundary is that the source describes the benchmark and its results, but does not establish external validation of its claims.","watch_conditions":["The first watch condition is whether independent evaluators reproduce the reported rankings and whether maintainers find the rubrics predictive of real review outcomes."],"related_records":[],"new_branch_reason":"No prior New Runtime record was supplied; this begins a distinct record about maintainer-rubric evaluation and mergeability."},"editor":{"role":"editor","name":"Andrey Reshetnikov","profile":"https://newruntime.com/owner-profile.md","profile_json":"https://newruntime.com/owner-profile.json","same_as":["https://www.linkedin.com/in/reshetnikov1/","https://t.me/lapetuse"]},"routes":{"html":"https://newruntime.com/posts/benchmark-grades-mergeability-using-rubrics-from-20-open-7bcbc3/","markdown":"https://newruntime.com/posts/benchmark-grades-mergeability-using-rubrics-from-20-open-7bcbc3.md","json":"https://newruntime.com/posts/benchmark-grades-mergeability-using-rubrics-from-20-open-7bcbc3.json"},"next_reads":[{"type":"collection","path":"/posts/","url":"https://newruntime.com/posts/","title":"Field Notes","reason":"Open the latest editorial analysis.","media_type":"text/html","json_url":"https://newruntime.com/posts.json"},{"type":"collection","path":"/source-ledger.json","url":"https://newruntime.com/source-ledger.json","title":"Source Ledger","reason":"Inspect the public source evidence graph.","media_type":"application/json"}]}