The company says the benchmark was built with maintainers of 36 open-source repositories, with more than 40 hours invested per task. Its criteria cover correctness, test quality, scope discipline, style, and repository-specific standards. Cognition describes 150 tasks across Extended, Main, and Diamond subsets. It reports Claude Opus 4.8 scores of 13.4% on Diamond, 34.3% on Main, and 51.8% on Extended. These are results from the benchmark’s creator; the supplied source does not provide an independent replication.
Raw SignalOct 12, 20261 min read1 source
Benchmark grades mergeability using rubrics from 20+ open-source maintainers , not just unit tests.
In one minute
Cognition has introduced FrontierCode, a benchmark intended to measure whether coding models produce changes maintainers would consider mergeable, not merely code that passes tests.