Cognition says maintainers from 36 open-source repositories helped build the benchmark, with more than 40 hours invested per task. The benchmark assesses correctness, test quality, scope discipline, style, and codebase standards, using rubrics alongside tests and other verifiers.
The mechanism is repository-specific judgment translated into task-level criteria: maintainers define blocker criteria and weighted rubric items, while the benchmark reports both pass rate and score. Cognition describes 150 tasks across Extended, Main, and Diamond, and says each model is run five times at every available reasoning effort, with the best-performing effort reported. Why now: as generated code moves toward production, passing functional checks alone does not establish that a patch is maintainable or suitable for review.
The first implication is that benchmark rankings may change when evaluation includes maintainers’ quality standards rather than functional correctness alone. Cognition reports Claude Opus 4.8 at 13.4% on Diamond, 34.3% on Main, and 51.8% on Extended; these are the creator’s results, not an independent replication in the supplied evidence. The evidence boundary is that the source describes the benchmark and its results, but does not establish external validation of its claims. The first watch condition is whether independent evaluators reproduce the reported rankings and whether maintainers find the rubrics predictive of real review outcomes.