The company introduces Rubric-Calibrated Preferences nDCG@10 (RCP-nDCG@10), which evaluates retrieved documents against a shared rubric with a calibrated AI judge rather than treating an existing answer key as complete.
The mechanism combines five yes/no rubric questions for each document with pairwise comparisons among documents; Cohere says the comparisons establish ordering while calibration puts relevance on a scale shared across queries. In its blind study, 46 annotators assessed top-five results from NanoBEIR, BRIGHT, and ViDoRe v3. Cohere reports 289 contests across 273 queries and says RCP-nDCG selected the system reviewers preferred in 77% of contests, versus 52% for conventional nDCG. These reported results make the evaluation method—not a newly launched search product—the material change.
The first implication is that retrieval teams may need to test whether their offline benchmarks undercount relevant results beyond their existing qrels. The evidence boundary is that these are Cohere-reported findings from its study, not independent replication; rubric design, judge calibration, task selection, and sampling may affect the result. Watch for independent evaluations across other corpora and for reproducible details of the rubric and calibration procedure. Source: https://cohere.com/blog/rcp-ndcg