Cohere says the method is designed to credit relevant results omitted from conventional judgment pools.
In a blind human study reported by Cohere, 46 annotators reviewed results from NanoBEIR, BRIGHT, and ViDoRe v3. Across 289 head-to-head contests covering 273 queries, RCP-nDCG selected the system preferred by reviewers 77% of the time, compared with 52% for conventional nDCG. The company also reports that where the two metrics named different winners, reviewers sided with RCP-nDCG 70% of the time. These are results from Cohere's described study, not independent validation. Source: https://cohere.com/blog/rcp-ndcg