Implication: Evaluation should test whether confidence ranks likely errors usefully, not only whether average confidence matches average accuracy.
Watch condition: Whether follow-up experiments show confidence-guided verification or abstention improves answer quality under a fixed review budget.
The result is a measurement proposal, not yet a demonstrated control loop. A rise or fall in confidence through reasoning can give context, but Cohere cautions that the final estimate is the prediction about answer correctness; the direction of change is not itself a verdict. Operational adoption will depend on evidence that these estimates remain calibrated on the tasks and populations where teams use them.
Thesis
Confidence in a reasoning model is more informative when it distinguishes expected ability before solving from confidence in the answer after solving.
Observed facts
Cohere says CALIBER trains models to report initial confidence, reasoning, an answer, and final confidence.
1Cohere reports training 7B and 30B models on 30,000 math problems and evaluating on 1,000 held-out problems plus three QA benchmarks.
1On BigMathDigits, Cohere reports 7B ECE of 0.028 for CALIBER versus 0.059 for the strongest single-confidence baseline.
1
Mechanism
CALIBER targets initial confidence to success frequency across repeated attempts on a question and final confidence to correctness of the particular answer, separating task-level expected ability from answer-level assessment.
Why now
Teams using model answers face limited review capacity, so confidence is useful only if it helps identify likely errors for scrutiny.
Implications
Evaluation should test whether confidence ranks likely errors usefully, not only whether average confidence matches average accuracy.
Deployment value remains unproven until confidence-guided verification or abstention is tested under real workflow constraints.
Evidence boundary
These are Cohere-reported benchmark experiments around a single reasoning process; the source does not test downstream verification, retrieval, or abstention decisions in real workflows.
Watch conditions
Whether follow-up experiments show confidence-guided verification or abstention improves answer quality under a fixed review budget.