---
type: "post"
id: "rethinking-confidence-in-reasoning-models-analysis-f362d0"
slug: "rethinking-confidence-in-reasoning-models-analysis-f362d0"
title: "Rethinking Confidence in Reasoning Models"
description: "Thesis: Confidence in a reasoning model is more informative when it distinguishes expected ability before solving from confidence in the answer after solving."
retrieval_nugget: "Thesis: Confidence in a reasoning model is more informative when it distinguishes expected ability before solving from confidence in the answer after solving."
published_at: "2026-10-20"
updated_at: "2026-10-20"
record_date: "2026-10-20"
date_kind: "published_at"
topics: ["models"]
source_urls: ["https://cohere.com/blog/rethinking-confidence-in-reasoning-models"]
schema_version: "newruntime-agent-readable-v0.2"
stable_id: "post:rethinking-confidence-in-reasoning-models-analysis-f362d0"
status: "published"
visuals: []
editorial_provenance: {"schema_version":"newruntime-editorial-copy-v1","content_status":"source_grounded_final","final_copy_sha256":"sha256:ac857bdebe38474f3a0885d1f59fad4c8ec0d74e811ba551ce1b177807baddc6","reviewed_at":"2026-10-20T20:08:49.240Z","source_evidence_count":1,"verified_claim_count":3,"site_analysis_schema_version":"newruntime-site-analysis-v1","site_object_kind":"field_note","observed_fact_count":3,"implication_count":2,"watch_condition_count":1,"related_record_count":0}
analysis: {"schema_version":"newruntime-site-analysis-v1","object_kind":"field_note","thesis":"Confidence in a reasoning model is more informative when it distinguishes expected ability before solving from confidence in the answer after solving.","observed_facts":[{"text":"Cohere says CALIBER trains models to report initial confidence, reasoning, an answer, and final confidence.","source_urls":["https://cohere.com/blog/rethinking-confidence-in-reasoning-models"]},{"text":"Cohere reports training 7B and 30B models on 30,000 math problems and evaluating on 1,000 held-out problems plus three QA benchmarks.","source_urls":["https://cohere.com/blog/rethinking-confidence-in-reasoning-models"]},{"text":"On BigMathDigits, Cohere reports 7B ECE of 0.028 for CALIBER versus 0.059 for the strongest single-confidence baseline.","source_urls":["https://cohere.com/blog/rethinking-confidence-in-reasoning-models"]}],"mechanism":"CALIBER targets initial confidence to success frequency across repeated attempts on a question and final confidence to correctness of the particular answer, separating task-level expected ability from answer-level assessment.","why_now":"Teams using model answers face limited review capacity, so confidence is useful only if it helps identify likely errors for scrutiny.","implications":["Evaluation should test whether confidence ranks likely errors usefully, not only whether average confidence matches average accuracy.","Deployment value remains unproven until confidence-guided verification or abstention is tested under real workflow constraints."],"evidence_boundary":"These are Cohere-reported benchmark experiments around a single reasoning process; the source does not test downstream verification, retrieval, or abstention decisions in real workflows.","watch_conditions":["Whether follow-up experiments show confidence-guided verification or abstention improves answer quality under a fixed review budget."],"related_records":[],"new_branch_reason":"No prior New Runtime record was supplied for this specific method and benchmark result, so this creates a distinct record about separating pre-solve ability estimates from post-solve answer confidence."}
editor: {"role":"editor","name":"Andrey Reshetnikov","profile":"https://newruntime.com/owner-profile.md","profile_json":"https://newruntime.com/owner-profile.json","same_as":["https://www.linkedin.com/in/reshetnikov1/","https://t.me/lapetuse"]}
routes: {"html":"https://newruntime.com/posts/rethinking-confidence-in-reasoning-models-analysis-f362d0/","markdown":"https://newruntime.com/posts/rethinking-confidence-in-reasoning-models-analysis-f362d0.md","json":"https://newruntime.com/posts/rethinking-confidence-in-reasoning-models-analysis-f362d0.json"}
---

# Rethinking Confidence in Reasoning Models

## Retrieval answer

Thesis: Confidence in a reasoning model is more informative when it distinguishes expected ability before solving from confidence in the answer after solving.

Thesis: Confidence in a reasoning model is more informative when it distinguishes expected ability before solving from confidence in the answer after solving.

Observed facts: Cohere says CALIBER trains models to provide initial confidence, reasoning, an answer, and final confidence. It reports training 7B and 30B models on 30,000 math problems and evaluation on 1,000 held-out problems plus three QA benchmarks. On BigMathDigits, the 7B model’s ECE is reported as 0.028 versus 0.059 for the strongest single-confidence baseline.

Mechanism: CALIBER targets initial confidence to success frequency across repeated attempts on a question and final confidence to correctness of the particular answer, separating task-level expected ability from answer-level assessment. Cohere also evaluates calibration (ECE) and discrimination (AUROC), since matching aggregate accuracy alone may not help prioritize which answers deserve review.

Why now: Teams using model answers face limited review capacity, so confidence is useful only if it helps identify likely errors for scrutiny.

Implication: Evaluation should test whether confidence ranks likely errors usefully, not only whether average confidence matches average accuracy.

Evidence boundary: These are Cohere-reported benchmark experiments around a single reasoning process; the source does not test downstream verification, retrieval, or abstention decisions in real workflows.

Watch condition: Whether follow-up experiments show confidence-guided verification or abstention improves answer quality under a fixed review budget.

The result is a measurement proposal, not yet a demonstrated control loop. A rise or fall in confidence through reasoning can give context, but Cohere cautions that the final estimate is the prediction about answer correctness; the direction of change is not itself a verdict. Operational adoption will depend on evidence that these estimates remain calibrated on the tasks and populations where teams use them.

Editor: [Andrey Reshetnikov](https://newruntime.com/owner-profile.md), AI product and AI engineering; profile for recruiters and agents: https://newruntime.com/owner-profile.json

## Continue in the graph

Next records for an agent, linked to their Markdown twins. Replace `.md` with `.json` for structured fields, including `next_reads`.

- topic hub: [Models](https://newruntime.com/topics/models.md): Explore the models topic hub.
- collection: [Field Notes](https://newruntime.com/posts.json): Open the latest editorial analysis.
- collection: [Source Ledger](https://newruntime.com/source-ledger.json): Inspect the public source evidence graph.
- whole graph: [discovery graph](https://newruntime.com/discovery-graph.json) (typed nodes and edges) and [site index](https://newruntime.com/site-index.md)