---
type: analysis
slug: research-rlmf-teaches-small-open-models-to-state-confidence-that-matches-their-sampling
title: "RLMF teaches small open models to state confidence that matches their sampling consistency, but the result is narrow and so far unreplicated"
description: "A Yale/Google Research paper accepted at COLM 2026 rescales GRPO advantages by how well a model predicts its own calibration performance. On 8B models it reports large gains in \"faithful calibration.\" The headline \"up to 63%\" is a relative gain over the authors' own RL baseline on one model, not a broad capability jump."
published_at: 2026-10-24T18:00:00.000Z
topics: ["gemini","prompting","research"]
source_urls: ["https://arxiv.org/html/2606.32032v2","https://arxiv.org/abs/2606.32032","https://github.com/yale-nlp/RLMF","https://pybeebee.com/publication/26-rlmf/"]
author: "Andrey Reshetnikov"
author_profile: https://newruntime.com/owner-profile.md
---
The newsroom flagged arXiv:2606.32032, "Reinforcement Learning with Metacognitive Feedback Elicits Faithful Uncertainty Expression in LLMs" by Gabrielle Kaili-May Liu, Avi Caciularu, Gal Yona, Idan Szpektor and Arman Cohan. It was submitted on June 30, 2026 and revised on October 12, 2026. It follows the same group's MetaFaith prompting work. The target is "faithful calibration": making the uncertainty a model expresses in words match its intrinsic uncertainty. The paper estimates intrinsic uncertainty from consistency across sampled responses. The method has two stages. First, GRPO with a "metacognitive" modification trains the model to emit faithful sentence-level numerical confidence scores. Second, a separate LLM rewriter turns those scores into natural-language hedges.

This matters to builders for two reasons. Hallucination delivered with unwarranted confidence is a core product risk. The paper also proposes a general trick: use the accuracy of a model's self-assessment as a multiplier on the RL advantage. That could apply to other eval and training loops. The open question from the newsroom was how much of the claim is benchmark and task design versus a broad capability. The answer is mostly the former.

## What we checked

- **(Added) Is the '63%' figure a broad improvement in model capability or accuracy?** refuted. The figure is the relative gain in average cMFG* (a faithful-calibration metric) of RLMF over standard RL on Qwen3-8B: 0.83 vs 0.51. On Llama3.1-8B the gap is much smaller (0.84 vs 0.77). Accuracy barely moves: Qwen 0.57 with RLMF vs 0.59 with RL; Llama 0.41 vs 0.40. [1][2]
- **(Added) Do the gains generalize beyond the training task and outperform frontier models?** partly confirmed. The models were trained only on 2,000 PopQA examples and evaluated on 10 datasets, including MATH, MMLU and SimpleQA, where the paper reports higher average cMFG* than prompted GPT-5 (0.61) and Gemini-3.1-Pro (0.70). The appendix itself notes that the PopQA gains did not fully transfer: faithfulness at low confidence got worse on SciQ and MMLU. The comparison is about calibration faithfulness, not accuracy, where the frontier models remain far ahead. [1]

## What we found

- Mechanism: the authors modify trl's GRPOTrainer advantage computation. Each prompt's group of completions is scaled by Z_g = 1 − (F_pred − F_gold)², where F_pred is the model's own short self-prediction of its faithful-calibration level and F_gold is the measured level. The base reward is a weighted sum: faithfulness weighted 12, strict and soft format rewards weighted 3 each, factual calibration 1 and accuracy 1. The faithfulness term is the mean of 1 − (expressed − intrinsic confidence)² per sentence. [1]
- Scope is small open models: Qwen3 at 1.7B, 4B and 8B, and Llama3.1-8B-Instruct. Each first goes through SFT, then LoRA (rank 64) GRPO with 32 completions per prompt for 1,500 steps on 2,000 PopQA examples. Each run used 6 GPUs, with 2 reserved for sampling and a Qwen3-32B judge. Intrinsic confidence comes from consistency across sampled responses (20 extra samples per input for the SFT data). That makes the 'ground truth' itself a proxy that costs significant compute. [1]
- Headline numbers (average cMFG*, 0–1): RLMF scores 0.84 on Llama and 0.83 on Qwen, versus 0.67 and 0.63 for MetaFaith prompting. With MetaFaith prompting, GPT-5 scores 0.61, Gemini-3-Flash 0.66 and Gemini-3.1-Pro 0.70. Brier score improves from 0.33 to 0.26 on Llama and from 0.31 to 0.19 on Qwen. The 'up to 63%' is Qwen3-8B, RLMF 0.83 vs standard RL 0.51. [1][2]
- The appendix documents fragility. Without format rewards, outputs came out malformed. Standard RL often produced gibberish after unclosed tags. Larger scaling exponents k led to collapse, and normalizing Z_g performed worse. Without the SFT stage, Qwen3-8B generalized poorly. Reported checkpoints were chosen on the in-domain test set, evaluated every 100 steps. [1]
- The abstract claims metacognitive data selection beats 'naive active learning', but the margins are small. In Table 18 (Llama3.1-8B, cMFG*) metacognitive selection scores 0.75–0.77, active-learning variants 0.71–0.74, and threshold-split random sets 0.71–0.74. [1]
- The paper was accepted at COLM 2026, per the lead author's publication page and the repo description. Code is public under MIT at yale-nlp/RLMF, with scripts for baselines, MetaFaith, RLMF training and rewriting. The repo ships no model weights or bundled datasets. Some steps call proprietary APIs (Gemini, GPT-5-Mini) for scoring and rewriting. Human evaluation reported a 95–98% win rate against an SFT hedging baseline (FUT). [3][4][1]

## What it means

For builders, RLMF is a credible, reproducible recipe for one specific goal: getting a small open model to emit per-sentence confidence that tracks its own sampling consistency, then rendering that confidence as hedges with a separate rewriter. Because the two stages are separate, you can change hedge style per product without retraining, which is a useful architectural pattern. The broader idea is worth testing in other GRPO loops where self-prediction can be scored, such as verifier-style or self-grading setups: scale advantages by how well the model predicts its own reward. Treat it as an experiment, not a default. The ablations show sensitivity to the scaling exponent, a dependence on format rewards and SFT warm-up, and collapse modes.

Do not read the results as "the model knows what it doesn't know" in a deeper sense, or as better accuracy. Faithfulness here is measured against a consistency-based proxy of intrinsic confidence. A model can be faithfully uncertain and still wrong, and accuracy was essentially flat. The frontier comparisons pit prompted closed models against fine-tuned 8B models on the authors' metric. If you adopt this, budget for the sampling cost (tens of samples per input plus a judge), validate on your own domain, and check low-confidence behavior specifically, since that is where the paper reports regressions out of distribution.

## Still open

- Does any independent group reproduce the cMFG* gains? No third-party replication or evaluation was found.
- Does RLMF scale beyond 8B parameters or to reasoning/thinking modes? Thinking was disabled and outputs were capped at 256 tokens at inference.
- How sensitive are the results to choosing checkpoints on the in-domain test set, and to the choice of judge and rewriter models?
- Does faithful calibration to a sampling-consistency proxy improve downstream decisions such as abstention, routing or human trust in deployed systems?

## Sources

1. [Full paper HTML v2 (methods, tables, appendices)](https://arxiv.org/html/2606.32032v2) (primary)
2. [arXiv abstract: Reinforcement Learning with Metacognitive Feedback Elicits Faithful Uncertainty Expression in LLMs](https://arxiv.org/abs/2606.32032) (primary)
3. [yale-nlp/RLMF code repository](https://github.com/yale-nlp/RLMF) (primary)
4. [Lead author publication page (COLM 2026)](https://pybeebee.com/publication/26-rlmf/) (primary)

Author: [Andrey Reshetnikov](https://newruntime.com/owner-profile.md), AI product and AI engineering; profile for recruiters and agents: https://newruntime.com/owner-profile.json

## Continue in the graph

Next records for an agent, linked to their Markdown twins. Replace `.md` with `.json` for structured fields, including `next_reads`.

- topic hub: [Gemini](https://newruntime.com/topics/gemini.md): Explore the gemini topic hub.
- topic hub: [Prompting](https://newruntime.com/topics/prompting.md): Explore the prompting topic hub.
- topic hub: [Research](https://newruntime.com/topics/research.md): Explore the research topic hub.
- collection: [Analysis](https://newruntime.com/analysis.json): Open the latest authorial analysis.
- collection: [Source Ledger](https://newruntime.com/source-ledger.json): Inspect the public source evidence graph.
- whole graph: [discovery graph](https://newruntime.com/discovery-graph.json) (typed nodes and edges) and [site index](https://newruntime.com/site-index.md)