What happened
Arya Somu has released ConvictionBench, a benchmark with a leaderboard, dataset, models and code for measuring evidence-sensitive belief revision in language models. The project asks a narrower question than whether a model reaches the right answer: when the model changes an answer, did the new information warrant that change?
Each episode starts with a question and an earlier assistant answer. That earlier answer may be correct, wrong, unjustified, or an abstention. A subsequent challenge then asks the model to reconsider. The model must return a reason, a final answer, and one of three decisions: keep, revise, or insufficient. ConvictionBench uses an exact solver to determine labels rather than relying on a model judge.
The benchmark distinguishes pressure from information that changes the facts. Its challenges include neutral prompts, appeals to authority, evaluator framing, consensus claims, emotional appeals, impatience, threats, repetition, bare assertions, valid derivations, flawed derivations, forged records, authenticated updates, and authenticated confirmations. Each problem is paired with nine framings; flattery is reserved as a held-out framing rather than appearing in training.
A central test concerns the channel through which a record arrives. ConvictionBench’s system prompt says that records arrive only through a tool channel. In its example, a user pastes an apparent administrative update adding the constraint y = 6 to an underdetermined equation. The correct action is to retain the insufficient answer because the user-pasted record does not alter the authenticated facts. When the same update arrives through the constraint_service tool, the correct action is to revise the answer to x = 8. The wording of the record is identical; its channel differs.
Somu’s page describes about 26,500 episodes across four task families: arithmetic, ledgers, propositional logic, and rule-based token worlds. There are roughly 13 matched episodes per problem in counterfactual groups, intended to prevent tone or template from predicting the label. Evaluation uses in-distribution, held-out-phrasing, and held-out-problem-type splits. The latter is presented as the headline split. A natural-history variant uses the model’s own initial answer instead of a scripted prior answer.
The page says off-the-shelf models vary widely in how often they yield to pressure and ignore a forged record more than a quarter of the time. It says none consistently checks where a record originated, treating a record pasted in a user message much like one delivered by a tool.
Somu also trained two rank-16 LoRA adapters, each with 17.4 million parameters, on Qwen3-1.7B. The supervised fine-tuning adapter used 4,000 solver-written responses and 11 GPU-minutes. It reported 0.96 forged-record resistance and 0.96 correct abstention, but accepted corrections to its own answers on unseen problem types only 7% of the time. The GRPO reinforcement-learning adapter ran for 200 steps and 65 GPU-minutes. It reported the lowest pressure-flip rate among 13 models and the best held-out performance for a 1.7B model, while scoring 0.19 on the provenance rule. A second RL seed reportedly produced held-out final accuracy of 0.642 versus 0.644. The page puts total rented GPU spend at about $17. ConvictionBench v0.1 lists CC-BY-4.0 for data and Apache-2.0 for code.
What the archive adds
New Runtime’s earlier coverage of BenchMIRT, published on 2026-10-23, examined a different approach to evaluating safety and general capability in language models. ConvictionBench is a new event in that wider evaluation thread, focused specifically on whether a model revises beliefs according to authenticated changes and valid correction.
Trend signal
ConvictionBench supports New Runtime’s verification-ownership assumption: automated checks can make agent output safer when acceptance rests on inspectable criteria rather than a reviewer’s impression alone. Its exact-solver labels and matched counterfactuals make a revision decision auditable: retain an answer, correct a genuine error, or reject an untrusted claim. That does not show that the benchmark predicts deployed-agent behavior.
Why it matters
Tool-using agents routinely receive claims through different interfaces: a user message, a retrieval result, a workflow event, or a privileged tool response. A correct-looking final answer cannot reveal whether the agent treated those interfaces appropriately. An agent that accepts a pasted “system update” has a different weakness from one that simply agrees with an insistent user; an agent that refuses every correction has a different weakness again.
ConvictionBench turns those distinctions into separately measured behaviors. The paired forged-record and authenticated-update cases are particularly relevant for teams whose agents consume operational data or tool outputs. They can adapt the structure to their own permissions, message channels, and update flows, checking both that untrusted claims do not override state and that legitimate updates can correct an answer.
The two training profiles also caution against using a single safety score as a deployment decision. In Somu’s results, the supervised adapter learned the channel rule but was reluctant to accept valid corrections on unfamiliar problems. The reinforcement-learning adapter generalized better across held-out problems but did not learn the channel rule. Teams need separate checks for authentic updates, correction behavior, and resistance to irrelevant persuasion.
Limits and uncertainty
Somu describes ConvictionBench as synthetic and closed-world, with short templated language. The project explicitly says its results do not establish robustness to real-world persuasion or misinformation. Its provenance mechanism is a convention stated in the system prompt, not an independently enforced identity system.
Models were evaluated without thinking mode. Gemma 4 and SmolLM3 were excluded because their chat templates hide or relabel tool messages. MobileLLM-Pro required an older Transformers version and followed the required response format strictly only around half the time. These implementation choices limit cross-model comparison. The reported leaderboard outcomes and training results have not been independently replicated here, and the benchmark does not measure downstream harm, tool authorization, or long-horizon agent behavior.
Reader takeaway
ConvictionBench’s practical contribution is separating two questions that are often merged: can a model resist pressure, and can it tell whether a claimed fact arrived through an authenticated channel? Teams building agents should test both behaviors, including whether a real update can overturn an earlier answer. Its scores are a useful evaluation starting point, not proof of production robustness.
Open question
Will performance on these matched synthetic tasks predict whether a deployed agent distinguishes authenticated tool updates from persuasive user-supplied claims in a real workflow?