Cheaper reasoning models often cost more per task, but the reversal numbers depend on a single study

What we checked

• If the mechanism generalizes beyond the reported setting, comparable primary studies should show the same direction under different models or tasks. partly confirmed. Nous Research (August 2025, different models, July 2025 OpenRouter prices) found that open-weight models used 1.5–4x more reasoning tokens, and that closed models' token efficiency often offset their higher per-token prices. That is the same direction. Two August 2026 papers back the mechanism: thinking dominates generated-token spend, and higher effort costs more without a detected accuracy gain. Only Chen et al. measure the pairwise reversal rate itself. [4][5][6][2]
• Are the headline reversal figures stable across versions of the paper? refuted. No. Version 1 covered 9 tasks and reported a 21.8% reversal rate, with Gemini 3 Flash 22% costlier than GPT-5.2. Version 2 covers 12 tasks with updated models and reports 32%, with Gemini 3 Flash 38% costlier than GPT-5.4. The direction holds, but the exact numbers depend on the model set, the task set and the price snapshot. [3][1]

What we found

• In version 2, the cheaper-listed model had the higher total cost in 32% of model-pair comparisons, and the gap reached up to 28x. In the headline example, Gemini 3 Flash is listed at $3.50 per million tokens (input plus output) against $17.50 for GPT-5.4, about 80% cheaper. Across all 12 datasets, Gemini 3 Flash cost about $705 and GPT-5.4 about $509, so the 'cheap' model was 38% more expensive. [1][2]
• Reversal rates vary a lot by task. They were 10.7% on ArenaHard and 14.3% on ARC-AGI, but 57.1% on MMLU-Pro and 39.3% on LiveCodeBench. Agentic tasks fell in between: 25.0% on TerminalBench 2.0, 35.7% on Cybench and 28.6% on GAIA. Knowledge-style tasks, where some models over-think, show the most reversals. [2]
• The authors trace the gap to thinking tokens and interaction turns, using a Shapley-value cost attribution. On the same query, one model may use 900% more thinking tokens than another, and on agentic tasks it may take 10x more turns. In version 1, removing thinking-token costs cut reversals by 70% and raised the Kendall's τ rank correlation between price ranking and cost ranking from 0.563 to 0.873. [1][3]
• Costs are also noisy for a single model. Repeated runs of the same query varied in thinking tokens by up to 9.7x, which the authors treat as a noise floor for any per-query cost predictor. They pose predicting the cost distribution as an open problem. [1]
• The paper states three limitations. It uses a single pricing snapshot (May 1, 2026) and runs each model at a single reasoning setting. It also measures cost without answer quality, so it says nothing about cost per correct answer. It promises to release code and data (more than 7.39B tokens, 6,877 tasks), but we found no repository link on the paper page. [2]
• Nous Research reached the same direction a year earlier with different models. Open-weight models used 1.5–4x more tokens than closed ones, up to 10x on simple knowledge questions. As a result, lower per-token prices 'don't always translate to lower cost per query'. On logic puzzles, where token counts varied little, cost mostly tracked list price. [4]
• Two August 2026 papers back the mechanism but do not measure reversals. 'The Reasoning Tax' (151 model-benchmark runs) finds that internal thinking often accounts for most inference spend, that low-efficiency behaviour clusters on knowledge-recall tasks like MMLU-Pro, and that higher effort gives diminishing returns. 'The Price of Thinking' tested Sonnet 5 on 30 AIME 2026 items. Explicit high effort raised cost per call and per correct answer ($0.08665 vs $0.07662) with no detected accuracy gain, and it documents that what happens when the effort parameter is omitted varies by model. [5][6]

What it means

Treat the per-token list price as an input, not an answer. What you actually pay per task is tokens times price, and the token side, especially hidden thinking tokens and agent turns, can swing by an order of magnitude between models and between runs. Before picking or routing a model on price, run it on a representative sample of your own traffic at the reasoning-effort setting you will deploy. Log the input, cached, output and thinking tokens and the turn count, and compare cost per completed task, ideally per correct task. Set the effort parameter explicitly rather than relying on defaults, since what an omitted value means differs across models. Budget for the spread of costs, not the mean: with up to 9.7x run-to-run variation, single-run estimates mislead.

Be careful with the headline numbers. The 32% rate and the Gemini 3 Flash vs GPT-5.4 comparison come from one paper. That paper uses one pricing snapshot, one reasoning setting per model, and no quality adjustment, and its figures moved substantially between versions. The direction (cheap per token is not cheap per task) is supported by independent work. The size of the effect is specific to your workload and will shift as providers reprice and retune their thinking behaviour.

Still open

• How do reversal rates change when cost is measured per correct answer rather than per query?
• How sensitive are reversals to reasoning-effort settings? Each model was tested at only one configuration.
• Will the promised code and 7.39B-token dataset be released so the results can be reproduced?
• How quickly do the results go stale as providers reprice after the May 1, 2026 snapshot or change how long models think?