Wisedocs, which sells medical record review tooling, published MLCR (Medical Long Context Reasoning) on June 18, 2026. It is a benchmark of 10 synthetic medical case files, roughly 25k–64k tokens each, mixed with filler forms and paired with questions in six difficulty tiers. The hardest tiers became a private hold-out set that Artificial Analysis now runs as the MLCR-AA leaderboard. The current headline says cost per task ranges from $0.006 (GPT-6 Luna) to $1.30 (Claude Fable 5.1), next to pass rates that top out around 42%.
For builders choosing a model for long-document clinical or insurance work, cost-versus-accuracy numbers like these feed straight into procurement decisions. So it matters whether the numbers can be checked, how cost is defined, and whether the scores hold steady when the grading rules change.
What we checked
- Can the MLCR dataset, scoring method, prompts, model settings, and cost calculations be independently inspected or reproduced? partly confirmed. The harness (with prompts, YAML model configs, and the three-judge scorer) and a public dataset (10 cases, 205 questions, filler OCR text) are on GitHub and Hugging Face, and filler noise is seeded. The MLCR-AA leaderboard that supplies the cost figures runs a private 60-question hold-out set with a different judge panel. No primary source documents a cost formula beyond token-category pricing, the settings of the evaluated models, or whether judge calls count toward cost. [2][3][1][4][5]
- Do the quoted per-task costs ($0.006 for GPT-6 Luna, $1.30 for Claude Fable 5.1) appear in a primary source? unresolved. The Wisedocs blog cited as the known source has no dollar figures and doesn't mention GPT-6 Luna or Fable 5.1. On the MLCR-AA page, cost per task appears only in charts I could not read as text, so I could not confirm the specific numbers. They seem to trace back to secondary coverage of a refreshed leaderboard. [1][4]
- Are MLCR-AA results comparable across leaderboard versions? refuted. The judge panel changed. In the v1.1 blog it was Claude Opus 4.6, GPT 5.5 and Gemini 3.1 Pro. AA's methodology now lists Claude Opus 5.5, GPT-6.1 Sol and Grok 4.7 at medium effort, and no longer mentions the concision gate that v1.1 had loosened from 3× to 5×. The launch top score (Claude Fable 5, 64.4%) and the current top pass rate (41.7%) come from different scoring regimes. [6][5][4]
What we found
- The Wisedocs launch post, dated June 18, 2026 and written by Head of ML Denys Linkov, describes 10 synthetic cases, six tiers (the last being a compound hallucination check), a junk_context_ratio for filler, and a three-model judge panel with majority vote and a conciseness gate. It has Pareto cost charts but no dollar figures in the text, and it never mentions GPT-6 Luna or Claude Fable 5.1. It also doesn't explain how cost was computed. [1]
- The public release is real but small, and its licensing is unclear. The GitHub harness has 2 commits on main, YAML configs for each model (providers google, anthropic_gcp, azure_openai), plan/dry-run/run/score commands, and per-row prompt.txt outputs. No license is shown. The Hugging Face dataset holds 10 cases, 205 questions and 262 filler files under a license marked only as 'other'. The blog's own question counts (250 per case vs '100+' open-sourced) don't match this. [2][3][1]
- The repo says providers report prompt, completion and thinking tokens, but it doesn't describe turning them into dollars. AA's MLCR-AA page says costs are split into input, cache hit, cache write, reasoning and answer token pricing 'where canonical token counts are available', with no formula and no price sources. Neither says whether judge calls are counted. AA's GDP.pdf eval explicitly leaves judge calls out, but MLCR-AA has no such statement. [2][4][5]
- MLCR-AA scores 60 private questions from the expert and compound tiers, each run 3 times, reported as pass@1. AA's methodology now lists judges Claude Opus 5.5, GPT-6.1 Sol and Grok 4.7 at medium reasoning effort and doesn't mention a concision gate. The only prompt instruction given is 'answer each part in one paragraph and cite evidence from the record.' Settings for the evaluated models, such as temperature, are not specified. MLCR-AA has no entry in AA's version history. [5][4]
- The rules have shifted more than once. Wisedocs's v1.1 post (Aug 31, 2026) raised the length gate from 3× to 5×, made accuracy cover only factual errors, and reported judge agreement on accuracy rising from 12.9% to 62.3% (Opus vs GPT). It also reported Claude Fable 5 at 64.4%. That post says MLCR-AA uses the 'three hardest tiers', while AA says two. The current AA leaderboard shows Claude Sonnet 5.5 and GPT-6 Astra tied at 41.7% pass rate, with GPT-6.1 Sol at 96.7% accuracy. [6][4]
What it means
Treat the $0.006–$1.30 range as a rough sign of how widely costs spread, not a number to quote. I could not find it in any primary text. It comes from a private question set, scored by a judge panel that was just swapped out, with a cost method that names token categories but not prices, cache assumptions, or whether judging is included. The two endpoints are also misleading together: secondary coverage reports no score for the cheapest model, and on the current leaderboard the most expensive configuration is not the most accurate. When a cheap model wins the cost line, check its pass rate before taking anything from the comparison.
The useful part for builders is the open harness. You can point it at your own models, prompt formats and filler ratios, read the exact prompts it generates, and compute cost from your own provider invoices, cache hit rates and reasoning-effort settings. Expect rankings to move with judge choice. Wisedocs's own calibration data shows judges agreeing on accuracy as little as about 13% of the time before recalibration. Before relying on any MLCR result in production, run your own small human-graded check, and keep pass rate, accuracy (no contradictions) and completeness as separate numbers.
Still open
- What exact per-model cost-per-task values does the current MLCR-AA chart show, and when was the refresh with the new judges published?
- Does MLCR-AA cost include judge calls, and which list prices and cache-hit assumptions are used per provider?
- What are the evaluated models' sampling and reasoning settings beyond the effort labels (Max, Xhigh)?
- Does MLCR-AA cover two or three private tiers, and how many questions does each case really have (205 public vs the 250 per case the blog claims)?