LiveNerf finds no large Opus 5.5 drop, but can't see small changes, model swaps or coding agents

What we checked

• Can the LiveNerf design detect practically meaningful changes in coding-agent performance or same-family model substitutions? partly confirmed. Partly: it can flag large reasoning-effort cuts through output tokens and accuracy drops of about 6.6 points or more per 10-day window, but it failed to distinguish Opus 5 from Opus 5.5 in validation and its QA panel does not measure coding-agent performance at all. [1][2][4]

What we found

• LiveNerf's headline result is a null: on a frozen 78-question panel, Opus 5.5 scored 58.4% in its launch baseline, 59.2% in days 11-20 and 57.1% in days 21-30, and neither window met the pre-registered change threshold. The author states the realized minimum detectable effect is about 6.6 points per 10-day window, so the null rules out only fairly large accuracy drops on this panel. [1]
• The instrument could not separate Opus 5 from Opus 5.5 in its own validation run: accuracy differed by -3.8 ± 6.3 points, and the validation report says the swap was not distinguishable at 99% in accuracy or tokens. A same-family substitution of that size is therefore outside what LiveNerf has shown it can detect, which the author says explicitly. [1][2]
• Reasoning-effort cuts show up far more clearly in output tokens than in accuracy. In the positive-control run, medium effort cut tokens 26% and low effort 62% while accuracy changes (-4.2 and -8.3 points) were inside the noise; the blog reports daily median output tokens stayed between 588 and 606 across windows. That makes token distributions the most sensitive signal LiveNerf has for a quiet effort reduction. [1][2]
• The panel does not measure coding-agent work. It is built from GPQA Diamond, MMLU-Pro, competition math and AIME questions, sent through a pinned headless Claude Code CLI on a Claude Max subscription; the Claude Code harness is only the transport. The author says the panel does not directly measure coding agents or long conversations, so system-prompt, context-management or tool changes could alter user experience without moving the score. [1][3]
• Item quality and statistical assumptions add further uncertainty. The audit found eight apparently wrong answer keys and about 30 ambiguous questions among 80 reviewed, kept in the frozen panel with a sensitivity analysis. The design document's MDE assumes item samples are independent day to day, and the validation A/A check (high vs high, split replicates) came out at +6.4 points with SE 3.6, consistent with zero but large, which suggests the noise floor is not small. [1][4][2]

What it means

Treat LiveNerf as evidence against a large, sustained accuracy drop or a big reasoning-effort cut on the subscription path it measures, not as evidence that nothing changed. It cannot rule out drops smaller than about 6.6 points, a same-family model substitution, or regressions in agent loops, tool use and long contexts, which is where most builder complaints live.

If your product depends on a hosted model, run your own longitudinal eval on your actual workload and log output-token distributions alongside pass rates: LiveNerf's own validation shows tokens move much more clearly than accuracy when effort is reduced. Pre-register thresholds and keep a control arm, as LiveNerf does, so you can tell model drift from harness or infrastructure changes.

Competing readings

• Opus 5.5 on this serving path has not suffered a large sustained accuracy drop or effort cut since launch (likely, about 70%). What would tell it apart: Window accuracies stayed within the noise of the 58.4% baseline [1]; Median output tokens stayed stable, while effort cuts move tokens sharply [1].
• A meaningful degradation exists but is below the panel's sensitivity or sits in agentic, tool or product layers the panel doesn't test (unlikely, about 25%). What would tell it apart: The instrument couldn't distinguish an Opus 5 swap and has a ~6.6-point MDE [1]; The panel doesn't measure coding agents, and a past real degradation came from an infrastructure misrouting bug [1][5].
• The measurement itself is too noisy or flawed (bad keys, correlated samples) to support any conclusion either way (very unlikely, about 10%). What would tell it apart: Many ambiguous items, an independence assumption behind the MDE, and a large A/A gap [1][4].

Base rate

The New Runtime archive holds only 4 earlier research stories about companies, models, products or projects from the last 12 months that are at least 3 months old: too few (minimum 5) for a base rate. The archive holds only 4 comparable earlier research stories (3 of them followed by a release), too few for a base rate, so this analysis does not lean on it.

What would change this view

• LiveNerf's third or later window shows a sustained drop meeting its pre-registered rule, or a sharp fall in median output tokens (model-listing signal): A real post-launch change on the measured path, most likely effort or serving changes.
• Anthropic publishes a postmortem or status note admitting a quality-affecting change to Opus 5.5 during the measured period (signal): Degradation occurred that the panel missed, confirming its blind spots.
• An independent agentic-coding longitudinal benchmark finds a significant Opus 5.5 regression over the same period (signal): Changes are concentrated in agent workloads the QA panel doesn't test.

Still open

• Does the realized 6.6-point MDE hold up once within-item day-to-day correlation is accounted for, given the large A/A gap in validation?
• Would a coding-agent or long-context panel show changes that the QA panel misses?
• Do API traffic, Bedrock or Vertex behave the same as the Claude Max subscription path LiveNerf measures?
• What does the third comparison window (days 31-40) show?