Field noteNov 18, 20265 min read1 source

Endor Labs finds Fable 5.2 below Fable 5.1 once memorized coding fixes are removed

In one minute

Endor Labs tested Claude Code with Fable 5.2 on coding tasks drawn from historically vulnerable code and found a 35.8% security pass rate after removing solutions it judged to be recalled from training. The run was faster and cheaper than its Fable 5.1 comparison, but the benchmark cautions that raw pass rates can overstate independent problem-solving.

What changed: Endor Labs reports Fable 5.2 scored 36.9% SecPass and 64.8% FuncPass in Claude Code, finding it less secure than Fable 5.1 after removing memorized fixes.

What happened

Endor Labs published a comparison of Claude Code paired with Fable 5.2 against earlier model-and-harness combinations in its Agent Security League benchmark. The company says Anthropic released Fable 5.2 on November 9, 2026; Endor Labs published its analysis on November 16 and updated it November 17. The benchmark asks agents to work on real code tasks based on code that was historically part of a security fix, without telling them that security is the point. Each agent runs once per task, and its patch is tested in an isolated Docker environment.

Endor Labs defines FuncPass as passing the functional tests, and SecPass as passing those tests plus hidden security tests associated with the original vulnerability fix. Its scoring also removes solutions it identifies as training recall: the company says it uses a multi-signal pipeline to flag suspicious cases and an LLM to adjudicate them, then manually reviews candidate unique security solves. Under those rules, Claude Code plus Fable 5.2 scored 63.7% FuncPass and 35.8% SecPass. Claude Code plus Fable 5.1 scored 86.6% and 36.9%, respectively; Codex plus GPT-6 Astra scored 82.1% and 34.6%.

The adjustment is central to the result. Endor Labs reports 65 confirmed recalled solutions for Fable 5.2, compared with 18 for Fable 5.1 and 42 for Fable 5.0. If those solutions were counted, Fable 5.2 would have scored 94.4% FuncPass and 60.3% SecPass; with them removed, its functional score falls below Fable 5.1's 86.6%. The lab says 44 of 55 recalled solutions in the scored set had also passed the security tests. Its result is therefore not simply that the model failed more often: the benchmark treats reproducing a known fix as different from independently deriving one.

The run had a $187 recorded prediction cost, compared with $672 for Fable 5.1, and median time per task of 1.9 minutes versus 7.2. Endor Labs attributes the cost gap partly to lower token use: Fable 5.2 wrote 68% fewer output tokens. The model also used about 11 tool calls per task on average, versus about 42 for Fable 5.1, and used shell commands for 88.5% of tool calls. The report says the run used Claude Code 2.1.321 at default effort; Fable 5.1 used version 2.1.258.

The benchmark includes one notable security result after recalled solutions were removed: an OpenStack Neutron task involving default security groups. Endor Labs says Fable 5.2 alone passed the hidden security tests on that task. The task description asked for a function to create a default group, but did not mention ownership checks. The agent initially implemented the described steps, then inspected callers, identified that request filters could supply another tenant's ID, and added an ownership check. Endor Labs reports that its ablation of the check was inconclusive because API test failures matched pre-existing environment failures.

What the archive adds

New Runtime covered Snorkel's matched-task Fable 5.2 evaluation on November 14, which reported 71% pass@1 against 53% for Fable 5.1. Endor Labs adds a different benchmark and a different question: security performance after removing fixes judged to be training recall. This is a new fact on a known model-comparison story, not a directly comparable score.

Trend signal

This does not yet fit a pattern New Runtime tracks. The result nonetheless highlights an editorial distinction worth retaining: an agent's passing patch and evidence that it independently derived a secure patch are different claims.

Why now

Anthropic positioned Fable 5.2 as a faster, lower-priced generation with coding gains. Endor Labs' November evaluation tests those claims against its own security-focused benchmark, while separating lower cost and faster execution from independently scored correctness.

Connections to other events

New Runtime's October 23 coverage of Gemini 4 Argon on Cursor also examined security benchmark results for a model-and-coding-tool combination. Endor Labs adds a separate methodology that explicitly removes fixes judged to be recalled from training, making the scores informative in different ways rather than directly comparable.

Why it matters

For teams choosing coding agents, this is a reminder to distinguish a passing patch from evidence that the agent can generalize to unfamiliar failures. If a benchmark includes public vulnerabilities or code fixes that may have appeared in training, test design and provenance checks can materially change the ranking. Endor Labs' adjustment moves Fable 5.2 from a seemingly exceptional functional score of 94.4% to 63.7%, while its security score is slightly below Fable 5.1.

The operational trade-off is also concrete: in this run, Fable 5.2 completed tasks faster and at lower recorded cost, but that efficiency did not translate into a higher adjusted score. Builders should treat cost, speed, secure correctness, and independence as separate measures. The one unique Neutron result suggests agents can sometimes notice a security property beyond the task wording, but one manually reviewed task is not a reliability estimate.

Limits and uncertainty

These results apply to Endor Labs' task set, one run per task, Claude Code harness versions, and default model effort. The compared runs did not use identical Claude Code versions, and the cost figures for Claude Code and Codex were calculated from different sources, so exact cross-combination cost comparisons need caution. The company describes its recall-detection and adjudication process, but the results do not independently establish how accurately it identifies training recall. Nor does the benchmark measure performance across all coding tasks, real production repositories, or other effort settings. The published analysis also says it did not test Mythos 5.2.

Reader takeaway

Fable 5.2 was substantially faster and cheaper in Endor Labs' test, but its adjusted functional score trailed Fable 5.1 and its security score was 1.1 points lower. For security-sensitive agent evaluation, count verified, independently derived fixes—not just patches that pass tests.

Evidence / sources

  1. 1endorlabs.com/learn/fable-5-2-cheaper-and-faster-not-more-secure-and-much-more-memorization
All Field Notes

Next reads

Read next

Open the graph
  1. 01topicClaudeExplore the claude topic hub.
  2. 02topicClaude CodeExplore the claude-code topic hub.
  3. 03topicAnthropicExplore the anthropic topic hub.
  4. 04archiveField NotesOpen the latest editorial analysis.
  5. 05sourcesSourcesSee the original sources this archive cites.
Who read this page?Machine requests, hidden until opened

Loading the privacy-safe route aggregate...

Open the JSON contract