What happened
Snorkel AI published its comparison of Fable 5.2 and Fable 5.1 on November 12, 2026. The company evaluated the two models on an expert-created set of 30 frontier coding tasks, with five attempts per task and 150 planned trials for each model. The analysis did not use every planned run for the direct comparison. Snorkel required both models to have at least three clean rollouts on a task, so that each task had enough valid attempts on both sides. Twenty-six of the 30 tasks met that requirement.
The matched analysis covers 106 trials for Fable 5.1 and 121 for Fable 5.2. Snorkel excluded trials lost to environment or endpoint errors from its headline figures. On the 26-task subset, Fable 5.2 achieved pass@1 of 71.4%, versus 53.1% for Fable 5.1, a gain of 18.3 percentage points. Its pass@3 was 80.4%, compared with 68.8%. At the trial level, Fable 5.2 passed 85 of 121 trials, or 70.2%; Fable 5.1 passed 58 of 106, or 54.7%.
Task-level results tell a more specific story than the aggregate pass rate. Both models solved 16 of 26 tasks at least once. Fable 5.2 alone solved five additional tasks, while Fable 5.1 alone solved two; neither solved the remaining three. Snorkel says Fable 5.2 passed every clean trial on 15 tasks, versus six for Fable 5.1. In other words, the newer model was not only more likely to solve a task at least once, but more likely to do so consistently across repeated attempts.
Snorkel also reviewed the traces for demonstrated skills. It recorded 302 verification instances for Fable 5.2 and rated 71% of them strong. The reported examples include creating adversarial fixtures with malformed inputs and corrupted archives, calculating expected results before comparing them with output, and rerunning completed steps to check that results were identical. The review identified 90 recovery episodes, 77 of which were successful. Snorkel reports a median recovery of one step and says 62 of the successful recoveries took two steps or fewer.
The company attributes the improvement partly to a changed failure profile. For Fable 5.1, premature termination accounted for 42% of failing trajectories, wrong output format for 25%, and wrong tool arguments for 26%; those classifications can overlap. Snorkel reports no premature terminations or output-format failures in the Fable 5.2 runs, and one wrong-arguments trace. Fable 5.2's remaining root-caused failures were wrong strategy at 35%, faulty inference at 28%, and misunderstanding the task at 23%. Two tasks that Fable 5.1 had passed reliably failed on every Fable 5.2 attempt because the newer model consistently misread the task.
What the archive adds
New Runtime’s earlier coverage of Vals’ comparison of Fable 5.2 with Fable 5.1 reported a clear coding improvement over the prior version. This is a new fact on that known story: Snorkel adds matched-task results, repeat-run consistency, and a breakdown of where failures moved rather than merely reporting an improvement claim.
Trend signal
This does not yet fit a pattern New Runtime tracks. The results make a useful distinction between execution failures and reasoning failures, but one benchmark comparison does not establish a broader change in how teams measure agent reliability, verification, or completed-work economics.
Why it matters
For a team selecting or operating a coding agent, the difference between an execution breakdown and a bad plan determines what can reasonably be fixed in the surrounding workflow. Snorkel’s results suggest that Fable 5.2 was less likely to lose otherwise viable work to terminal, output-format, or tool-argument errors. On tasks similar to those tested, that could mean fewer repeated runs caused by fragile execution rather than by the underlying implementation.
The residual failures are less accommodating. A model that chooses the wrong strategy, infers the wrong expected behavior, or misreads the task can execute cleanly and still produce a confidently incorrect result. Snorkel’s repeated failures on particular tasks show why sampling alone is not a sufficient mitigation: retries may reproduce the same interpretation. Teams should evaluate not only completion rates, but also whether their task specifications, test harnesses, and review points can catch an agent that verifies work against a mistaken premise.
Limits and uncertainty
Snorkel’s findings concern 26 matched tasks from its 30-task set, not a representative sample of every codebase, language, repository, or developer workflow. The comparison also uses unequal counts of clean trials after exclusions: 106 for Fable 5.1 and 121 for Fable 5.2. Snorkel does not report dollar cost per completed task, latency, token consumption, or an external replication.
The verification-skill ratings and root-cause labels are Snorkel’s own trace analysis. The article does not describe external review of those judgements or establish how the models would behave with a different harness, permissions, test suite, or task distribution. The figures are useful benchmark evidence, not a production-performance guarantee.
Skill to build now
Create a small repeat-run evaluation for the coding tasks that matter in your repository. For each task, run the candidate agent several times, preserve the traces, and separate failures into execution problems, such as malformed commands or lost state, and interpretation problems, such as an incorrect strategy or a mistaken reading of requirements. Add a verifier that tests the required outcome rather than accepting the agent’s own explanation of success. If you see the same wrong conclusion repeatedly, improve the task contract or introduce a review checkpoint instead of spending the budget on more retries.
Reader takeaway
Fable 5.2’s reported gain is substantial on Snorkel’s matched set, but the durable lesson is the change in failure mode. Mechanical instability may be reduced while task understanding remains a hard limit. Evaluate coding agents through repeated, representative runs and inspect why they fail, because one pass-rate number cannot show whether retries or scaffolding will solve the problem.