Agent Skills are filesystem packages—each a folder with a required SKILL.md plus optional scripts, references, and assets—that Anthropic launched on 16 October 2025 and later published as an open standard. At startup the agent loads only name and description; it reads the body and bundled files only if the skill looks relevant. That progressive-disclosure loop is now the usual way to specialize coding agents without fine-tuning.
Jiang, Huang, Xing, and colleagues argue that most skill papers still report only whether success rose. Their 14 August 2026 preprint reconstructs the pipeline: the same traces become Workflow Memory or a SKILL.md; paired trajectories are labeled; retrieval is tested separately from execution. The work sits on Terminal-Bench 2.0 (89 CLI tasks in the parent paper) and SkillsBench, run through Harbor. For builders, the live question is which failure a skill actually fixes and which new failure it creates.
What we checked
- Do Agent Skills mainly help by stabilizing procedures rather than by injecting missing facts? partly confirmed. On 528 matched triples, an LLM judge tags procedural_anchor in 65.7% of skill cases versus knowledge_injection in 4.5%, and Skill beats Workflow Memory by 6.06 points from the same traces. The taxonomy map was human-checked, but the 65.7/4.5 split itself is LLM-assisted on truncated transcripts from about 3% of records, so the directional claim stands while the exact share does not. [3][1][2]
- As skill pools grow, does exact ground-truth skill use collapse while downstream task success stays roughly stable? confirmed. In the paper’s SkillsBench retrieval design, parsed actual-use precision falls from 29.6% at pool size 5 to 3.3% at 100, while averaged success moves only from 36.4% to 39.3%. A separate SkillsBench study likewise finds that high invocation does not guarantee resolution and that 13 of 87 tasks have negative skill deltas. [3][1][6]
What we found
- On 528 matched triples covering SkillsBench, Terminal-Bench 2.0, and Terminal-Bench-Pro, Skill succeeds in 327 of 528 runs (61.9%) versus 295 of 528 (55.9%) for Workflow Memory and 312 of 528 (59.1%) for Raw. The Skill–Workflow delta is +6.06 points with a 95% bootstrap CI of [+0.76, +11.36]; Skill versus Raw is +2.84 points and that CI includes zero. Table 1 shows Skill beating Workflow on most success-heavy mixtures (Codex on Terminal-Bench-2 at 5s0f: 0.7548 versus 0.4452), while failure-only pools can drop Skill below Raw (Codex 0s5f: 0.5161 versus 0.5935). Compact short-plan and test-first baselines on 26 Terminal-Bench-2 tasks (47.7% and 59.2%) sit below Skill (79.2%). [1][2]
- The paper’s mechanism story is that skills stabilize action rather than supply missing facts: procedural_anchor is 65.7% of skill labels and knowledge_injection is 4.5%. Taxonomy construction from 238 of 240 open-coded records was human-checked (714 grounding checks; 95.8% exact mode mapping, Cohen’s kappa 0.952), but the 528-triple mechanism split is still LLM-judged on head-and-tail transcript excerpts. Authors also note that open coding covers about 3% of 8,135 normalized records. Treat 65.7 versus 4.5 as directional, not a settled population share. [3][1]
- Execution-layer SC2 modes fall from 37.3% of Raw labels and 33.3% of Workflow labels to 23.5% of Skill labels. environment_infrastructure_failure goes from 5.3% to 0.2%, output_format_schema_mismatch from 7.4% to 3.2%, and background_service_lifecycle_failure from 2.7% to 0.8%. algorithmic_logic_error stays 8.3%/11.0%/7.4% across Raw/Workflow/Skill, and static_verification_without_runtime stays 12.5%/12.5%/11.7%. Skills fix operational fragility; they do not repair a wrong algorithm or force runtime checks the agent is skipping. [1]
- The same abstraction creates a new failure surface. skill_guidance_misapplied_or_ignored is 10.0% of Skill arms versus 0.8% Raw and 0.4% Workflow. Workflow instead times out: timeout_budget_exhaustion is 10.6% versus 1.7% Raw and 4.4% Skill. Distillation cuts process noise and adds applicability errors. The official spec loads only name and description until the agent decides to read SKILL.md, so a vague description or a confusable neighbor can trigger the wrong checklist. [1][4][5]
- Retrieval is a separate bottleneck. On SkillsBench, Arm 3 actual-use precision falls from 29.6% at k=5 to 3.3% at k=100 while averaged success moves from 36.4% to 39.3%. Offline embedding top-1 goes 88.3% to 76.9%; explicit agent selection goes 70.0% to 63.7%. Similar distractors are the hard case (Arm 1 similar 70.5% to 53.4% versus dissimilar 96.6% to 93.2%). Arm 3 recall at k=100 stays 54.3–73.6%, so agents often inspect many skills but do not isolate the gold one. Exact gold invocation is neither sufficient nor necessary. [1][2]
- How a skill is built matters as much as whether one exists. Outcome labels barely matter on success-only pools but matter once failures enter: Gemini on Terminal-Bench-2 at 3s2f is 0.7462 with labels versus 0.4000 without. On 83 matched tasks Skill is 69.6% success at 521.5K tokens versus Workflow 64.8% at 426.2K. A separate SkillsBench study of 87 tasks across 18 model–harness stacks finds curated skills lift mean pass rate from 33.9% to 50.5% (+16.6 points); 13 of 87 tasks go negative; compact skills beat exhaustive docs (+19.0 and +21.5 points versus +0.7); and self-generated skills fall below the no-skill baseline. Terminal-Bench 2.0 itself remains hard (frontier agents under 65% on 89 tasks), which is why procedural packaging shows up at all. [1][6][7][8]
What it means
Write skills as short procedures with when-to-use bounds, verification steps, and a cheap fallback, not as encyclopedias or dumped traces. Keep the live library small and mutually distinguishable: the open spec uses description text as the first-stage trigger, and confusable neighbors are what break identification. Do not auto-distill mixed success/failure traces without telling the creator which runs passed. Score skill systems on paired task success and on parsed use, not on exact gold-skill match alone. Do not expect a skill to repair a wrong algorithm or to force runtime tests the agent is skipping. Audit skills as executable code, because Anthropic’s own guidance treats untrusted skills as a code-execution and data-exfiltration risk.
Reproduction code now exists under zhiyuanjiang04/demystify-agent-skills, with a clean Harbor pipeline and checked-in taxonomy artifacts, but large traces live outside the repo and need Git LFS or a fresh noise-pool download. Until an independent team reruns the Skill-versus-Workflow comparison, treat the six-point packaging gain as a well-specified preprint result, not a production constant.
Still open
- Numeric Codex-to-Gemini transfer rates are shown as a figure without tabulated values in the opened text.
- There is no public human-agreement statistic on the 65.7% versus 4.5% mechanism split across the 528 triples.
- Whether the retrieval collapse holds under production progressive disclosure rather than a fully mounted candidate pool is untested here.
- The study is limited to terminal and tool-use harnesses and two main model pairings, as the authors state.