---
type: analysis
slug: research-skills-pin-down-how-agents-act-more-than-they-teach-new-facts
title: "Skills pin down how agents act more than they teach new facts"
description: "A 14 August 2026 preprint holds past traces fixed and shows distilled SKILL.md files beat noisy workflow dumps by about six points. Exact skill picking collapses as libraries grow, while task success stays roughly flat, so skills should be treated as checklists with retrieval hygiene rather than as knowledge bases."
published_at: 2026-10-25T07:00:00.000Z
topics: ["skills","workflow","agents","research"]
source_urls: ["https://arxiv.org/html/2608.14036v1","https://zhiyuanjiang04.github.io/demystify-agent-skills/","https://arxiv.org/abs/2608.14036","https://www.anthropic.com/engineering/equipping-agents-for-the-real-world-with-agent-skills","https://agentskills.io/specification","https://arxiv.org/abs/2602.12670","https://arxiv.org/abs/2601.11868","https://raw.githubusercontent.com/zhiyuanjiang04/demystify-agent-skills/main/README.md","https://arxiv.org/pdf/2608.14036","https://github.com/zhiyuanjiang04/demystify-agent-skills"]
author: "Andrey Reshetnikov"
author_profile: https://newruntime.com/owner-profile.md
---
Agent Skills are filesystem packages—each a folder with a required SKILL.md plus optional scripts, references, and assets—that Anthropic launched on 16 October 2025 and later published as an open standard. At startup the agent loads only name and description; it reads the body and bundled files only if the skill looks relevant. That progressive-disclosure loop is now the usual way to specialize coding agents without fine-tuning.

Jiang, Huang, Xing, and colleagues argue that most skill papers still report only whether success rose. Their 14 August 2026 preprint reconstructs the pipeline: the same traces become Workflow Memory or a SKILL.md; paired trajectories are labeled; retrieval is tested separately from execution. The work sits on Terminal-Bench 2.0 (89 CLI tasks in the parent paper) and SkillsBench, run through Harbor. For builders, the live question is which failure a skill actually fixes and which new failure it creates.

## What we checked

- **Do Agent Skills mainly help by stabilizing procedures rather than by injecting missing facts?** partly confirmed. On 528 matched triples, an LLM judge tags procedural_anchor in 65.7% of skill cases versus knowledge_injection in 4.5%, and Skill beats Workflow Memory by 6.06 points from the same traces. The taxonomy map was human-checked, but the 65.7/4.5 split itself is LLM-assisted on truncated transcripts from about 3% of records, so the directional claim stands while the exact share does not. [3][1][2]
- **As skill pools grow, does exact ground-truth skill use collapse while downstream task success stays roughly stable?** confirmed. In the paper’s SkillsBench retrieval design, parsed actual-use precision falls from 29.6% at pool size 5 to 3.3% at 100, while averaged success moves only from 36.4% to 39.3%. A separate SkillsBench study likewise finds that high invocation does not guarantee resolution and that 13 of 87 tasks have negative skill deltas. [3][1][6]

## What we found

- On 528 matched triples covering SkillsBench, Terminal-Bench 2.0, and Terminal-Bench-Pro, Skill succeeds in 327 of 528 runs (61.9%) versus 295 of 528 (55.9%) for Workflow Memory and 312 of 528 (59.1%) for Raw. The Skill–Workflow delta is +6.06 points with a 95% bootstrap CI of [+0.76, +11.36]; Skill versus Raw is +2.84 points and that CI includes zero. Table 1 shows Skill beating Workflow on most success-heavy mixtures (Codex on Terminal-Bench-2 at 5s0f: 0.7548 versus 0.4452), while failure-only pools can drop Skill below Raw (Codex 0s5f: 0.5161 versus 0.5935). Compact short-plan and test-first baselines on 26 Terminal-Bench-2 tasks (47.7% and 59.2%) sit below Skill (79.2%). [1][2]
- The paper’s mechanism story is that skills stabilize action rather than supply missing facts: procedural_anchor is 65.7% of skill labels and knowledge_injection is 4.5%. Taxonomy construction from 238 of 240 open-coded records was human-checked (714 grounding checks; 95.8% exact mode mapping, Cohen’s kappa 0.952), but the 528-triple mechanism split is still LLM-judged on head-and-tail transcript excerpts. Authors also note that open coding covers about 3% of 8,135 normalized records. Treat 65.7 versus 4.5 as directional, not a settled population share. [3][1]
- Execution-layer SC2 modes fall from 37.3% of Raw labels and 33.3% of Workflow labels to 23.5% of Skill labels. environment_infrastructure_failure goes from 5.3% to 0.2%, output_format_schema_mismatch from 7.4% to 3.2%, and background_service_lifecycle_failure from 2.7% to 0.8%. algorithmic_logic_error stays 8.3%/11.0%/7.4% across Raw/Workflow/Skill, and static_verification_without_runtime stays 12.5%/12.5%/11.7%. Skills fix operational fragility; they do not repair a wrong algorithm or force runtime checks the agent is skipping. [1]
- The same abstraction creates a new failure surface. skill_guidance_misapplied_or_ignored is 10.0% of Skill arms versus 0.8% Raw and 0.4% Workflow. Workflow instead times out: timeout_budget_exhaustion is 10.6% versus 1.7% Raw and 4.4% Skill. Distillation cuts process noise and adds applicability errors. The official spec loads only name and description until the agent decides to read SKILL.md, so a vague description or a confusable neighbor can trigger the wrong checklist. [1][4][5]
- Retrieval is a separate bottleneck. On SkillsBench, Arm 3 actual-use precision falls from 29.6% at k=5 to 3.3% at k=100 while averaged success moves from 36.4% to 39.3%. Offline embedding top-1 goes 88.3% to 76.9%; explicit agent selection goes 70.0% to 63.7%. Similar distractors are the hard case (Arm 1 similar 70.5% to 53.4% versus dissimilar 96.6% to 93.2%). Arm 3 recall at k=100 stays 54.3–73.6%, so agents often inspect many skills but do not isolate the gold one. Exact gold invocation is neither sufficient nor necessary. [1][2]
- How a skill is built matters as much as whether one exists. Outcome labels barely matter on success-only pools but matter once failures enter: Gemini on Terminal-Bench-2 at 3s2f is 0.7462 with labels versus 0.4000 without. On 83 matched tasks Skill is 69.6% success at 521.5K tokens versus Workflow 64.8% at 426.2K. A separate SkillsBench study of 87 tasks across 18 model–harness stacks finds curated skills lift mean pass rate from 33.9% to 50.5% (+16.6 points); 13 of 87 tasks go negative; compact skills beat exhaustive docs (+19.0 and +21.5 points versus +0.7); and self-generated skills fall below the no-skill baseline. Terminal-Bench 2.0 itself remains hard (frontier agents under 65% on 89 tasks), which is why procedural packaging shows up at all. [1][6][7][8]

## What it means

Write skills as short procedures with when-to-use bounds, verification steps, and a cheap fallback, not as encyclopedias or dumped traces. Keep the live library small and mutually distinguishable: the open spec uses description text as the first-stage trigger, and confusable neighbors are what break identification. Do not auto-distill mixed success/failure traces without telling the creator which runs passed. Score skill systems on paired task success and on parsed use, not on exact gold-skill match alone. Do not expect a skill to repair a wrong algorithm or to force runtime tests the agent is skipping. Audit skills as executable code, because Anthropic’s own guidance treats untrusted skills as a code-execution and data-exfiltration risk.

Reproduction code now exists under zhiyuanjiang04/demystify-agent-skills, with a clean Harbor pipeline and checked-in taxonomy artifacts, but large traces live outside the repo and need Git LFS or a fresh noise-pool download. Until an independent team reruns the Skill-versus-Workflow comparison, treat the six-point packaging gain as a well-specified preprint result, not a production constant.

## Still open

- Numeric Codex-to-Gemini transfer rates are shown as a figure without tabulated values in the opened text.
- There is no public human-agreement statistic on the 65.7% versus 4.5% mechanism split across the 528 triples.
- Whether the retrieval collapse holds under production progressive disclosure rather than a fully mounted candidate pool is untested here.
- The study is limited to terminal and tool-use harnesses and two main model pairings, as the authors state.

## Sources

1. [Demystifying Agent Skills HTML preprint (v1)](https://arxiv.org/html/2608.14036v1) (primary)
2. [Demystifying Agent Skills project website](https://zhiyuanjiang04.github.io/demystify-agent-skills/) (primary)
3. [Demystifying Agent Skills: Why They Work—Until They Don’t (arXiv:2608.14036)](https://arxiv.org/abs/2608.14036) (primary)
4. [Equipping agents for the real world with Agent Skills (Anthropic)](https://www.anthropic.com/engineering/equipping-agents-for-the-real-world-with-agent-skills) (primary)
5. [Agent Skills open specification](https://agentskills.io/specification) (primary)
6. [SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Tasks](https://arxiv.org/abs/2602.12670)
7. [Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces](https://arxiv.org/abs/2601.11868)
8. [demystify-agent-skills README (reproduction workflow)](https://raw.githubusercontent.com/zhiyuanjiang04/demystify-agent-skills/main/README.md) (primary)
9. [Demystifying Agent Skills PDF preprint](https://arxiv.org/pdf/2608.14036) (primary)
10. [zhiyuanjiang04/demystify-agent-skills repository](https://github.com/zhiyuanjiang04/demystify-agent-skills) (primary)

Author: [Andrey Reshetnikov](https://newruntime.com/owner-profile.md), AI product and AI engineering; profile for recruiters and agents: https://newruntime.com/owner-profile.json

## Continue in the graph

Next records for an agent, linked to their Markdown twins. Replace `.md` with `.json` for structured fields, including `next_reads`.

- topic hub: [Skills](https://newruntime.com/topics/skills.md): Explore the skills topic hub.
- topic hub: [Workflow](https://newruntime.com/topics/workflow.md): Explore the workflow topic hub.
- topic hub: [Agents](https://newruntime.com/topics/agents.md): Explore the agents topic hub.
- collection: [Analysis](https://newruntime.com/analysis.json): Open the latest authorial analysis.
- collection: [Source Ledger](https://newruntime.com/source-ledger.json): Inspect the public source evidence graph.
- whole graph: [discovery graph](https://newruntime.com/discovery-graph.json) (typed nodes and edges) and [site index](https://newruntime.com/site-index.md)