---
type: analysis
slug: research-curated-agent-skills-lift-lab-scores-wild-retrieval-mostly-does-not
title: "Curated agent skills lift lab scores; wild retrieval mostly does not"
description: "SkillsBench reports a 16.6-point pass-rate lift when agents receive gold-curated skill packs. Later primary studies show that lift is fragile: it shrinks on real software-engineering work and collapses when agents must find skills in a large public library."
published_at: 2026-10-25T11:00:00.000Z
topics: ["skills","claude","agents","research"]
source_urls: ["https://arxiv.org/html/2602.12670v4","https://arxiv.org/abs/2602.12670","https://skillsbench.ai/","https://www.anthropic.com/engineering/equipping-agents-for-the-real-world-with-agent-skills","https://arxiv.org/html/2604.04323","https://arxiv.org/html/2603.15401","https://arxiv.org/abs/2603.15401","https://arxiv.org/abs/2604.04323","https://agentskills.io/specification"]
author: "Andrey Reshetnikov"
author_profile: https://newruntime.com/owner-profile.md
---
Agent Skills are filesystem packages of procedural knowledge: a required SKILL.md with YAML name and description, plus optional scripts, references, and assets. Anthropic introduced them in Claude products on 16 October 2025 and later published the layout as an open standard. A harness preloads metadata, then reads the body and bundled files only when the agent judges them relevant. That pattern now runs in Claude Code, Codex, Gemini CLI, and open harnesses, while public catalogs have grown into the millions of source-partitioned entries.

A news item could not settle whether those packages actually move task success. Three primary evaluation lines now exist. BenchFlow’s SkillsBench pairs 87 containerized, deterministically verified tasks with curated skills across 18 model–harness configurations. SWE-Skills-Bench injects 49 public software-engineering skills into pinned GitHub projects. A UCSB/MIT study forces retrieval from 34,198 real-world skills. Read together, they separate an upper-bound laboratory signal from production-like noise.

## What we checked

- **Do curated Agent Skills raise agent pass rates under matched no-skill versus with-skill evaluation?** confirmed. On SkillsBench’s current 87-task public aggregate, curated skills raise mean task-macro pass rate from 33.9% to 50.5% (+16.6 percentage points) across 18 model–harness configurations, with every configuration improving. The official SkillsBench leaderboard republishes the same paired scores for skillsbench@1.1. [1][2][3]
- **Do SkillsBench-scale gains hold when skills must be retrieved from public libraries or applied to real software-engineering repositories?** partly confirmed. Headline magnitude does not hold. SWE-Skills-Bench finds only a +1.2% average pass-rate gain, with 39 of 49 skills at zero delta. A UCSB/MIT study finds pass rates approaching no-skill baselines once agents retrieve from 34,198 public skills with curated packs removed. Specialized skills and query-specific refinement still help in pockets, including a Terminal-Bench 2.0 lift from 57.7% to 65.5% for Claude Opus 4.6. [6][5]

## What we found

- SkillsBench’s current public aggregate is 87 containerized tasks across eight professional domains, scored with pytest-style verifiers and no LLM-as-judge. Across 18 configurations and a fixed three-trial frame, curated skills raise task-macro pass rate from 33.9% to 50.5% (+16.6 percentage points; 25.5% normalized gain). Every configuration improves, but the lift ranges from +4.1 to +25.7 points. OpenHands with GPT-5.5 reaches 67.3% with skills; OpenHands with Gemini 3.1 Flash Lite is nearly flat at +4.1. The official site republishes the same 87-task skillsbench@1.1 leaderboard, recomputed 16 July 2026. [1][2][3]
- The average hides large domain and design effects. Natural science gains +28.8 points; software engineering only +11.6. Thirteen of 87 tasks show negative deltas, with the worst at −7.4 points, typically when a skill forces a heavyweight pipeline, crowds out a stronger native strategy, or points at a solver the agent cannot debug. Compact and standard-length skills (+19.0 and +21.5 points) beat comprehensive documentation (+0.7), and one to three skills beat bundles of four or more (+10.1). [1]
- Agents cannot currently write the skills they benefit from consuming. On Claude Code with Opus 4.7, Codex with GPT-5.5, and Gemini CLI with Gemini 3.1 Pro, a self-generated condition—author packs with Anthropic’s skill-creator, then solve with only those packs—lands 8.1 to 11.5 points below the no-skill baseline, while curated skills add 18.2 to 24.8 points on the same stacks. Trajectory audits attribute the deficit to unused generated packs, creator/solver interference, and confidently wrong pack content. [1][4]
- SkillsBench construction is stricter than a marketplace dump and still optimistic. Tasks come from 400 submissions by 142 contributors, with 87 retained (22% acceptance), leakage gates against magic numbers and oracle commands, and skills drawn from a top quality quartile (benchmark mean 10.1/12 versus an ecosystem mean of 6.2/12). The authors call this an optimistic scenario. An independent paper argues the pairing remains too task-specific: the flood-risk example ships API URLs and counting snippets that amount to a solution guide, and curated packs are placed in the environment rather than retrieved. [1][5]
- SWE-Skills-Bench, a preprint labeled work in progress, isolates skill utility on about 565 requirement-driven instances over 49 public SWE skills and pinned GitHub repos, using Claude Code with Claude Haiku 4.5. Average pass-rate gain is +1.2% (89.8% without skills to 91.0% with). Thirty-nine of 49 skills show zero delta; 24 of those already sit at 100% in both conditions, so many cells have no headroom. Seven specialized skills help, up to +30% on risk-metrics-calculation; three hurt, up to −10%, via version-mismatched templates. Token overhead is decoupled from correctness and reaches +450.8% on service-mesh-observability with no pass-rate change. [6][7]
- When agents must retrieve, SkillsBench-scale lifts collapse. On 84 SkillsBench tasks, Claude Opus 4.6 falls from 55.4% with force-loaded curated skills to 51.2% when loading is optional, 43.5% with distractors, 40.1% retrieving from 34,198 skills that still include the curated packs, and 38.4% when those packs are removed—only 3.0 points above a 35.4% no-skill baseline. Kimi K2.5 and Qwen3.5-397B-A17B drop below their no-skill baselines without curated skills. Query-specific refinement recovers much of the gap when retrieved skills are relevant (Claude 40.1% to 48.2% with curated still in the pool) and raises Claude’s Terminal-Bench 2.0 pass rate from 57.7% to 65.5%; it does not mint missing knowledge. Best agentic hybrid Recall@5 of curated packs in the 34k pool is 65.5%. [5][8][9]

## What it means

Treat the SkillsBench +16.6 point figure as an upper bound for a small set of high-quality, task-matched skills that the harness actually surfaces, not as the expected return on installing a marketplace. Ship compact skills that encode procedures, executable scripts, and verifier-facing constraints the model cannot infer; put applicability boundaries and a cheap fallback in SKILL.md; and run paired with-versus-without evaluation on your own tasks before rollout. Do not rely on the agent to author those packs at solve time.

If the library is larger than a handful of in-house skills, the bottleneck is selection and retrieval, not packaging. Prefer agentic hybrid search plus query-specific refinement over dumping SKILL.md files into context. For software engineering, assume most generic public skills will add tokens without raising pass rate, and that version-mismatched templates can reduce it. Audit third-party skills before install: the open format can bundle executable code.

## Still open

- How much of the SkillsBench delta is reusable expertise versus near-solution guidance that survived leakage filters?
- Would SWE-Skills-Bench still show about +1.2% with stronger models and more headroom, given many 100% cells, a single Haiku 4.5 configuration, and a GitHub URL that returned 404 in this session?
- Can length-matched or irrelevant-text controls isolate procedural structure from extra context, as SkillsBench itself flags as future work?
- Does query-specific refinement generalize outside the Harbor/Claude Code loops studied, at a cost builders will pay?

## Sources

1. [SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Tasks (HTML v4)](https://arxiv.org/html/2602.12670v4) (primary)
2. [SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Tasks (arXiv abs)](https://arxiv.org/abs/2602.12670) (primary)
3. [SkillsBench official site and leaderboard](https://skillsbench.ai/) (primary)
4. [Equipping agents for the real world with Agent Skills (Anthropic)](https://www.anthropic.com/engineering/equipping-agents-for-the-real-world-with-agent-skills) (primary)
5. [How Well Do Agentic Skills Work in the Wild (HTML)](https://arxiv.org/html/2604.04323)
6. [SWE-Skills-Bench: Do Agent Skills Actually Help in Real-World Software Engineering? (HTML)](https://arxiv.org/html/2603.15401)
7. [SWE-Skills-Bench: Do Agent Skills Actually Help in Real-World Software Engineering? (arXiv abs)](https://arxiv.org/abs/2603.15401)
8. [How Well Do Agentic Skills Work in the Wild (arXiv abs)](https://arxiv.org/abs/2604.04323)
9. [Agent Skills specification](https://agentskills.io/specification) (primary)

Author: [Andrey Reshetnikov](https://newruntime.com/owner-profile.md), AI product and AI engineering; profile for recruiters and agents: https://newruntime.com/owner-profile.json

## Continue in the graph

Next records for an agent, linked to their Markdown twins. Replace `.md` with `.json` for structured fields, including `next_reads`.

- topic hub: [Skills](https://newruntime.com/topics/skills.md): Explore the skills topic hub.
- topic hub: [Claude](https://newruntime.com/topics/claude.md): Explore the claude topic hub.
- topic hub: [Agents](https://newruntime.com/topics/agents.md): Explore the agents topic hub.
- collection: [Analysis](https://newruntime.com/analysis.json): Open the latest authorial analysis.
- collection: [Source Ledger](https://newruntime.com/source-ledger.json): Inspect the public source evidence graph.
- whole graph: [discovery graph](https://newruntime.com/discovery-graph.json) (typed nodes and edges) and [site index](https://newruntime.com/site-index.md)