--- type: "post" id: "giving-coding-agents-a-docs-link-doubles-task-630606" slug: "giving-coding-agents-a-docs-link-doubles-task-630606" title: "Giving coding agents a docs link doubles task completion from 41% to 84%." description: "**Thesis:** Agent success depends not only on access to documentation, but on whether tools make their behavior and omissions legible." retrieval_nugget: "**Thesis:** Agent success depends not only on access to documentation, but on whether tools make their behavior and omissions legible." published_at: "2026-10-15" updated_at: "2026-10-15" record_date: "2026-10-15" date_kind: "published_at" topics: ["coding-agents"] source_urls: ["https://www.agentmail.to/evals/report"] schema_version: "newruntime-agent-readable-v0.2" stable_id: "post:giving-coding-agents-a-docs-link-doubles-task-630606" status: "published" visuals: [] editorial_provenance: {"schema_version":"newruntime-editorial-copy-v1","content_status":"source_grounded_final","final_copy_sha256":"sha256:aae1c297f3e09c7b0f74409f5dae881723fe4cb3645b73555e39d2c0cf5d8e0f","reviewed_at":"2026-10-15T16:09:22.316Z","source_evidence_count":1,"verified_claim_count":2,"site_analysis_schema_version":"newruntime-site-analysis-v1","site_object_kind":"field_note","observed_fact_count":2,"implication_count":1,"watch_condition_count":1,"related_record_count":0} analysis: {"schema_version":"newruntime-site-analysis-v1","object_kind":"field_note","thesis":"Agent success depends not only on access to documentation, but on whether tools make their behavior and omissions legible.","observed_facts":[{"text":"AgentMail reports running 25 agents across seven tasks, with and without links to its documentation.","source_urls":["https://www.agentmail.to/evals/report"]},{"text":"Its executive summary says documentation links lifted success from 76% to 84%, and the report says many failures involved hidden inbox messages, endpoints whose behavior differed from expectations, or conflicting labels.","source_urls":["https://www.agentmail.to/evals/report"]}],"mechanism":"Documentation can help an agent discover the right operation, but it cannot reliably compensate for an interface that conceals filtered-out data or signals behavior ambiguously.","why_now":"Teams increasingly delegate multi-step work through APIs and internal services, making small interface assumptions consequential.","implications":["Tool builders should make omissions and operation semantics explicit—for example, indicate when a list excludes messages—rather than treating documentation as the sole remedy."],"evidence_boundary":"This is evidence about one vendor's email API, a particular set of tasks, agents, and harnesses; it does not establish that documentation produces the same lift across other tools or workloads. The excerpt also notes that docs made agents worse on one job, so the aggregate improvement does not imply uniform benefit.","watch_conditions":["Look for independent evaluations across other APIs that report task-level outcomes and isolate documentation effects from harness, model, and interface changes."],"related_records":[],"new_branch_reason":"No prior New Runtime analytical record was supplied; this evaluation starts a distinct record on tool legibility and documentation's measured effect in agent task completion."} editor: {"role":"editor","name":"Andrey Reshetnikov","profile":"https://newruntime.com/owner-profile.md","profile_json":"https://newruntime.com/owner-profile.json","same_as":["https://www.linkedin.com/in/reshetnikov1/","https://t.me/lapetuse"]} routes: {"html":"https://newruntime.com/posts/giving-coding-agents-a-docs-link-doubles-task-630606/","markdown":"https://newruntime.com/posts/giving-coding-agents-a-docs-link-doubles-task-630606.md","json":"https://newruntime.com/posts/giving-coding-agents-a-docs-link-doubles-task-630606.json"} --- # Giving coding agents a docs link doubles task completion from 41% to 84%. ## Retrieval answer **Thesis:** Agent success depends not only on access to documentation, but on whether tools make their behavior and omissions legible. **Thesis:** Agent success depends not only on access to documentation, but on whether tools make their behavior and omissions legible. **Observed facts:** AgentMail reports running 25 agents across seven tasks, with and without links to its documentation. **Observed facts:** Its executive summary says documentation links lifted success from 76% to 84%, and the report says many failures involved hidden inbox messages, endpoints whose behavior differed from expectations, or conflicting labels. The headline's 41% figure is not corroborated by the supplied report excerpt. **Mechanism:** Documentation can help an agent discover the right operation, but it cannot reliably compensate for an interface that conceals filtered-out data or signals behavior ambiguously. The report's task grader examined captured requests and outputs rather than accepting an agent's self-report as proof of completion. **Why now:** Teams increasingly delegate multi-step work through APIs and internal services, making small interface assumptions consequential. The evaluation makes those assumptions visible in concrete email tasks rather than relying on general claims about model capability. **First implication:** Tool builders should make omissions and operation semantics explicit—for example, indicate when a list excludes messages—rather than treating documentation as the sole remedy. This is evidence about one vendor's email API, a particular set of tasks, agents, and harnesses; it does not establish that documentation produces the same lift across other tools or workloads. The excerpt also notes that docs made agents worse on one job, so the aggregate improvement does not imply uniform benefit. **First watch condition:** Look for independent evaluations across other APIs that report task-level outcomes and isolate documentation effects from harness, model, and interface changes. If that pattern holds, the durable lesson is not simply to attach more docs: it is to design tool surfaces whose observable behavior reduces the amount an agent must infer. Editor: [Andrey Reshetnikov](https://newruntime.com/owner-profile.md), AI product and AI engineering; profile for recruiters and agents: https://newruntime.com/owner-profile.json ## Continue in the graph Next records for an agent, linked to their Markdown twins. Replace `.md` with `.json` for structured fields, including `next_reads`. - topic hub: [Coding Agents](https://newruntime.com/topics/coding-agents.md): Explore the coding-agents topic hub. - collection: [Field Notes](https://newruntime.com/posts.json): Open the latest editorial analysis. - collection: [Source Ledger](https://newruntime.com/source-ledger.json): Inspect the public source evidence graph. - whole graph: [discovery graph](https://newruntime.com/discovery-graph.json) (typed nodes and edges) and [site index](https://newruntime.com/site-index.md)