{"type":"post","id":"giving-coding-agents-a-docs-link-doubles-task-630606","slug":"giving-coding-agents-a-docs-link-doubles-task-630606","title":"Giving coding agents a docs link doubles task completion from 41% to 84%.","description":"**Thesis:** Agent success depends not only on access to documentation, but on whether tools make their behavior and omissions legible.","retrieval_nugget":"**Thesis:** Agent success depends not only on access to documentation, but on whether tools make their behavior and omissions legible.","published_at":"2026-10-15","updated_at":"2026-10-15","record_date":"2026-10-15","date_kind":"published_at","topics":["coding-agents"],"source_urls":["https://www.agentmail.to/evals/report"],"schema_version":"newruntime-agent-readable-v0.2","stable_id":"post:giving-coding-agents-a-docs-link-doubles-task-630606","status":"published","visuals":[],"editorial_provenance":{"schema_version":"newruntime-editorial-copy-v1","content_status":"source_grounded_final","final_copy_sha256":"sha256:aae1c297f3e09c7b0f74409f5dae881723fe4cb3645b73555e39d2c0cf5d8e0f","reviewed_at":"2026-10-15T16:09:22.316Z","source_evidence_count":1,"verified_claim_count":2,"site_analysis_schema_version":"newruntime-site-analysis-v1","site_object_kind":"field_note","observed_fact_count":2,"implication_count":1,"watch_condition_count":1,"related_record_count":0},"analysis":{"schema_version":"newruntime-site-analysis-v1","object_kind":"field_note","thesis":"Agent success depends not only on access to documentation, but on whether tools make their behavior and omissions legible.","observed_facts":[{"text":"AgentMail reports running 25 agents across seven tasks, with and without links to its documentation.","source_urls":["https://www.agentmail.to/evals/report"]},{"text":"Its executive summary says documentation links lifted success from 76% to 84%, and the report says many failures involved hidden inbox messages, endpoints whose behavior differed from expectations, or conflicting labels.","source_urls":["https://www.agentmail.to/evals/report"]}],"mechanism":"Documentation can help an agent discover the right operation, but it cannot reliably compensate for an interface that conceals filtered-out data or signals behavior ambiguously.","why_now":"Teams increasingly delegate multi-step work through APIs and internal services, making small interface assumptions consequential.","implications":["Tool builders should make omissions and operation semantics explicit—for example, indicate when a list excludes messages—rather than treating documentation as the sole remedy."],"evidence_boundary":"This is evidence about one vendor's email API, a particular set of tasks, agents, and harnesses; it does not establish that documentation produces the same lift across other tools or workloads. The excerpt also notes that docs made agents worse on one job, so the aggregate improvement does not imply uniform benefit.","watch_conditions":["Look for independent evaluations across other APIs that report task-level outcomes and isolate documentation effects from harness, model, and interface changes."],"related_records":[],"new_branch_reason":"No prior New Runtime analytical record was supplied; this evaluation starts a distinct record on tool legibility and documentation's measured effect in agent task completion."},"editor":{"role":"editor","name":"Andrey Reshetnikov","profile":"https://newruntime.com/owner-profile.md","profile_json":"https://newruntime.com/owner-profile.json","same_as":["https://www.linkedin.com/in/reshetnikov1/","https://t.me/lapetuse"]},"routes":{"html":"https://newruntime.com/posts/giving-coding-agents-a-docs-link-doubles-task-630606/","markdown":"https://newruntime.com/posts/giving-coding-agents-a-docs-link-doubles-task-630606.md","json":"https://newruntime.com/posts/giving-coding-agents-a-docs-link-doubles-task-630606.json"},"next_reads":[{"type":"topic","path":"/topics/coding-agents/","url":"https://newruntime.com/topics/coding-agents/","title":"Coding Agents","reason":"Explore the coding-agents topic hub.","media_type":"text/html","markdown_url":"https://newruntime.com/topics/coding-agents.md","json_url":"https://newruntime.com/topics/coding-agents.json"},{"type":"collection","path":"/posts/","url":"https://newruntime.com/posts/","title":"Field Notes","reason":"Open the latest editorial analysis.","media_type":"text/html","json_url":"https://newruntime.com/posts.json"},{"type":"collection","path":"/source-ledger.json","url":"https://newruntime.com/source-ledger.json","title":"Source Ledger","reason":"Inspect the public source evidence graph.","media_type":"application/json"}]}