Observed facts: AgentMail reports running 25 agents across seven tasks, with and without links to its documentation. Observed facts: Its executive summary says documentation links lifted success from 76% to 84%, and the report says many failures involved hidden inbox messages, endpoints whose behavior differed from expectations, or conflicting labels. The headline's 41% figure is not corroborated by the supplied report excerpt.
Mechanism: Documentation can help an agent discover the right operation, but it cannot reliably compensate for an interface that conceals filtered-out data or signals behavior ambiguously. The report's task grader examined captured requests and outputs rather than accepting an agent's self-report as proof of completion.
Why now: Teams increasingly delegate multi-step work through APIs and internal services, making small interface assumptions consequential. The evaluation makes those assumptions visible in concrete email tasks rather than relying on general claims about model capability.
First implication: Tool builders should make omissions and operation semantics explicit—for example, indicate when a list excludes messages—rather than treating documentation as the sole remedy.
This is evidence about one vendor's email API, a particular set of tasks, agents, and harnesses; it does not establish that documentation produces the same lift across other tools or workloads. The excerpt also notes that docs made agents worse on one job, so the aggregate improvement does not imply uniform benefit.
First watch condition: Look for independent evaluations across other APIs that report task-level outcomes and isolate documentation effects from harness, model, and interface changes.
If that pattern holds, the durable lesson is not simply to attach more docs: it is to design tool surfaces whose observable behavior reduces the amount an agent must infer.