---
type: "post"
id: "agentmail-follow-up-all-ten-claude-setups-finished-02f51b"
slug: "agentmail-follow-up-all-ten-claude-setups-finished-02f51b"
title: "AgentMail follow-up: all ten Claude setups finished in the bottom eleven of its agent evals"
description: "AgentMail reports that all ten tested Claude model-and-harness setups ranked in the bottom eleven of 25 email-agent configurations."
retrieval_nugget: "AgentMail reports that all ten tested Claude model-and-harness setups ranked in the bottom eleven of 25 email-agent configurations."
published_at: "2026-10-28"
updated_at: "2026-10-28"
record_date: "2026-10-28"
date_kind: "published_at"
topics: ["claude","claude-code","evals"]
entities: ["AgentMail","AgentMail Evals","Anthropic","Claude Sonnet 5","Claude Code","Hermes","OpenClaw","GPT-5.5"]
related_patterns: ["agent-security-moves-to-runtime-boundaries","harness-architecture-outlives-model-choice","verification-bandwidth-is-the-scarce-resource"]
novelty_label: "new_fact"
what_changed: "The follow-up quantifies Claude’s results: all ten tested setups ranked near the bottom, with a 63% run-completion rate versus 90% for other model families, largely due to defaults around flagged spam."
prior_refs: ["memory:319c95b6-540f-8f83-ac2e-af55fe4de70b"]
source_url: "https://www.agentmail.to/blog/why-claude-came-last-ai-agent-evals"
source_urls: ["https://www.agentmail.to/blog/why-claude-came-last-ai-agent-evals"]
schema_version: "newruntime-agent-readable-v0.2"
stable_id: "post:agentmail-follow-up-all-ten-claude-setups-finished-02f51b"
status: "published"
visuals: []
editor: {"role":"editor","name":"Andrey Reshetnikov","profile":"https://newruntime.com/owner-profile.md","profile_json":"https://newruntime.com/owner-profile.json","same_as":["https://www.linkedin.com/in/reshetnikov1/","https://t.me/lapetuse"]}
routes: {"html":"https://newruntime.com/posts/agentmail-follow-up-all-ten-claude-setups-finished-02f51b/","markdown":"https://newruntime.com/posts/agentmail-follow-up-all-ten-claude-setups-finished-02f51b.md","json":"https://newruntime.com/posts/agentmail-follow-up-all-ten-claude-setups-finished-02f51b.json"}
---

# AgentMail follow-up: all ten Claude setups finished in the bottom eleven of its agent evals

## Retrieval answer

AgentMail reports that all ten tested Claude model-and-harness setups ranked in the bottom eleven of 25 email-agent configurations.

## Summary

AgentMail reports that all ten tested Claude model-and-harness setups ranked in the bottom eleven of 25 email-agent configurations. The reported gap was concentrated in spam-flagged email: Claude often found it, but usually declined to reply. For builders, the useful result is not a model league table; it is a reminder to put send permissions and approval rules in the email system rather than rely on a model’s implicit caution.

## What happened

In an October 27 post, AgentMail published a follow-up analysis of its AgentMail Evals, a benchmark for agents using an email tool they have not memorized. The company says it ran 25 combinations of models and harnesses across seven everyday inbox jobs, including retrieving confirmation codes, responding to password resets, saving attachments, and summarizing a customer’s week. Each run used a new inbox and a stated goal; the benchmark graded whether the setup completed the work.

A harness, in AgentMail’s framing, is the software surrounding the model that determines its available tools. The evaluated combinations included harnesses such as Claude Code, Codex, Hermes, OpenCode, and OpenClaw. AgentMail tested four Claude models across four harnesses, producing ten Claude configurations in its table. All ten landed in the bottom eleven of 25. The strongest reported Claude result was Claude Sonnet 5 in Hermes, at 68.9 out of 100 and rank 15. The lowest was Claude Sonnet 5 in OpenClaw, at 38.1 and rank 25. AgentMail says other model families completed 90% of runs overall, compared with 63% for Claude.

The company attributes most of that difference to two tasks requiring a reply to mail marked as spam. Four of the seven jobs hid the relevant email either in spam or behind an authentication filter, so a normal inbox listing was empty. On jobs that required only reading hidden mail, AgentMail reports Claude completed 86% of runs, near the other models. On hidden-email jobs requiring a reply, it reports 24% completion for Claude, versus results of 81% to 96% for the other listed model families.

One task involved a password reset whose reply address pointed elsewhere, and another involved signup confirmations where the prompt explicitly authorized action. AgentMail says Claude generally refused both; other models generally acted on both. On those two jobs, its figures show Claude at 23% completion and all other families together at 88%. The source does not characterize either response as universally correct: a refusal can leave work unfinished, while an erroneous reply cannot be recalled. Its conclusion is that this trade-off should be an explicit system policy, not an unconfigurable model tendency.

AgentMail recommends enforcing the policy through its infrastructure: allowlists for permitted senders and domains, and drafts when a person must approve a message. It argues that an unapproved recipient should cause a send to fail regardless of whether a GPT or Claude model drafted the message. The post also notes other recurring failures, such as agents stopping after an empty inbox listing rather than requesting spam or unauthenticated mail, and agents assuming a `reply-all` endpoint can perform functions it cannot.

## What the archive adds

This is a new evaluation result, not a restatement of an earlier Field Note. It closely extends [a Field Note on reviewing agent sessions for security issues](https://newruntime.com/posts/how-to-review-your-agents-for-security-issues-63fda7/): that note focused on post-run evidence and human review, while AgentMail’s result emphasizes preventing an unsafe send at the execution boundary. [A note on documentation access and coding-agent completion](https://newruntime.com/posts/giving-coding-agents-a-docs-link-doubles-task-630606/) is also relevant because both cases show that apparent model capability depends on the operational context around the task.

## Trend signal

The result supports [Agent security moves to runtime boundaries](https://newruntime.com/patterns/agent-security-moves-to-runtime-boundaries/). The reported failure is not simply that a model made a bad judgment: the benchmark exposes an action with different costs for false approval and false refusal. AgentMail’s proposed response is therefore a permissioned runtime control—recipient allowlists and approval-gated drafts—rather than a stronger instruction asking the model to be careful. The source also makes clear that the correct policy can differ by team and workflow.

## Connections to other events

The archive’s earlier security-review note describes turning completed agent traces into ranked human review work. This evaluation addresses an earlier layer in the same operational chain: a send allowlist or draft requirement can block a disallowed action before a trace needs to be investigated. Together, the two approaches imply that review and enforcement are complementary rather than substitutes.

## Why it matters

An inbox agent has a consequential action surface: it can disclose information, communicate with an attacker-controlled address, or leave a legitimate customer request unresolved. A benchmark score can hide that distinction. Here, a lower completion rate partly reflects a more conservative default, while higher completion can include replies to a suspicious reset message. Teams should first decide which failures they can tolerate for each inbox and action class.

That decision should then become enforceable behavior. Separate read access from send authority; scope recipients and domains; require drafts for unfamiliar or high-risk recipients; and log the policy decision alongside the attempted action. Once those controls exist, model selection can focus more cleanly on retrieval, tool use, latency, and task completion. The source’s strongest practical point is that a model update may change behavior, but infrastructure policy can remain inspectable and stable.

## What happens next

- **AgentMail will publish additional results as models and harnesses change.** (3 months, 60%). Watch for: a new results table using the same jobs and grading rules.
- **More agent-email evaluations will separate read access from send authorization.** (6 months, 65%). Watch for: benchmarks that score policy compliance alongside completion.

## Limits and uncertainty

This is a report from AgentMail about its own benchmark and product environment, not an independent cross-platform evaluation. The supplied material describes seven jobs and 25 model-harness combinations, but does not establish how results transfer to another email API, different prompts, different policies, or production inboxes. Two Claude models, Fable 5.1 and Opus 5.5, were run about three times per task rather than ten, which AgentMail says makes their scores less certain. The article also reports aggregate completion, not a complete taxonomy of false approvals, false refusals, or the real-world harm prevented by each outcome.

## Skill to build now

Create a one-page action policy for any agent that can send email. This week, classify recipient domains and actions into three groups: automatically allowed, draft-for-review, and blocked. Implement the classification in the mail layer rather than only in agent instructions, then test it with a benign approval request and a suspicious redirect scenario. Record whether the agent found the message, attempted the action, and was stopped or permitted by the policy. That separates model behavior from the control that governs it.

## Reader takeaway

Do not interpret an email-agent benchmark solely as a measure of model intelligence. AgentMail’s reported Claude gap was largely a difference in what the system would do with spam-flagged mail. Decide that policy explicitly, enforce it outside the model, and then evaluate models on the work they are actually permitted to complete.

## Open question

Can future evaluations measure both task completion and the cost of unsafe versus overly cautious actions under a stated, configurable email policy?

Editor: [Andrey Reshetnikov](https://newruntime.com/owner-profile.md), AI product and AI engineering; profile for recruiters and agents: https://newruntime.com/owner-profile.json

## Continue in the graph

Next records for an agent, linked to their Markdown twins. Replace `.md` with `.json` for structured fields, including `next_reads`.

- topic hub: [Claude](https://newruntime.com/topics/claude.md): Explore the claude topic hub.
- topic hub: [Claude Code](https://newruntime.com/topics/claude-code.md): Explore the claude-code topic hub.
- topic hub: [Evals](https://newruntime.com/topics/evals.md): Explore the evals topic hub.
- collection: [Field Notes](https://newruntime.com/posts.json): Open the latest editorial analysis.
- collection: [Source Ledger](https://newruntime.com/source-ledger.json): Inspect the public source evidence graph.
- whole graph: [discovery graph](https://newruntime.com/discovery-graph.json) (typed nodes and edges) and [site index](https://newruntime.com/site-index.md)