---
type: "post"
id: "agentphishing-cmu-benchmark-measures-how-llm-email-agents-b197a6"
slug: "agentphishing-cmu-benchmark-measures-how-llm-email-agents-b197a6"
title: "AgentPhishing: CMU benchmark measures how LLM email agents respond to adaptive phishing attacks"
description: "Justin Tan, Miao Li, and Ding Zhao introduced AgentPhishing, an arXiv benchmark for testing LLM email agents against adaptive phishing attacks."
retrieval_nugget: "Justin Tan, Miao Li, and Ding Zhao introduced AgentPhishing, an arXiv benchmark for testing LLM email agents against adaptive phishing attacks."
published_at: "2026-10-24"
updated_at: "2026-10-24"
record_date: "2026-10-24"
date_kind: "published_at"
topics: ["evals","agent-security","prompt-injection"]
entities: ["AgentPhishing","PhishOpt","Justin Tan","Miao Li","Ding Zhao","arXiv"]
related_patterns: ["agent-security-moves-to-runtime-boundaries"]
novelty_label: "new_fact"
what_changed: "Introduces AgentPhishing, a benchmark of adaptive phishing against LLM email agents, reporting four of nine agents with detection below 10% and compliance above 45%."
prior_refs: ["memory:e6abccb2-b63d-8404-a06a-7d6fbb1e9ce2"]
source_url: "https://arxiv.org/abs/2610.31404"
source_urls: ["https://arxiv.org/abs/2610.31404"]
schema_version: "newruntime-agent-readable-v0.2"
stable_id: "post:agentphishing-cmu-benchmark-measures-how-llm-email-agents-b197a6"
status: "published"
visuals: []
editor: {"role":"editor","name":"Andrey Reshetnikov","profile":"https://newruntime.com/owner-profile.md","profile_json":"https://newruntime.com/owner-profile.json","same_as":["https://www.linkedin.com/in/reshetnikov1/","https://t.me/lapetuse"]}
routes: {"html":"https://newruntime.com/posts/agentphishing-cmu-benchmark-measures-how-llm-email-agents-b197a6/","markdown":"https://newruntime.com/posts/agentphishing-cmu-benchmark-measures-how-llm-email-agents-b197a6.md","json":"https://newruntime.com/posts/agentphishing-cmu-benchmark-measures-how-llm-email-agents-b197a6.json"}
---

# AgentPhishing: CMU benchmark measures how LLM email agents respond to adaptive phishing attacks

## Retrieval answer

Justin Tan, Miao Li, and Ding Zhao introduced AgentPhishing, an arXiv benchmark for testing LLM email agents against adaptive phishing attacks.

Justin Tan, Miao Li, and Ding Zhao introduced AgentPhishing, an arXiv benchmark for testing LLM email agents against adaptive phishing attacks. The paper was submitted on October 22, 2026.

The benchmark tests whether agents both detect persuasive phishing messages and refuse malicious requests. Its PhishOpt method optimizes reusable email-generation skills for individual agents under fixed benchmark and realism constraints. The authors also analyze attacks using Cialdini persuasion principles and construct a universal generation skill.

Across nine agents, four had phishing-detection rates below 10% and malicious-request compliance above 45%. The paper reports that agents can handle pre-existing phishing emails safely while remaining susceptible to attacks adapted to their behavior. It does not establish the security of specific deployed email products, real-world attack prevalence, or whether any mitigation eliminates the reported failures.

Source: https://arxiv.org/abs/2610.31404

Editor: [Andrey Reshetnikov](https://newruntime.com/owner-profile.md), AI product and AI engineering; profile for recruiters and agents: https://newruntime.com/owner-profile.json

## Continue in the graph

Next records for an agent, linked to their Markdown twins. Replace `.md` with `.json` for structured fields, including `next_reads`.

- topic hub: [Evals](https://newruntime.com/topics/evals.md): Explore the evals topic hub.
- topic hub: [Agent Security](https://newruntime.com/topics/agent-security.md): Explore the agent-security topic hub.
- topic hub: [Prompt Injection](https://newruntime.com/topics/prompt-injection.md): Explore the prompt-injection topic hub.
- collection: [Field Notes](https://newruntime.com/posts.json): Open the latest editorial analysis.
- collection: [Source Ledger](https://newruntime.com/source-ledger.json): Inspect the public source evidence graph.
- whole graph: [discovery graph](https://newruntime.com/discovery-graph.json) (typed nodes and edges) and [site index](https://newruntime.com/site-index.md)