---
type: "post"
id: "understanding-and-mitigating-role-spoofing-in-llms-8df394"
slug: "understanding-and-mitigating-role-spoofing-in-llms-8df394"
title: "Understanding and Mitigating Role Spoofing in LLMs"
description: "A paper submitted to arXiv on October 29, 2026 studies role spoofing: malicious text in a tool output can make a language model attribute that text to the user."
retrieval_nugget: "A paper submitted to arXiv on October 29, 2026 studies role spoofing: malicious text in a tool output can make a language model attribute that text to the user. The authors analyze the cues behind this failure and propose activation-based probes as a detector, giving agent builders a specific failure mode to test for—without establishing a production-ready defense."
published_at: "2026-11-01"
updated_at: "2026-11-01"
record_date: "2026-11-01"
date_kind: "published_at"
topics: ["agent-security","ai-security"]
entities: ["Ezra Edelman","Benjamin Edelman","Surbhi Goel","Eran Malach","arXiv"]
related_patterns: ["agent-security-moves-to-runtime-boundaries"]
novelty_label: "new_fact"
what_changed: "This UPenn paper adds evidence that combined fake structure and content enable role spoofing, finds adversarial training leaves structure-based attacks, and proposes linear-probe detection."
prior_refs: []
source_url: "https://arxiv.org/abs/2610.40396"
source_urls: ["https://arxiv.org/abs/2610.40396"]
schema_version: "newruntime-agent-readable-v0.2"
stable_id: "post:understanding-and-mitigating-role-spoofing-in-llms-8df394"
status: "published"
visuals: []
editor: {"role":"editor","name":"Andrey Reshetnikov","profile":"https://newruntime.com/owner-profile.md","profile_json":"https://newruntime.com/owner-profile.json","same_as":["https://www.linkedin.com/in/reshetnikov1/","https://t.me/lapetuse"]}
routes: {"html":"https://newruntime.com/posts/understanding-and-mitigating-role-spoofing-in-llms-8df394/","markdown":"https://newruntime.com/posts/understanding-and-mitigating-role-spoofing-in-llms-8df394.md","json":"https://newruntime.com/posts/understanding-and-mitigating-role-spoofing-in-llms-8df394.json"}
markdown_url: "https://newruntime.com/posts/understanding-and-mitigating-role-spoofing-in-llms-8df394.md"
---

# Understanding and Mitigating Role Spoofing in LLMs

## Retrieval answer

A paper submitted to arXiv on October 29, 2026 studies role spoofing: malicious text in a tool output can make a language model attribute that text to the user. The authors analyze the cues behind this failure and propose activation-based probes as a detector, giving agent builders a specific failure mode to test for—without establishing a production-ready defense.

## Summary
A paper submitted to arXiv on October 29, 2026 studies role spoofing: malicious text in a tool output can make a language model attribute that text to the user. The authors analyze the cues behind this failure and propose activation-based probes as a detector, giving agent builders a specific failure mode to test for—without establishing a production-ready defense.

## What happened
Ezra Edelman, Benjamin Edelman, Surbhi Goel, and Eran Malach describe role spoofing in their paper, “Understanding and Mitigating Role Spoofing in LLMs,” submitted to arXiv on October 29, 2026. The work focuses on a form of prompt injection in which malicious text in a tool output—for example, a webpage retrieved by an agent—can impersonate the user. Rather than merely persuading a model to follow an instruction, the attack exploits confusion about who said something.

The authors investigate how models track “who said what” in a conversation. They report that models use both content cues, including semantics and writing style, and structural cues, including control tokens and formatting. They find that attacks combining user-like content with fake structure can be effective at role spoofing. This matters because a tool result is supposed to be data from an external source, not a message carrying the user’s authority; if a model assigns it the wrong role, downstream behavior may change.

The paper also examines a toy setting for adversarial training. Training that teaches a model to ignore content-based injections mitigates those attacks in the setup studied, but leaves the model susceptible to structure-based attacks. The authors suggest this distinction may help explain why frontier models that resist many injection attempts can still remain vulnerable to role spoofing. The supplied source does not quantify a general attack success rate across deployed systems, nor does it establish that all frontier models behave this way.

As a mitigation, the researchers propose linear probes trained to predict roles in a conversation from model activations. The detector aims to identify cases where the model internally treats tool output as if it came from the user. The paper’s abstract says the method is effective across different settings. The accompanying source summary says probes trained on random-token conversations separated clean and spoofed examples almost perfectly, with low overhead and fewer false positives than prior methods. Those results describe the study’s experiments, not a demonstrated production deployment. The supplied material gives no product release, pricing, user access terms, or operational integration details.

## What the archive adds
The earlier Telegram post from June 28 described the broader problem of models confusing instructions with data and the intended authority boundaries between system, user, tool, and assistant messages. This paper is a new research result on that thread: it isolates tool-output role impersonation and studies an activation-based detection approach. The archive contains no directly related Field Note.

## Trend signal
This research supports [Agent security moves to runtime boundaries](https://newruntime.com/patterns/agent-security-moves-to-runtime-boundaries/): the problem is not just what an instruction says, but what the model is allowed to treat as authoritative input. The proposed probe is a research method, not evidence that a runtime boundary or deployable safeguard is already available.

## Connections to other events
An earlier Telegram post discussed role confusion between instructions and data, while another described role drift in compound LLM pipelines. This paper narrows the concern to a specific attack path: external tool text that the model attributes to the user. Its proposed probes examine internal role assignment, complementing those earlier concerns without showing that the same mechanism or mitigation applies to every pipeline.

## Why it matters
Teams building agents often treat retrieved pages, tool responses, and user messages as separate categories in their orchestration code. This paper is a reminder that correct labeling outside the model does not guarantee correct attribution inside it. A model may use wording and formatting together to infer a speaker, so an attacker who controls tool content may try to imitate both.

For builders, the immediate value is a sharper test plan: test tool outputs that mimic user language and conversation structure, not only pages containing obvious malicious instructions. The proposed activation probes also point to a possible way to detect when internal attribution diverges from the external role label. But the evidence supplied is a research result, not proof of a robust defense across models, agent stacks, or real-world traffic. Teams should treat it as a threat-model input and evaluate mitigations in their own systems.

## Limits and uncertainty
The source describes experiments and a toy adversarial-training setting; it does not establish the prevalence of role spoofing in production agents or the degree of risk for any particular model or deployment. The supplied material does not name evaluated frontier models, provide a broad benchmark, or specify a production-ready implementation of the probes. “Effective across different settings” and near-perfect separation in the described random-token setup do not establish reliable performance on realistic tool outputs. The paper also does not show that probes prevent an attack; they are presented as a detection method. Operational costs, deployment requirements, and effects on agent behavior remain unclear.

## Reader takeaway
A tool result can be dangerous not only because of the instruction it contains, but because the model may mistake its speaker. Test for forged roles and formatting in external content as well as for ordinary prompt injection. The proposed activation probes are promising research, not a substitute for validating defenses in the actual agent stack.

## Open question
How well do activation probes detect role spoofing in realistic, long-running agent conversations across different model families?

Editor: [Andrey Reshetnikov](https://newruntime.com/owner-profile.md), AI product and AI engineering; profile for recruiters and agents: https://newruntime.com/owner-profile.json

## Continue in the graph

Next records for an agent, linked to their Markdown twins. Replace `.md` with `.json` for structured fields, including `next_reads`.

- topic hub: [Agent Security](https://newruntime.com/topics/agent-security.md): Explore the agent-security topic hub.
- topic hub: [Ai Security](https://newruntime.com/topics/ai-security.md): Explore the ai-security topic hub.
- collection: [Field Notes](https://newruntime.com/posts.json): Open the latest editorial analysis.
- collection: [Sources](https://newruntime.com/sources/): See the original sources this archive cites.
- whole graph: [discovery graph](https://newruntime.com/discovery-graph.json) (typed nodes and edges) and [site index](https://newruntime.com/site-index.md)