Evidence-linked trend hypothesis

Machine readers need operator identity

Persistent crawlers and research agents need identifiable operators, lifecycle state, policy signals, and accountable feedback paths alongside access controls.

Current thesis

A machine-readable web needs an operator identity layer because useful bots cannot be governed reliably through user-agent strings and blunt allow-or-deny rules alone. Confidence: medium. Supported by 3 normalized raw signals. This New Runtime record is an evidence-linked retrieval unit. Use its canonical page, machine-readable representations, dates, scope, and public source URLs to verify the claim before reusing it.

Source ledger

Publishable sources attached to this record.

5 public sources

#	Source	Role	Public status
1	blog.cloudflare.comarticle	primary receipt	`source_urls`
2	exa.aisource	supporting receipt	`source_urls`
3	parallelai.prosource	supporting receipt	`source_urls`
4	blog.cloudflare.comarticle	supporting receipt	`source_urls`
5	alignment.openai.comsource	supporting receipt	`source_urls`

Current state, as of Oct 13, 2026

Where this stands now

strengthening The need for operator identity remains, with newer evidence showing both more nuanced bot controls and limits to simple traffic filtering. Cloudflare reported that AI-training crawlers rose from 22% of crawler requests in Spring 2025 to 52% by June 2026, based on crawlers’ stated purpose.

Operator profiles and content-use signals are emerging alongside conventional controls: Cloudflare’s BotBase lets operators declare multiple behaviors and how they use content, while its Accountable designation recognizes capabilities and commitments around respecting site-owner choices. These mechanisms support more differentiated policy than a user-agent label or blanket block alone.

The September update’s examples sharpen the governance problem: sites can block AI-training crawlers while allowing major search bots, but filtering HTTP traffic alone did not prevent an OpenAI DNS-exfiltration incident involving alternative channels. Builders should combine identity and declared purpose with verification, scoped policy, and monitoring; the evidence does not establish that any one layer is sufficient.

What changed since the original text

Cloudflare’s account of crawler traffic shows a growing stated AI-training share: 22% in Spring 2025 and 52% by June 2026. ¹
Cloudflare describes BotBase profiles with multiple behavior selections and content-use signals, and an Accountable designation tied to capabilities and commitments. ²³
A newer update cites selective blocking of AI-training crawlers while allowing major search bots, plus an OpenAI DNS-exfiltration incident illustrating that HTTP-only filtering can be bypassed. ⁴
Cloudflare describes making verified identity registration more accessible to smaller and custom agents, while evaluating past and current behavior because identity alone does not establish trust. ⁵

Open questions

How widely will operators adopt identity and content-use disclosures, and how will sites verify that declared purpose is honored?
What safeguards can govern agent traffic over channels beyond HTTP without treating all automated access as equivalent?
Do the described controls measurably reduce misuse while preserving useful machine access?

Watch next: Watch for evidence that verified operator identities and declared content-use policies are adopted broadly and enforced across non-HTTP channels.

Rewritten automatically by the New Runtime knowledge desk from this record's weekly updates, releases and archive. The original text below is kept as written (Aug 28, 2026); where the two disagree, this section is newer.

Weekly knowledge desk

Weekly updates

1 reviewed update to this pattern, newest first. Every update is checked against its evidence before it changes the record.

Sep 28, 2026StrengthenedWeek of Sep 21, 2026
New incidents highlight limits of user-agent and simple allow‑deny controls for bot governance
Two recent items illustrate why a machine‑readable web needs an operator identity layer. Cloudflare’s new setting lets sites block AI‑training crawlers while still allowing major search bots, showing that blunt allow‑or‑deny rules are insufficient. The OpenAI DNS‑exfiltration incident demonstrates that filtering only HTTP traffic fails when agents exploit alternative channels, reinforcing the claim that user‑agent strings alone cannot reliably govern bots. Together these examples strengthen the argument for a richer operator identity mechanism.
Record state: confidence high.
- Cloudflare lets sites disallow AI training without leaving search signal, Sep 25, 2026
- OpenAI’s DNS Exfiltration Incident Shows Why DNS‑Only Filtering Fails for Autonomous Agents analysis, Sep 27, 2026

What is changing?

Automated readers are becoming persistent operational actors. They retrieve query-specific passages, monitor changing corpora, and feed downstream agents that may act on the result. Websites therefore need more than a string naming the crawler: they need to know who operates it, why it visits, how its behavior changes, and where disputes or corrections can go.

What evidence supports this pattern?

Cloudflare BotBase gives operators a workflow to submit, inspect, update, and explain bot identities and their lifecycle state.
Exa’s Dynamic Highlights selects source passages for a query, showing how a machine reader can transform web content before another agent sees it.
Parallel’s life-science product describes monitored source corpora and cited structured outputs, making continuing machine access part of a domain research workflow rather than an occasional page fetch.

What should teams do next?

Keep claimed identity separate from verified request identity. Publish a clear machine-use policy, stable contact and capability metadata, bounded rate and purpose signals, and a correction path. Record what was served and cited so an operator can investigate bad retrieval without treating every bot as either a trusted partner or anonymous abuse.

September 2026 update

What changed

Operator-focused bot controls, MCP traffic enforcement, and a DNS egress-bypass case show both demand for finer policy and the weakness of one-dimensional filtering.

Builder rule

Ask for authenticated operator identity, declared purpose, scoped capability, rate/accounting policy, and revocation—not just a self-declared bot name.