Raw signalSep 18, 20261 min read1 source

Cloudflare lets sites disallow AI training without leaving search

In one minute

Cloudflare's Disallow AI Training setting refuses training by mixed-use crawlers while Applebot, Bingbot, and Googlebot keep indexing for search. Cloudflare introduced a Disallow AI Training setting for its bot controls. It publishes a no-training preference in robots.txt through Bot Preference Sync, keeps Accountable mixed-use crawlers such as Applebot, Bingbot, and Googlebot allowed for search, and blocks every other training crawler.

Cloudflare introduced a Disallow AI Training setting for its bot controls. It publishes a no-training preference in robots.txt through Bot Preference Sync, keeps Accountable mixed-use crawlers such as Applebot, Bingbot, and Googlebot allowed for search, and blocks every other training crawler. Apple, Google, and Microsoft honor the setting or have committed to honor it within a stated time frame. Cloudflare reports that fewer than 1% of its sites block search bots, while 17% use some mechanism to block training.

The Accountable designation requires a training opt-out, an opt-out for AI summaries, URL-level visibility into which pages were used for training, and an assurance that opting out of training does not affect search. Block and Block on pages with ads now also stop mixed-use crawlers, which removes search as well; Block AI Bots and Managed Robots.txt are being deprecated in favor of the Search, Training, and Agent controls. Control over how much content appears in AI summaries is a goal for early next year, not a shipped feature.

Evidence / sources

  1. 1blog.cloudflare.com/accountable-mixed-use-ai-crawlers
All Signals

Next reads

Read next

Open the graph
  1. 01topicSecurityExplore the security topic hub.
  2. 02archiveSignalsOpen the signals and the raw-signal archive.
  3. 03sourcesSourcesSee the original sources this archive cites.
Who read this page?Machine requests, hidden until opened

Loading the privacy-safe route aggregate...

Open the JSON contract