Cloudflare introduced a Disallow AI Training setting for its bot controls. It publishes a no-training preference in robots.txt through Bot Preference Sync, keeps Accountable mixed-use crawlers such as Applebot, Bingbot, and Googlebot allowed for search, and blocks every other training crawler. Apple, Google, and Microsoft honor the setting or have committed to honor it within a stated time frame. Cloudflare reports that fewer than 1% of its sites block search bots, while 17% use some mechanism to block training.
The Accountable designation requires a training opt-out, an opt-out for AI summaries, URL-level visibility into which pages were used for training, and an assurance that opting out of training does not affect search. Block and Block on pages with ads now also stop mixed-use crawlers, which removes search as well; Block AI Bots and Managed Robots.txt are being deprecated in favor of the Search, Training, and Agent controls. Control over how much content appears in AI summaries is a goal for early next year, not a shipped feature.