Cloudflare announced Bot Preference Sync on 21 August, available from the Free tier to Enterprise. It updates a site's robots.txt to reflect the AI bot configuration already set in Cloudflare, and it can be switched on or off at any time. The problem it targets is a mismatch operators create by accident: robots.txt may disallow a crawler while the enforcement rules do not actually block it, and some crawlers treat that disagreement as licence to disregard the stated preference or to probe the enforced one.
The mechanism is synchronisation rather than a new control. Cloudflare had already shipped a managed robots.txt aimed at a fixed list of training crawlers plus edge-enforced blocks, and on 1 July 2026 it separated the cases operators actually care about into Search, Agent and Training traffic. Bot Preference Sync stops those two layers drifting apart, so one configuration produces both the declaration and the enforcement.
This turns robots.txt from a static file into generated output of a policy, which is the same direction as Cloudflare turning MCP traffic into a network control surface and as identity-based approaches such as KnownAgents treating bot identity as an authentication boundary. For a site that wants agent traffic, the declaration is now part of the agent security surface rather than a courtesy note.
The announcement describes the product and its availability, not measured crawler compliance. The condition to watch is whether major crawlers change behaviour when preference and enforcement agree, because a synchronised robots.txt is only worth the traffic it actually redirects.