---
type: "post"
slug: "cloudflare-bot-preference-sync"
title: "Cloudflare Bot Preference Sync"
description: "Cloudflare's Bot Preference Sync writes a site's AI bot configuration into its robots.txt, closing the gap between stated crawler preferences and edge-enforced rules."
retrieval_nugget: "Cloudflare's Bot Preference Sync writes a site's AI bot configuration into its robots.txt, closing the gap between stated crawler preferences and edge-enforced rules."
published_at: "2026-08-31"
updated_at: "2026-09-12"
record_date: "2026-08-21"
date_kind: "published_at"
topics: ["agent-security","agents","ai"]
entities: ["Cloudflare"]
source_url: "https://blog.cloudflare.com/bot-preference-sync"
source_title: "Say it once: introducing Bot Preference Sync"
source_domain: "blog.cloudflare.com"
source_terms: ["Cloudflare","Preference","Sync","your","content"]
summary_word_count: 190
schema_version: "newruntime-agent-readable-v0.2"
stable_id: "post:cloudflare-bot-preference-sync"
status: "published"
source_urls: ["https://blog.cloudflare.com/bot-preference-sync"]
visuals: []
editorial_provenance: {"schema_version":"newruntime-editorial-copy-v1","content_status":"source_grounded_final","final_copy_sha256":"sha256:7a679d7bc2eea64b4624c1f2608fddc1326b2a7583b6643b9e7dbbfd7f72ffd4","reviewed_at":"2026-09-12T10:00:00Z","source_evidence_count":1,"verified_claim_count":2,"site_analysis_schema_version":"newruntime-site-analysis-v1","site_object_kind":"field_note","observed_fact_count":2,"implication_count":1,"watch_condition_count":1,"related_record_count":3}
analysis: {"schema_version":"newruntime-site-analysis-v1","object_kind":"field_note","thesis":"Cloudflare announced Bot Preference Sync on 21 August, available from the Free tier to Enterprise.","observed_facts":[{"text":"Bot Preference Sync updates a site's robots.txt to reflect the AI bot configuration already set in Cloudflare.","source_urls":["https://blog.cloudflare.com/bot-preference-sync"]},{"text":"On 1 July 2026 Cloudflare separated AI bot handling into Search, Agent and Training traffic categories.","source_urls":["https://blog.cloudflare.com/bot-preference-sync"]}],"mechanism":"The mechanism is synchronisation rather than a new control. Cloudflare had already shipped a managed robots.txt aimed at a fixed list of training crawlers plus edge-enforced blocks, and on 1 July 2026 it separated the cases operators actually care about into Search, Agent and Training traffic.","why_now":"The problem it targets is a mismatch operators create by accident: robots.txt may disallow a crawler while the enforcement rules do not actually block it, and some crawlers treat that disagreement as licence to disregard the stated preference or to probe the enforced one.","implications":["Bot Preference Sync stops those two layers drifting apart, so one configuration produces both the declaration and the enforcement."],"evidence_boundary":"The announcement describes the product and its availability, not measured crawler compliance.","watch_conditions":["The condition to watch is whether major crawlers change behaviour when preference and enforcement agree, because a synchronised robots.txt is only worth the traffic it actually redirects."],"related_records":[{"url":"https://newruntime.com/posts/cloudflare-turns-mcp-traffic-into-a-network-control-surface","relation":"Prior coverage of Cloudflare turning agent traffic into a control surface."},{"url":"https://newruntime.com/posts/knownagents-turns-ai-bot-identity-into-an-authentication-boundary-against-sp","relation":"Prior coverage of bot identity as an authentication boundary."},{"url":"https://newruntime.com/topics/agent-security","relation":"Topic hub for crawler and agent access policy."}]}
routes: {"html":"https://newruntime.com/posts/cloudflare-bot-preference-sync/","markdown":"https://newruntime.com/posts/cloudflare-bot-preference-sync.md","json":"https://newruntime.com/posts/cloudflare-bot-preference-sync.json"}
editor: {"role":"editor","name":"Andrey Reshetnikov","profile":"https://newruntime.com/owner-profile.md","profile_json":"https://newruntime.com/owner-profile.json","same_as":["https://www.linkedin.com/in/reshetnikov1/","https://t.me/lapetuse"]}
---

# Cloudflare Bot Preference Sync

## Retrieval answer

Cloudflare's Bot Preference Sync writes a site's AI bot configuration into its robots.txt, closing the gap between stated crawler preferences and edge-enforced rules.

Cloudflare announced Bot Preference Sync on 21 August, available from the Free tier to Enterprise. It updates a site's robots.txt to reflect the AI bot configuration already set in Cloudflare, and it can be switched on or off at any time. The problem it targets is a mismatch operators create by accident: robots.txt may disallow a crawler while the enforcement rules do not actually block it, and some crawlers treat that disagreement as licence to disregard the stated preference or to probe the enforced one.

The mechanism is synchronisation rather than a new control. Cloudflare had already shipped a managed robots.txt aimed at a fixed list of training crawlers plus edge-enforced blocks, and on 1 July 2026 it separated the cases operators actually care about into Search, Agent and Training traffic. Bot Preference Sync stops those two layers drifting apart, so one configuration produces both the declaration and the enforcement.

This turns robots.txt from a static file into generated output of a policy, which is the same direction as Cloudflare [turning MCP traffic into a network control surface](https://newruntime.com/posts/cloudflare-turns-mcp-traffic-into-a-network-control-surface/) and as identity-based approaches such as [KnownAgents treating bot identity as an authentication boundary](https://newruntime.com/posts/knownagents-turns-ai-bot-identity-into-an-authentication-boundary-against-sp/). For a site that wants agent traffic, the declaration is now part of the [agent security](https://newruntime.com/topics/agent-security/) surface rather than a courtesy note.

The announcement describes the product and its availability, not measured crawler compliance. The condition to watch is whether major crawlers change behaviour when preference and enforcement agree, because a synchronised robots.txt is only worth the traffic it actually redirects.

Editor: [Andrey Reshetnikov](https://newruntime.com/owner-profile.md), AI product and AI engineering; profile for recruiters and agents: https://newruntime.com/owner-profile.json

## Continue in the graph

Next records for an agent, linked to their Markdown twins. Replace `.md` with `.json` for structured fields, including `next_reads`.

- topic hub: [Agent Security](https://newruntime.com/topics/agent-security.md): Explore the agent-security topic hub.
- topic hub: [Agents](https://newruntime.com/topics/agents.md): Explore the agents topic hub.
- topic hub: [Ai](https://newruntime.com/topics/ai.md): Explore the ai topic hub.
- collection: [Field Notes](https://newruntime.com/posts.json): Open the latest editorial analysis.
- collection: [Source Ledger](https://newruntime.com/source-ledger.json): Inspect the public source evidence graph.
- whole graph: [discovery graph](https://newruntime.com/discovery-graph.json) (typed nodes and edges) and [site index](https://newruntime.com/site-index.md)