What happened
On October 30, 2026, Google announced plans to support a shared AI-training control in robots.txt. The proposed mechanism is based on work from the AIPREF working group at the Internet Engineering Task Force (IETF). Google presents it as a way for website operators to express a preference about use of their content for AI training across multiple participating providers in a single step.
Today, a publisher that wants to express training preferences may need to maintain provider-specific instructions. Google has offered its own Google-Extended control since 2023. Under the planned shared approach, a website owner could use a common policy token in robots.txt to state a broad preference, while providers that choose to participate interpret that signal for their own training systems.
Google describes an important default for its implementation. If a site uses the shared token to opt out of training and does not explicitly specify Google-Extended, Google plans to treat the signal as though the site had disallowed Google-Extended. The company says its existing, company-specific controls will continue to operate alongside the shared signal. That leaves operators able to make a broad choice through the joint mechanism while still setting a distinct rule for Google.
That distinction matters because Google ties an explicit allowance of Google-Extended to eligibility for grounding in Gemini Apps: websites that specifically allow it can appear in responses and receive clicks through that grounding. An operator therefore cannot treat the new signal as a simple replacement for every existing directive without deciding whether it wants that Google-specific outcome.
Google says the proposal uses the IETF’s AI-training definition while work on the wider AIPREF vocabulary continues. It frames the initiative as a response to requests from publishers, civil-society groups, regulators, and other stakeholders for industry-wide training controls. Google also says the proposal is consistent with the EU Code of Practice for General Purpose AI Model Providers under the EU AI Act, which recognizes robots.txt as a primary means for publishers to express machine-readable preferences. Google aims to support the control alongside other providers in 2027 and says it will provide progress updates.
What the archive adds
This is a new fact on a known story. Our earlier Field Note, Making it easier for websites to express AI training preferences, recorded OpenAI’s statement that it would honor a shared robots.txt signal. Google now explains its own intended interpretation, its relationship to Google-Extended, and its 2027 target.
Trend signal
None of the current New Runtime patterns directly fits: this is a proposed web-governance convention, not yet evidence that a specific agent architecture, capability boundary, or operator-identity layer has become durable.
Why now
Google says the draft reflects calls for cross-industry controls from publishers, civil society, regulators, and other stakeholders. It also situates the work beside the EU GPAI code’s recognition of robots.txt as a primary machine-readable expression of publisher preferences, while the IETF process continues to refine the broader vocabulary.
Connections to other events
A September archive post described Cloudflare separating permission to search from permission to train AI, so a site could remain searchable while declining training use. Google’s planned interpretation makes the same separation operationally important at the directive level: a broad training opt-out can coexist with a deliberate Google-specific allowance for Gemini grounding.
Why it matters
For publishers, product teams, and infrastructure operators, the announcement makes robots.txt policy more consequential than a generic crawler blocklist. A single shared preference may eventually reduce configuration sprawl across participating AI providers, but it will not eliminate the need for provider-specific review. Google’s stated interaction with Google-Extended creates a concrete policy decision: whether to opt out broadly, whether to grant Google a distinct training permission, and whether eligibility for Gemini grounding is worth that choice.
For teams operating data acquisition or model-training pipelines, the direction is equally practical. Compliance logic will need to distinguish a shared AI-training preference from other crawling permissions and preserve explicit provider-level overrides. The announcement does not ship an implementation, but it identifies the policy surface likely to matter when providers begin support.
What happens next
- Google publishes implementation progress or support details for the shared control. (6 months, 65%). Watch for: updated Google crawler documentation or an IETF draft revision.
- Participating providers clarify how their existing training controls interact with the shared token. (12 months, 60%). Watch for: provider documentation that specifies precedence and exceptions.
Limits and uncertainty
This announcement is a plan, not evidence that the control is currently honored by Google. Google gives a 2027 target but no launch date, parser specification, rollout scope, or enforcement details. The post says other AI providers can handle the mechanism similarly, but it does not name providers, confirm their commitments, or establish uniform behavior. It also does not resolve the wider AIPREF vocabulary, because Google says multistakeholder discussion on that work is still continuing. Finally, the announcement describes preferences expressed in robots.txt; it does not establish a legal outcome, retroactive treatment of training data, or a universal technical guarantee.
Skill to build now
Inventory your existing robots.txt rules and document the intent behind each one: search visibility, general crawling, AI training, and Google-specific Gemini grounding are not interchangeable goals. Create a policy owner and a test checklist for any future shared token: verify precedence against existing Google-Extended directives, record the desired grounding outcome, and ensure crawler or ingestion services preserve the distinction rather than collapsing all AI traffic into one rule.
Reader takeaway
The important change is not a new crawler setting available today. It is the emergence of a common language for AI-training preferences in robots.txt, with Google planning to map that broad choice onto its existing Google-Extended control. Teams should prepare for shared policy signals while retaining explicit provider-level decisions.
Open question
Will participating providers converge on the same precedence rules when a shared training preference conflicts with their existing provider-specific directives?