AI model crawlers

Google-Extended

Not a crawler: a standalone robots.txt product token that lets publishers control whether content crawled by Google may be used to train future Gemini models and for grounding in Gemini Apps and Vertex AI. It has no separate HTTP user-agent string; crawling uses existing Google user agents.

Dossier

Operator: Google
Documentation: developers.google.com
robots.txt token: Google-Extended
Feeds: model training
Follows robots.txt: Yes, per its documentationAddressed via its own user-agent group in robots.txt; does not affect Google Search inclusion or ranking. Because Google never sends 'Google-Extended' as a request UA, any request carrying it (our 'Google-Extended token probe') is not Google.
How to verify: No published method
Names in our logs: google-extended
Researched: 23 Oct 2026

Observed on New Runtime

Baseline 2026-10-23 · 93-day window · loading live numbers…

Requests: 106
Days active: 7 / 93
First seen: 24 Jul 2026
Last seen: 25 Sept 2026

Dormantother_bot

Daily requests across the observed window, square-root scale.

Identity check claimed vs verified

No per-family identity check is published for this reader; its name is a claim.

What it reads crawler journeys · current window

The crawler-journeys aggregate tracks reading depth for the busiest named crawlers only; this family is not among them.

Sources

developers.google.comdevelopers.google.com/search/docs/crawling-indexing/google-common-crawlers