{"schema_version":"newruntime-reader-dossiers-v1","type":"reader_dossier","id":"reader:google-extended","slug":"google-extended","name":"Google-Extended","kind":"ai_training","kind_label":"AI model crawlers","operator":{"name":"Google","url":"https://www.google.com/"},"docs_url":"https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers","robots_tokens":["Google-Extended"],"purpose":"Not a crawler: a standalone robots.txt product token that lets publishers control whether content crawled by Google may be used to train future Gemini models and for grounding in Gemini Apps and Vertex AI. It has no separate HTTP user-agent string; crawling uses existing Google user agents.","feeds":["model_training"],"respects_robots":"documented_yes","robots_note":"Addressed via its own user-agent group in robots.txt; does not affect Google Search inclusion or ranking. Because Google never sends 'Google-Extended' as a request UA, any request carrying it (our 'Google-Extended token probe') is not Google.","verification":{"method":"none_published","url":null},"why_it_matters":{"text":"The robots.txt switch governing Gemini training/grounding use of New Runtime; request-level hits claiming it are spoofed probes, not Google.","kind":"new_runtime_interpretation"},"confidence":"high","researched_at":"2026-10-23","sources":["https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers"],"icon_url":"https://newruntime.com/readers/icons/google.png","matching":{"aliases":["google-extended"],"rule":"Reduce each agent_days label to a token: for labels starting with Mozilla/ take the product after 'compatible;', else a versioned crawler token such as 'FooBot/1.0' (a truncated 'Mozilla/5.0 (Linux; Android 7.0;) AppleWebKit/537.36 (KHTML, like Gecko) Mobile' prefix logged before 23 Oct 2026 is PetalBot; anything else is unidentified-android, unidentified-linux-chrome or unidentified-browser); otherwise take the text before the first '/', '(' or ';' and drop a trailing ' token probe'. Lowercase it. A dossier owns the tokens in its aliases."},"observed_baseline":{"as_of":"2026-10-23T08:01:01.990Z","window_first_day":"2026-07-23","window_last_day":"2026-10-23","observed_days":93,"note":"Refreshed daily in production from the live aggregates (dated by as_of). For current numbers aggregate live_sources.history agent_days by matching.aliases.","total_requests":106,"first_seen":"2026-07-24T19:30:07.626Z","last_seen":"2026-09-25T20:02:15.082Z","days_active":7,"recency":"dormant","categories":["other_bot"],"label_variants":1,"journeys":null},"identity_note":"A family name is what a client claims in its user agent. It is verified only when its request came from the operator's published IP ranges or passed forward-confirmed reverse DNS, as reported per family in /crawler-journeys.json. Anyone can claim any name.","privacy":{"scope":"family-level aggregates only","ip_addresses":"not_output","raw_user_agents":"not_output (labels are sanitized; URLs inside them are redacted)","visitor_ids":"not_output","query_strings":"not_output"},"live_sources":{"history":"https://newruntime.com/bot-traffic-history.json","latest":"https://newruntime.com/bot-traffic-latest.json","machine_consumption":"https://newruntime.com/machine-consumption.json","crawler_journeys":"https://newruntime.com/crawler-journeys.json"},"routes":{"html":"https://newruntime.com/readers/google-extended/","markdown":"https://newruntime.com/readers/google-extended.md","json":"https://newruntime.com/readers/google-extended.json","index":"https://newruntime.com/readers.json","index_html":"https://newruntime.com/readers/"}}