{"schema_version":"newruntime-research-report-v1","type":"research_report","stable_id":"research:state-of-machine-readership-2026-10","slug":"state-of-machine-readership-2026-10","title":"State of Machine Readership: October 2026","description":"Original New Runtime research from its own privacy-safe telemetry: machine reading grew 4.1× in a month, AI model crawlers read in single-day sweeps, new Field Notes reach AI crawlers within a day, half of 'live assistant' traffic probes for secrets, and llms.txt is almost never read.","published_at":"2026-10-23","language":"en","author":"New Runtime","routes":{"html":"https://newruntime.com/research/state-of-machine-readership-2026-10/","markdown":"https://newruntime.com/research/state-of-machine-readership-2026-10.md","json":"https://newruntime.com/research/state-of-machine-readership-2026-10.json"},"windows":["Mix, depth, identity, live fetches and speed: 62 complete 24-hour windows, 21 Aug – 21 Oct 2026 (UTC, each ending 23:30). Halves compared: H1 21 Aug – 20 Sep, H2 21 Sep – 21 Oct.","MCP endpoint: 28 days of JSON-RPC method counts, 26 Sep – 23 Oct 2026.","First-seen dates: the public 93-day history from 23 Jul 2026. Late burst: the 23 Oct 14:02 UTC public snapshot."],"findings":[{"id":"mix","claim":"Machine reading quadrupled in a month, and the new volume came from AI answer engines and agent registries, not search.","numbers":["38,281 → 157,953 requests between the two 31-day halves (×4.1).","AI search & answer crawlers: 8.2% → 16.3% of named automation (×8.2 in volume).","Agentic web & MCP registries: 0.9% → 12.1% (×56.5).","Search engines: 30.7% → 17.8%; SEO crawlers 38% → 26.9%."],"method":"Every named client in each 24-hour window is mapped to a visitor-dossier family and kind; scanners, probes and our own monitoring are excluded.","caveat":"Requests, not unique readers. The site kept publishing through the window, so supply grew too; two new unnamed crawlers sit in 'other'.","interpretation":"The site is now read mostly by systems that answer questions or index agents, while classic search is a shrinking share of a growing total.","would_change_our_mind":"The next 31 days revert to a search-led mix while our publishing rate stays flat."},{"id":"sweeps","claim":"AI model crawlers read in single-day sweeps; answer engines and search engines read every day.","numbers":["GPTBot: 53.3% of its 62-day requests on 25 Sep (3,274 of 6,137).","ClaudeBot: 50.2% on 19 Oct (5,472 of 10,905).","Meta-ExternalAgent: 47.1% on 4 Oct, then 6,948 requests in ten hours on 22–23 Oct, 3.4× its previous 62 days.","OAI-SearchBot 7.9%, PerplexityBot 7.8%, Bingbot 5.3%, Googlebot 10.2% on their busiest day."],"method":"Share of each family's 62-day requests that fell in its busiest 24-hour window; families with at least 1,000 requests.","caveat":"Amazonbot (10%) and Reflectionbot (21%) are training crawlers that behave more like maintainers, so this holds for three of five, not for the class.","interpretation":"A page that misses a training sweep may wait weeks for the next one. Freshness inside AI answers depends on answer-engine crawlers, not trainers.","would_change_our_mind":"Over the next 90 days GPTBot, ClaudeBot and Meta-ExternalAgent each show no day above 25% of their volume."},{"id":"speed","claim":"New Field Notes reach AI crawlers within a day; live assistants almost never fetch them.","numbers":["93% of 601 Field Notes were first fetched by an AI crawler within 24 hours, 98.8% within 48 hours.","Analysis: 49.2% / 93.8% (65 items). Signals: 35.9% / 60.8% (590).","Search engines reached 67.4% of Field Notes within 24 hours.","A live assistant fetch reached 13% of Field Notes at any point, 2.8% within 24 hours."],"method":"First fetch day per canonical page against its site publication time, for items published 22 Aug – 14 Oct (at least a week of follow-up).","caveat":"Windows close at 23:30 UTC, so the hours are upper bounds. Category level, not family. A fetch is not indexing.","interpretation":"Field Notes are the format AI crawlers pick up fastest; signals sit deeper in the archive and wait longer.","would_change_our_mind":"The share of Field Notes fetched by an AI crawler within 48 hours falls below 80% in the next window."},{"id":"live","claim":"More than half of 'live assistant' traffic is not assistants answering people: it is probing for secrets.","numbers":["1,829 requests named a live assistant fetcher (ChatGPT-User, Claude-User, Perplexity-User) in 62 days.","52.5% went to paths that do not exist here: cloud credentials, agent configs (.claude, .codex), SSRF-style /fetch/ and /proxy/.","34.1% fetched the homepage; 8.6% fetched an article (139 distinct).","Verified ChatGPT-User: 677 requests, 43 article reads, 99.1% HTML, no Markdown. Live fetches fell from 1,231 to 696 between halves."],"method":"User-agent category from the end-of-day aggregates; destinations classified by route. Probe paths are excluded before identity checks.","caveat":"We cannot say how many probe-path requests came from the assistants' own networks.","interpretation":"For a small site, the live-answer channel is mostly people asking an assistant about the site itself, plus scanners borrowing a trusted name.","would_change_our_mind":"Article fetches overtake homepage fetches among verified live-assistant requests."},{"id":"identity","claim":"A crawler's name is a claim: 65% of requests calling themselves Googlebot were not Google.","numbers":["Googlebot: 3,199 failed claims vs 1,756 verified; failed claims got a success response 2% of the time.","Across nine families with a published verifier, 8.7% of claims failed. Success rate: 98.6% for verified, 6.1% for failed claims.","242 requests named Google-Extended, a robots.txt token that has no crawler of its own.","47.5% of named requests come from families with no published verification method, among them ClaudeBot, Meta-ExternalAgent, AhrefsBot and SemrushBot."],"method":"Official CIDR lists or forward-confirmed reverse DNS, checked privately per request; only status counts are published.","caveat":"Failed verification can also mean a verifier list went stale; ours are refreshed daily and listed in /crawler-journeys.json.","interpretation":"Any per-bot ranking that does not verify identity overstates Googlebot and blends scanners into 'AI traffic'.","would_change_our_mind":"Google publishes ranges under which those requests verify."},{"id":"maps","claim":"Machines read page alternates, not site maps.","numbers":["AI crawlers fetched llms.txt or llms-full.txt 15 times in 62 days, against 39,404 AI-crawler requests.","Named automation fetched Markdown twins 36,089 times and JSON twins 35,982 times; for GPTBot about half of all requests were twins.","Typed next-reads: 49 of 4,062 exposed verified sessions (1.2%) took one; 1.8% of transitions matched.","Manifest: 2,653 verified sessions read /machine-consumption.json; 12.9% fetched content within 30 minutes. Applebot: 1,291 sessions, 1 follow-through."],"method":"Format counts from the end-of-day aggregates; journeys from verified 30-minute sessions with alternate representations collapsed to one material.","caveat":"Every page's footer script requests /machine-consumption.json, so many manifest reads are probably a side effect of JavaScript rendering, not navigation.","interpretation":"Crawlers find .md and .json twins through each page's alternate links and ignore the maps we write for them.","would_change_our_mind":"AI-crawler fetches of llms.txt exceed one per 1,000 AI-crawler requests, or the documented next-read rate passes 10%."},{"id":"agentic","claim":"The agentic web arrived in one wave, and it shakes hands rather than uses tools.","numbers":["19 of 21 registry and MCP-collector families were first seen on or after 25 Sep, the day the MCP server was listed in the MCP registry and the agent card went live.","Their requests: 338 → 19,110 between halves.","MCP endpoint: 74,413 JSON-RPC messages from 303 client names; 97.9% initialize, list or discover; 1.5% tool calls.","27 clients ever called a tool; one made 54.8% of all calls, and one liveness monitor sent 38.8% of all messages."],"method":"Registry first-seen dates from the history and end-of-day aggregates; MCP method counts per client name from the endpoint's own log, our smoke tests removed.","caveat":"MCP counting began on 26 Sep, so pre-listing traffic is unmeasured; client names are self-declared.","interpretation":"Directories catalogue agent-ready sites quickly, but almost nothing yet uses the tools they catalogue.","would_change_our_mind":"Tool calls exceed 10% of MCP messages in a 28-day window."}],"methods":["Sources: the public aggregates listed below, plus private end-of-day aggregates computed on the server from access logs. Only counts leave the server; no IP address, raw user agent, query string, session or per-visitor record is published here.","Units are successful and failed HTTP requests unless a line says sessions (30-minute inactivity windows per claimed family) or materials (one canonical page with its HTML, Markdown and JSON forms collapsed).","Excluded everywhere: New Runtime monitoring and utility clients, marked synthetic checks, scanners and sensitive-path probes, and the readership widget's own requests.","Identity: 'verified' means the request matched the operator's published IP ranges or forward-confirmed reverse DNS. 'Not verifiable' means the operator publishes no supported method. A name alone is never treated as identity.","Correlation, not causation: request order is a server-side sequence, not a crawler's decision path, and a fetch proves neither indexing, training nor citation.","Instrument note: the public daily history (/bot-traffic-history.json) keeps the last, partial view of each day; for 21 Aug – 21 Oct it holds 22% of the AI-crawler requests the end-of-day aggregates counted. We use it only for first-seen dates. The 'median discovery latency' in /crawler-journeys.json is censored by about two days of log retention, so the speed section uses publication timestamps instead.","Interpretation lines are New Runtime's own reading of the numbers; each finding names the result that would change it."],"sources":[{"label":"Report as Markdown","url":"https://newruntime.com/research/state-of-machine-readership-2026-10.md"},{"label":"Bot traffic history (93 days, public)","url":"https://newruntime.com/bot-traffic-history.json"},{"label":"Machine consumption (live, 30-minute refresh)","url":"https://newruntime.com/machine-consumption.json"},{"label":"Crawler journeys (24h / 7d / 30d)","url":"https://newruntime.com/crawler-journeys.json"},{"label":"Visitor dossiers","url":"https://newruntime.com/readers.json"},{"label":"Discovery graph","url":"https://newruntime.com/discovery-graph.json"}],"privacy":{"ip_addresses":"not_output","raw_user_agents":"not_output","query_strings":"not_output","sessions":"not_output","aggregates":"counts and percentages only"},"interpretation_note":"Each finding separates measured numbers from New Runtime's interpretation and names what would change it.","aggregates":{"window":{"first_seen_history":{"first_day":"2026-07-23","last_day":"2026-10-23","last_snapshot":"2026-10-23T14:01:06.737Z","observed_days":93,"use":"first-seen dates only"},"periods":[{"id":"H1","first_day":"2026-08-21","last_day":"2026-09-20","days":31},{"id":"H2","first_day":"2026-09-21","last_day":"2026-10-21","days":31}],"identity_and_depth":{"first_day":"2026-08-21","last_day":"2026-10-21","daily_windows":62,"source":"end-of-day 24-hour crawler-journey aggregates (private, aggregate only)"},"live_and_latency":{"first_day":"2026-08-21","last_day":"2026-10-21","daily_windows":62,"source":"end-of-day 24-hour machine-consumption aggregates (private, aggregate only)"},"mcp":{"first_day":"2026-09-26","last_day":"2026-10-23","days":28}},"mix":{"unit":"requests by named external automation in complete 24-hour windows; scanners, probes and token probes excluded","totals":{"H1":38281,"H2":157953},"growth_x":4.1,"kinds":[{"kind":"ai_training","label":"AI model crawlers","requests":{"H1":6109,"H2":28818},"share_percent":{"H1":16,"H2":18.2},"growth_x":4.7},{"kind":"ai_answer","label":"AI search & answer crawlers","requests":{"H1":3129,"H2":25709},"share_percent":{"H1":8.2,"H2":16.3},"growth_x":8.2},{"kind":"ai_live","label":"Live assistant fetches","requests":{"H1":1231,"H2":696},"share_percent":{"H1":3.2,"H2":0.4},"growth_x":0.6},{"kind":"search","label":"Search engines","requests":{"H1":11753,"H2":28185},"share_percent":{"H1":30.7,"H2":17.8},"growth_x":2.4},{"kind":"seo","label":"SEO & backlink crawlers","requests":{"H1":14562,"H2":42437},"share_percent":{"H1":38,"H2":26.9},"growth_x":2.9},{"kind":"agentic_registry","label":"Agentic web & MCP registries","requests":{"H1":338,"H2":19110},"share_percent":{"H1":0.9,"H2":12.1},"growth_x":56.5},{"kind":"other","label":"Link previews & unidentified","requests":{"H1":1159,"H2":12998},"share_percent":{"H1":3,"H2":8.2},"growth_x":11.2}],"weekly":[{"week_start":"2026-08-21","week_end":"2026-08-27","days":7,"per_day_by_kind":{"ai_training":137,"ai_answer":106,"ai_live":43,"search":228,"seo":317,"agentic_registry":0,"other":15}},{"week_start":"2026-08-28","week_end":"2026-09-03","days":7,"per_day_by_kind":{"ai_training":470,"ai_answer":56,"ai_live":33,"search":335,"seo":431,"agentic_registry":0,"other":33}},{"week_start":"2026-09-04","week_end":"2026-09-10","days":7,"per_day_by_kind":{"ai_training":56,"ai_answer":157,"ai_live":64,"search":468,"seo":754,"agentic_registry":24,"other":78}},{"week_start":"2026-09-11","week_end":"2026-09-17","days":7,"per_day_by_kind":{"ai_training":201,"ai_answer":98,"ai_live":33,"search":475,"seo":392,"agentic_registry":25,"other":39}},{"week_start":"2026-09-18","week_end":"2026-09-24","days":7,"per_day_by_kind":{"ai_training":19,"ai_answer":183,"ai_live":9,"search":356,"seo":516,"agentic_registry":0,"other":2}},{"week_start":"2026-09-25","week_end":"2026-10-01","days":7,"per_day_by_kind":{"ai_training":700,"ai_answer":538,"ai_live":21,"search":459,"seo":2036,"agentic_registry":679,"other":17}},{"week_start":"2026-10-02","week_end":"2026-10-08","days":7,"per_day_by_kind":{"ai_training":653,"ai_answer":645,"ai_live":16,"search":969,"seo":775,"agentic_registry":694,"other":47}},{"week_start":"2026-10-09","week_end":"2026-10-15","days":7,"per_day_by_kind":{"ai_training":765,"ai_answer":969,"ai_live":27,"search":1262,"seo":1100,"agentic_registry":706,"other":837}},{"week_start":"2026-10-16","week_end":"2026-10-21","days":6,"per_day_by_kind":{"ai_training":2321,"ai_answer":1596,"ai_live":36,"search":1345,"seo":2125,"agentic_registry":759,"other":1114}}]},"ai_leaders_by_period":{"H1":[{"family":"ClaudeBot","kind":"ai_training","requests":4817},{"family":"OAI-SearchBot","kind":"ai_answer","requests":2144},{"family":"GPTBot","kind":"ai_training","requests":1120},{"family":"PerplexityBot / Perplexity-User","kind":"ai_answer","requests":826},{"family":"ChatGPT-User","kind":"ai_live","requests":793},{"family":"Claude-User","kind":"ai_live","requests":438},{"family":"Claude-SearchBot","kind":"ai_answer","requests":157},{"family":"Bytespider","kind":"ai_training","requests":145}],"H2":[{"family":"PerplexityBot / Perplexity-User","kind":"ai_answer","requests":11631},{"family":"Reflectionbot","kind":"ai_training","requests":9007},{"family":"Amazonbot","kind":"ai_training","requests":6566},{"family":"ExaSearchBot / ExaUser","kind":"ai_answer","requests":6180},{"family":"ClaudeBot","kind":"ai_training","requests":6088},{"family":"OAI-SearchBot","kind":"ai_answer","requests":5483},{"family":"GPTBot","kind":"ai_training","requests":5017},{"family":"Meta-ExternalAgent","kind":"ai_training","requests":2068}]},"history_completeness":{"public_history_ai_crawler_requests":8677,"end_of_day_ai_crawler_requests":39404,"ratio_percent":22},"burstiness":{"unit":"share of a family's 62-day requests that fell in its single busiest 24-hour window; dossier families with at least 1,000 requests","rows":[{"family":"DataForSeoBot","slug":"dataforseobot","kind":"seo","requests":4082,"active_days":5,"busiest_day":"2026-09-27","busiest_day_requests":4015,"busiest_day_share_percent":98.4},{"family":"ShapBot","slug":"shapbot","kind":"ai_answer","requests":1247,"active_days":5,"busiest_day":"2026-10-01","busiest_day_requests":1223,"busiest_day_share_percent":98.1},{"family":"SERankingBacklinksBot","slug":"serankingbacklinksbot","kind":"seo","requests":3121,"active_days":22,"busiest_day":"2026-10-01","busiest_day_requests":1941,"busiest_day_share_percent":62.2},{"family":"GPTBot","slug":"gptbot","kind":"ai_training","requests":6137,"active_days":62,"busiest_day":"2026-09-25","busiest_day_requests":3274,"busiest_day_share_percent":53.3},{"family":"ClaudeBot","slug":"claudebot","kind":"ai_training","requests":10905,"active_days":55,"busiest_day":"2026-10-19","busiest_day_requests":5472,"busiest_day_share_percent":50.2},{"family":"Meta-ExternalAgent","slug":"meta-externalagent","kind":"ai_training","requests":2069,"active_days":16,"busiest_day":"2026-10-04","busiest_day_requests":974,"busiest_day_share_percent":47.1},{"family":"Reflectionbot","slug":"reflectionbot","kind":"ai_training","requests":9007,"active_days":18,"busiest_day":"2026-10-18","busiest_day_requests":1859,"busiest_day_share_percent":20.6},{"family":"MJ12bot","slug":"mj12bot","kind":"seo","requests":6824,"active_days":44,"busiest_day":"2026-10-18","busiest_day_requests":1252,"busiest_day_share_percent":18.3},{"family":"ExaSearchBot / ExaUser","slug":"exa","kind":"ai_answer","requests":6180,"active_days":22,"busiest_day":"2026-10-15","busiest_day_requests":918,"busiest_day_share_percent":14.9},{"family":"Amazonbot","slug":"amazonbot","kind":"ai_training","requests":6566,"active_days":20,"busiest_day":"2026-10-15","busiest_day_requests":676,"busiest_day_share_percent":10.3},{"family":"Googlebot","slug":"googlebot","kind":"search","requests":3150,"active_days":62,"busiest_day":"2026-09-07","busiest_day_requests":321,"busiest_day_share_percent":10.2},{"family":"YandexBot","slug":"yandexbot","kind":"search","requests":6247,"active_days":57,"busiest_day":"2026-09-25","busiest_day_requests":629,"busiest_day_share_percent":10.1},{"family":"AhrefsBot","slug":"ahrefsbot","kind":"seo","requests":21115,"active_days":62,"busiest_day":"2026-10-21","busiest_day_requests":1997,"busiest_day_share_percent":9.5},{"family":"DotBot","slug":"dotbot","kind":"seo","requests":1348,"active_days":25,"busiest_day":"2026-09-29","busiest_day_requests":128,"busiest_day_share_percent":9.5},{"family":"OAI-SearchBot","slug":"oai-searchbot","kind":"ai_answer","requests":7627,"active_days":62,"busiest_day":"2026-10-21","busiest_day_requests":599,"busiest_day_share_percent":7.9},{"family":"PerplexityBot / Perplexity-User","slug":"perplexitybot","kind":"ai_answer","requests":12457,"active_days":42,"busiest_day":"2026-10-21","busiest_day_requests":971,"busiest_day_share_percent":7.8},{"family":"BrickBlueBot","slug":"brickbluebot","kind":"agentic_registry","requests":1872,"active_days":27,"busiest_day":"2026-09-27","busiest_day_requests":133,"busiest_day_share_percent":7.1},{"family":"facebookexternalhit","slug":"facebookexternalhit","kind":"link_preview","requests":4521,"active_days":60,"busiest_day":"2026-10-12","busiest_day_requests":315,"busiest_day_share_percent":7},{"family":"ChatGPT-User","slug":"chatgpt-user","kind":"ai_live","requests":1355,"active_days":62,"busiest_day":"2026-08-22","busiest_day_requests":92,"busiest_day_share_percent":6.8},{"family":"PetalBot","slug":"petalbot","kind":"search","requests":18188,"active_days":21,"busiest_day":"2026-10-06","busiest_day_requests":1020,"busiest_day_share_percent":5.6},{"family":"Bingbot","slug":"bingbot","kind":"search","requests":7393,"active_days":62,"busiest_day":"2026-09-11","busiest_day_requests":389,"busiest_day_share_percent":5.3},{"family":"rokmcp-collector","slug":"rokmcp-collector","kind":"agentic_registry","requests":2289,"active_days":26,"busiest_day":"2026-10-04","busiest_day_requests":105,"busiest_day_share_percent":4.6},{"family":"SemrushBot","slug":"semrushbot","kind":"seo","requests":20160,"active_days":62,"busiest_day":"2026-10-03","busiest_day_requests":893,"busiest_day_share_percent":4.4},{"family":"Applebot","slug":"applebot","kind":"search","requests":4759,"active_days":62,"busiest_day":"2026-10-20","busiest_day_requests":207,"busiest_day_share_percent":4.3},{"family":"mcpbeat","slug":"mcpbeat","kind":"agentic_registry","requests":11786,"active_days":26,"busiest_day":"2026-09-29","busiest_day_requests":462,"busiest_day_share_percent":3.9}]},"registries":{"families":21,"first_seen_from_2026_09_25":19,"requests":{"H1":338,"H2":19110},"mcp_registry_listing_day":"2026-09-25","rows":[{"family":"OraBot","first_seen_day":"2026-09-06","requests_62d":332},{"family":"AgentTrustBot","first_seen_day":"2026-09-14","requests_62d":6},{"family":"agent-tools.cloud crawler","first_seen_day":"2026-09-25","requests_62d":586},{"family":"BrickBlueBot","first_seen_day":"2026-09-25","requests_62d":1872},{"family":"GolemreachTrustBot","first_seen_day":"2026-09-25","requests_62d":498},{"family":"WellknownBot","first_seen_day":"2026-09-25","requests_62d":388},{"family":"TalandorBot","first_seen_day":"2026-09-25","requests_62d":457},{"family":"mcpbeat","first_seen_day":"2026-09-26","requests_62d":11786},{"family":"rokmcp-collector","first_seen_day":"2026-09-26","requests_62d":2289},{"family":"histor","first_seen_day":"2026-09-26","requests_62d":135},{"family":"KnownGood-Verifier","first_seen_day":"2026-09-26","requests_62d":20},{"family":"MCPHereBot","first_seen_day":"2026-09-26","requests_62d":21},{"family":"AgentIndexBot","first_seen_day":"2026-09-26","requests_62d":54},{"family":"GAIP-Observatory","first_seen_day":"2026-10-01","requests_62d":23},{"family":"mcpcensusbot","first_seen_day":"2026-10-01","requests_62d":12},{"family":"7Maps-bot","first_seen_day":"2026-10-02","requests_62d":182},{"family":"AgenstryBot","first_seen_day":"2026-10-03","requests_62d":252},{"family":"StealthStackCrawler","first_seen_day":"2026-10-04","requests_62d":52},{"family":"GlintbaseResearch","first_seen_day":"2026-10-12","requests_62d":12},{"family":"MCPStationBot","first_seen_day":"2026-10-16","requests_62d":30},{"family":"x402-router-crawler","first_seen_day":"2026-10-19","requests_62d":12}]},"depth":{"unit":"verified or not-verifiable families with at least 500 requests across 62 daily 24-hour windows","rows":[{"family":"PerplexityBot","identity":"verified","active_days":26,"requests":11900,"material_reads":9424,"median_unique_materials_per_active_day":227,"max_unique_materials_in_a_day":657,"format_share_percent":{"html":72.8,"markdown":12.4,"json":12.7,"other":2.1},"manifest_sessions":0,"manifest_followthrough_sessions":0,"manifest_followthrough_percent":null},{"family":"SemrushBot","identity":"not_verifiable","active_days":54,"requests":11751,"material_reads":9286,"median_unique_materials_per_active_day":79,"max_unique_materials_in_a_day":514,"format_share_percent":{"html":58.9,"markdown":15.1,"json":15.1,"other":11},"manifest_sessions":2,"manifest_followthrough_sessions":1,"manifest_followthrough_percent":50},{"family":"ClaudeBot","identity":"not_verifiable","active_days":39,"requests":10770,"material_reads":7471,"median_unique_materials_per_active_day":0,"max_unique_materials_in_a_day":1957,"format_share_percent":{"html":59.5,"markdown":14.1,"json":13.5,"other":12.9},"manifest_sessions":1,"manifest_followthrough_sessions":1,"manifest_followthrough_percent":100},{"family":"AhrefsBot","identity":"not_verifiable","active_days":53,"requests":9999,"material_reads":8898,"median_unique_materials_per_active_day":45,"max_unique_materials_in_a_day":628,"format_share_percent":{"html":33,"markdown":29.9,"json":31.2,"other":5.9},"manifest_sessions":8,"manifest_followthrough_sessions":0,"manifest_followthrough_percent":0},{"family":"OpenAI SearchBot","identity":"verified","active_days":62,"requests":6830,"material_reads":2410,"median_unique_materials_per_active_day":13.5,"max_unique_materials_in_a_day":173,"format_share_percent":{"html":34.7,"markdown":1.7,"json":55.1,"other":8.5},"manifest_sessions":1230,"manifest_followthrough_sessions":323,"manifest_followthrough_percent":26.3},{"family":"MJ12bot","identity":"not_verifiable","active_days":35,"requests":6778,"material_reads":5462,"median_unique_materials_per_active_day":18,"max_unique_materials_in_a_day":478,"format_share_percent":{"html":78.4,"markdown":3,"json":3.4,"other":15.2},"manifest_sessions":0,"manifest_followthrough_sessions":0,"manifest_followthrough_percent":null},{"family":"GPTBot","identity":"verified","active_days":62,"requests":5820,"material_reads":4318,"median_unique_materials_per_active_day":0,"max_unique_materials_in_a_day":1044,"format_share_percent":{"html":48.4,"markdown":24.3,"json":25.4,"other":1.9},"manifest_sessions":6,"manifest_followthrough_sessions":1,"manifest_followthrough_percent":16.7},{"family":"Bingbot","identity":"verified","active_days":62,"requests":5695,"material_reads":4283,"median_unique_materials_per_active_day":54,"max_unique_materials_in_a_day":138,"format_share_percent":{"html":62.7,"markdown":15.8,"json":16.9,"other":4.6},"manifest_sessions":63,"manifest_followthrough_sessions":6,"manifest_followthrough_percent":9.5},{"family":"YandexBot","identity":"verified","active_days":57,"requests":4133,"material_reads":2650,"median_unique_materials_per_active_day":23,"max_unique_materials_in_a_day":339,"format_share_percent":{"html":62.1,"markdown":10.4,"json":6.7,"other":20.7},"manifest_sessions":1,"manifest_followthrough_sessions":0,"manifest_followthrough_percent":0},{"family":"Applebot","identity":"verified","active_days":62,"requests":3001,"material_reads":198,"median_unique_materials_per_active_day":2,"max_unique_materials_in_a_day":20,"format_share_percent":{"html":14.8,"markdown":0.8,"json":74.5,"other":10},"manifest_sessions":1291,"manifest_followthrough_sessions":1,"manifest_followthrough_percent":0.1},{"family":"Googlebot","identity":"verified","active_days":62,"requests":1756,"material_reads":406,"median_unique_materials_per_active_day":3,"max_unique_materials_in_a_day":40,"format_share_percent":{"html":28.3,"markdown":0.1,"json":5.2,"other":66.5},"manifest_sessions":62,"manifest_followthrough_sessions":11,"manifest_followthrough_percent":17.7},{"family":"ChatGPT-User","identity":"verified","active_days":62,"requests":677,"material_reads":43,"median_unique_materials_per_active_day":0,"max_unique_materials_in_a_day":4,"format_share_percent":{"html":99.1,"markdown":0,"json":0.3,"other":0.6},"manifest_sessions":0,"manifest_followthrough_sessions":0,"manifest_followthrough_percent":null}],"verified_graph_coverage_24h_percent":{"median":13.45,"min":4.5,"max":90.2},"manifest":{"sessions":2653,"followthrough_sessions":342,"followthrough_percent":12.9},"typed_next_reads":{"first_day":"2026-09-12","exposed_sessions":4062,"sessions_with_documented_transition":49,"session_percent":1.2,"documented_transitions":99,"other_site_transitions":5429,"transition_match_percent":1.8}},"formats":{"unit":"requests by named external automation, 62 daily windows","totals":{"html":117423,"json":35982,"markdown":36089,"robots_sitemap":6187,"feed":763,"llms":188},"llms_txt_and_llms_full_by_ai_crawlers":15,"ai_crawler_requests":39404},"live_fetches":{"unit":"requests whose user agent names a live assistant fetcher (ChatGPT-User, Claude-User, Perplexity-User and similar), 62 daily windows","total":1829,"destinations":{"home":623,"articles":157,"other_site_routes":88,"nonexistent_probe_paths":961},"destinations_percent":{"home":34.1,"articles":8.6,"other_site_routes":4.8,"nonexistent_probe_paths":52.5},"distinct_articles":139,"verified_chatgpt_user":{"requests":677,"material_reads":43,"html_percent":99.1}},"discovery_latency":{"unit":"share of items published 2026-08-22..2026-10-14 first fetched by each category within N hours; upper bound from daily 23:30 UTC windows","rows":[{"collection":"posts","label":"Field Notes","items":601,"ai_crawler":{"within_24h_percent":93,"within_48h_percent":98.8,"within_7d_percent":100,"ever_percent":100},"search":{"within_24h_percent":67.4,"within_48h_percent":68.2,"within_7d_percent":98,"ever_percent":100},"seo_link_crawler":{"within_24h_percent":74,"within_48h_percent":89,"within_7d_percent":100,"ever_percent":100},"ai_user_agent":{"within_24h_percent":2.8,"within_48h_percent":6.3,"within_7d_percent":9.8,"ever_percent":13}},{"collection":"analysis","label":"Analysis","items":65,"ai_crawler":{"within_24h_percent":49.2,"within_48h_percent":93.8,"within_7d_percent":100,"ever_percent":100},"search":{"within_24h_percent":76.9,"within_48h_percent":76.9,"within_7d_percent":98.5,"ever_percent":100},"seo_link_crawler":{"within_24h_percent":89.2,"within_48h_percent":98.5,"within_7d_percent":100,"ever_percent":100},"ai_user_agent":{"within_24h_percent":0,"within_48h_percent":0,"within_7d_percent":0,"ever_percent":3.1}},{"collection":"signals","label":"Signals","items":590,"ai_crawler":{"within_24h_percent":35.9,"within_48h_percent":60.8,"within_7d_percent":81.7,"ever_percent":100},"search":{"within_24h_percent":55.3,"within_48h_percent":55.8,"within_7d_percent":57.6,"ever_percent":60.5},"seo_link_crawler":{"within_24h_percent":58.1,"within_48h_percent":74.9,"within_7d_percent":100,"ever_percent":100},"ai_user_agent":{"within_24h_percent":0.3,"within_48h_percent":0.7,"within_7d_percent":0.8,"ever_percent":1}}]},"identity":{"unit":"requests by families that publish a verifier (CIDR list or forward-confirmed reverse DNS), 62 daily windows","by_status":{"verified":{"requests":39812,"successful":39272},"not_verifiable":{"requests":39371,"successful":38319},"claimed_unverified":{"requests":3778,"successful":229}},"verifiable_family_requests":43590,"failed_claim_requests":3778,"failed_claim_share_percent":8.7,"rows":[{"family":"Googlebot","verified_requests":1756,"verified_success_percent":100,"failed_claim_requests":3199,"failed_claim_success_percent":2,"failed_claim_share_percent":64.6},{"family":"OpenAI SearchBot","verified_requests":6830,"verified_success_percent":99.3,"failed_claim_requests":174,"failed_claim_success_percent":15.5,"failed_claim_share_percent":2.5},{"family":"GPTBot","verified_requests":5820,"verified_success_percent":97.9,"failed_claim_requests":153,"failed_claim_success_percent":29.4,"failed_claim_share_percent":2.6},{"family":"ChatGPT-User","verified_requests":677,"verified_success_percent":100,"failed_claim_requests":117,"failed_claim_success_percent":24.8,"failed_claim_share_percent":14.7},{"family":"Applebot","verified_requests":3001,"verified_success_percent":99.3,"failed_claim_requests":59,"failed_claim_success_percent":42.4,"failed_claim_share_percent":1.9},{"family":"YandexBot","verified_requests":4133,"verified_success_percent":100,"failed_claim_requests":27,"failed_claim_success_percent":66.7,"failed_claim_share_percent":0.6},{"family":"Perplexity-User","verified_requests":0,"verified_success_percent":null,"failed_claim_requests":27,"failed_claim_success_percent":11.1,"failed_claim_share_percent":100},{"family":"PerplexityBot","verified_requests":11900,"verified_success_percent":97.8,"failed_claim_requests":13,"failed_claim_success_percent":100,"failed_claim_share_percent":0.1},{"family":"Bingbot","verified_requests":5695,"verified_success_percent":98.5,"failed_claim_requests":9,"failed_claim_success_percent":66.7,"failed_claim_share_percent":0.2}],"not_verifiable_share_percent":47.5},"mcp":{"window":{"first_day":"2026-09-26","last_day":"2026-10-23","days":28},"json_rpc_messages":74413,"client_names":303,"handshake_and_listing_messages":72886,"handshake_and_listing_percent":97.9,"tools_call_messages":1100,"tools_call_percent":1.5,"resources_read_messages":241,"clients_that_called_a_tool":27,"top_caller_share_percent":54.8,"top_caller":"x402-observer","largest_client_share_percent":38.8,"largest_client":"SentinelOracle","method_totals":{"tools/list":21376,"initialize":20006,"notifications/initialized":17069,"server/discover":5821,"prompts/list":4275,"resources/list":4224,"tools/call":1100,"resources/read":241,"none":132,"resources/templates/list":115}},"meta_late_burst":{"family":"Meta-ExternalAgent","requests":6948,"first_seen":"2026-10-22T18:13:12Z","last_seen":"2026-10-23T04:01:41Z","source":"/machine-consumption.json snapshot of 2026-10-23T14:02Z","ratio_to_previous_62_days":3.4}}}
