# State of Machine Readership: October 2026

> What 62 days of complete server aggregates say about who reads an agent-ready site by machine, how fast, how deep, and under whose name.

Published 2026-10-23 by New Runtime. Canonical: https://newruntime.com/research/state-of-machine-readership-2026-10/

## Windows

- Mix, depth, identity, live fetches and speed: 62 complete 24-hour windows, 21 Aug – 21 Oct 2026 (UTC, each ending 23:30). Halves compared: H1 21 Aug – 20 Sep, H2 21 Sep – 21 Oct.
- MCP endpoint: 28 days of JSON-RPC method counts, 26 Sep – 23 Oct 2026.
- First-seen dates: the public 93-day history from 23 Jul 2026. Late burst: the 23 Oct 14:02 UTC public snapshot.

## Key findings

1. Machine reading quadrupled in a month, and the new volume came from AI answer engines and agent registries, not search.
2. AI model crawlers read in single-day sweeps; answer engines and search engines read every day.
3. New Field Notes reach AI crawlers within a day; live assistants almost never fetch them.
4. More than half of 'live assistant' traffic is not assistants answering people: it is probing for secrets.
5. A crawler's name is a claim: 65% of requests calling themselves Googlebot were not Google.
6. Machines read page alternates, not site maps.
7. The agentic web arrived in one wave, and it shakes hands rather than uses tools.

## 1. Machine reading quadrupled in a month, and the new volume came from AI answer engines and agent registries, not search.

- 38,281 → 157,953 requests between the two 31-day halves (×4.1).
- AI search & answer crawlers: 8.2% → 16.3% of named automation (×8.2 in volume).
- Agentic web & MCP registries: 0.9% → 12.1% (×56.5).
- Search engines: 30.7% → 17.8%; SEO crawlers 38% → 26.9%.

Method: Every named client in each 24-hour window is mapped to a visitor-dossier family and kind; scanners, probes and our own monitoring are excluded.

Caveat: Requests, not unique readers. The site kept publishing through the window, so supply grew too; two new unnamed crawlers sit in 'other'.

Our interpretation: The site is now read mostly by systems that answer questions or index agents, while classic search is a shrinking share of a growing total.

What would change our mind: The next 31 days revert to a search-led mix while our publishing rate stays flat.

## 2. AI model crawlers read in single-day sweeps; answer engines and search engines read every day.

- GPTBot: 53.3% of its 62-day requests on 25 Sep (3,274 of 6,137).
- ClaudeBot: 50.2% on 19 Oct (5,472 of 10,905).
- Meta-ExternalAgent: 47.1% on 4 Oct, then 6,948 requests in ten hours on 22–23 Oct, 3.4× its previous 62 days.
- OAI-SearchBot 7.9%, PerplexityBot 7.8%, Bingbot 5.3%, Googlebot 10.2% on their busiest day.

Method: Share of each family's 62-day requests that fell in its busiest 24-hour window; families with at least 1,000 requests.

Caveat: Amazonbot (10%) and Reflectionbot (21%) are training crawlers that behave more like maintainers, so this holds for three of five, not for the class.

Our interpretation: A page that misses a training sweep may wait weeks for the next one. Freshness inside AI answers depends on answer-engine crawlers, not trainers.

What would change our mind: Over the next 90 days GPTBot, ClaudeBot and Meta-ExternalAgent each show no day above 25% of their volume.

## 3. New Field Notes reach AI crawlers within a day; live assistants almost never fetch them.

- 93% of 601 Field Notes were first fetched by an AI crawler within 24 hours, 98.8% within 48 hours.
- Analysis: 49.2% / 93.8% (65 items). Signals: 35.9% / 60.8% (590).
- Search engines reached 67.4% of Field Notes within 24 hours.
- A live assistant fetch reached 13% of Field Notes at any point, 2.8% within 24 hours.

Method: First fetch day per canonical page against its site publication time, for items published 22 Aug – 14 Oct (at least a week of follow-up).

Caveat: Windows close at 23:30 UTC, so the hours are upper bounds. Category level, not family. A fetch is not indexing.

Our interpretation: Field Notes are the format AI crawlers pick up fastest; signals sit deeper in the archive and wait longer.

What would change our mind: The share of Field Notes fetched by an AI crawler within 48 hours falls below 80% in the next window.

## 4. More than half of 'live assistant' traffic is not assistants answering people: it is probing for secrets.

- 1,829 requests named a live assistant fetcher (ChatGPT-User, Claude-User, Perplexity-User) in 62 days.
- 52.5% went to paths that do not exist here: cloud credentials, agent configs (.claude, .codex), SSRF-style /fetch/ and /proxy/.
- 34.1% fetched the homepage; 8.6% fetched an article (139 distinct).
- Verified ChatGPT-User: 677 requests, 43 article reads, 99.1% HTML, no Markdown. Live fetches fell from 1,231 to 696 between halves.

Method: User-agent category from the end-of-day aggregates; destinations classified by route. Probe paths are excluded before identity checks.

Caveat: We cannot say how many probe-path requests came from the assistants' own networks.

Our interpretation: For a small site, the live-answer channel is mostly people asking an assistant about the site itself, plus scanners borrowing a trusted name.

What would change our mind: Article fetches overtake homepage fetches among verified live-assistant requests.

## 5. A crawler's name is a claim: 65% of requests calling themselves Googlebot were not Google.

- Googlebot: 3,199 failed claims vs 1,756 verified; failed claims got a success response 2% of the time.
- Across nine families with a published verifier, 8.7% of claims failed. Success rate: 98.6% for verified, 6.1% for failed claims.
- 242 requests named Google-Extended, a robots.txt token that has no crawler of its own.
- 47.5% of named requests come from families with no published verification method, among them ClaudeBot, Meta-ExternalAgent, AhrefsBot and SemrushBot.

Method: Official CIDR lists or forward-confirmed reverse DNS, checked privately per request; only status counts are published.

Caveat: Failed verification can also mean a verifier list went stale; ours are refreshed daily and listed in /crawler-journeys.json.

Our interpretation: Any per-bot ranking that does not verify identity overstates Googlebot and blends scanners into 'AI traffic'.

What would change our mind: Google publishes ranges under which those requests verify.

## 6. Machines read page alternates, not site maps.

- AI crawlers fetched llms.txt or llms-full.txt 15 times in 62 days, against 39,404 AI-crawler requests.
- Named automation fetched Markdown twins 36,089 times and JSON twins 35,982 times; for GPTBot about half of all requests were twins.
- Typed next-reads: 49 of 4,062 exposed verified sessions (1.2%) took one; 1.8% of transitions matched.
- Manifest: 2,653 verified sessions read /machine-consumption.json; 12.9% fetched content within 30 minutes. Applebot: 1,291 sessions, 1 follow-through.

Method: Format counts from the end-of-day aggregates; journeys from verified 30-minute sessions with alternate representations collapsed to one material.

Caveat: Every page's footer script requests /machine-consumption.json, so many manifest reads are probably a side effect of JavaScript rendering, not navigation.

Our interpretation: Crawlers find .md and .json twins through each page's alternate links and ignore the maps we write for them.

What would change our mind: AI-crawler fetches of llms.txt exceed one per 1,000 AI-crawler requests, or the documented next-read rate passes 10%.

## 7. The agentic web arrived in one wave, and it shakes hands rather than uses tools.

- 19 of 21 registry and MCP-collector families were first seen on or after 25 Sep, the day the MCP server was listed in the MCP registry and the agent card went live.
- Their requests: 338 → 19,110 between halves.
- MCP endpoint: 74,413 JSON-RPC messages from 303 client names; 97.9% initialize, list or discover; 1.5% tool calls.
- 27 clients ever called a tool; one made 54.8% of all calls, and one liveness monitor sent 38.8% of all messages.

Method: Registry first-seen dates from the history and end-of-day aggregates; MCP method counts per client name from the endpoint's own log, our smoke tests removed.

Caveat: MCP counting began on 26 Sep, so pre-listing traffic is unmeasured; client names are self-declared.

Our interpretation: Directories catalogue agent-ready sites quickly, but almost nothing yet uses the tools they catalogue.

What would change our mind: Tool calls exceed 10% of MCP messages in a 28-day window.

## Methods and limits

- Sources: the public aggregates listed below, plus private end-of-day aggregates computed on the server from access logs. Only counts leave the server; no IP address, raw user agent, query string, session or per-visitor record is published here.
- Units are successful and failed HTTP requests unless a line says sessions (30-minute inactivity windows per claimed family) or materials (one canonical page with its HTML, Markdown and JSON forms collapsed).
- Excluded everywhere: New Runtime monitoring and utility clients, marked synthetic checks, scanners and sensitive-path probes, and the readership widget's own requests.
- Identity: 'verified' means the request matched the operator's published IP ranges or forward-confirmed reverse DNS. 'Not verifiable' means the operator publishes no supported method. A name alone is never treated as identity.
- Correlation, not causation: request order is a server-side sequence, not a crawler's decision path, and a fetch proves neither indexing, training nor citation.
- Instrument note: the public daily history (/bot-traffic-history.json) keeps the last, partial view of each day; for 21 Aug – 21 Oct it holds 22% of the AI-crawler requests the end-of-day aggregates counted. We use it only for first-seen dates. The 'median discovery latency' in /crawler-journeys.json is censored by about two days of log retention, so the speed section uses publication timestamps instead.
- Interpretation lines are New Runtime's own reading of the numbers; each finding names the result that would change it.

## Data

- [Report aggregates (this report, JSON)](https://newruntime.com/research/state-of-machine-readership-2026-10.json)
- [Report as Markdown](https://newruntime.com/research/state-of-machine-readership-2026-10.md)
- [Bot traffic history (93 days, public)](https://newruntime.com/bot-traffic-history.json)
- [Machine consumption (live, 30-minute refresh)](https://newruntime.com/machine-consumption.json)
- [Crawler journeys (24h / 7d / 30d)](https://newruntime.com/crawler-journeys.json)
- [Visitor dossiers](https://newruntime.com/readers.json)
- [Discovery graph](https://newruntime.com/discovery-graph.json)
