MACHINE DISCOVERY LAB / LIVE EXPERIMENT

Can a manifest become a path into the site?

/machine-consumption.json became New Runtime’s leading machine route in one 24-hour snapshot. This Lab turns that attention into a falsifiable test: do verified crawlers move from the gateway into canonical content, or do they only poll the file?

Manifest reads / 24h
125
Search-classified
121
AI-classified
4
Claimed Applebot / site
274

Starting observation, not attribution: the baseline did not prove that Applebot read the manifest or followed its links.

01 / MEASUREMENT MODEL

Five outcomes, no composite score

Each stage can fail independently. Keeping them separate makes polling, shallow crawl, and real return traffic visible.

  1. 01DiscoveryManifest, feeds, sitemap, API Catalog
  2. 02CrawlVerified canonical-node coverage
  3. 03Understanding proxyFormats and graph traversal
  4. 04RecallExact canonical citation probes
  5. 05AuthorityAttributable referrals and citations
02 / OBSERVED CRAWL

Expected graph × observed fetches

Machine discovery sequenceA manifest gateway leads to graph shards, canonical content, citation proxies, and attributable return signals.MANIFEST125 readsGRAPH1020 nodesCONTENTawaiting fetchRETURNattributed onlytyped next_readsverified coveragecitation proxyreferral / citation / repeat fetch
Verified graph coverageawaiting snapshot
Manifest → contentawaiting snapshot
Median discovery latencyawaiting snapshot
Verified bot familiesawaiting snapshot
Accessible graph and observation fallback
LayerExpectedObservedInterpretation
GatewayTyped routes in manifest v1.1Baseline: 125 readsRepeated reads alone are polling.
Physical graph10,191 links across 1,020 canonical nodesAwaiting verified overlayCoverage uses nodes available during the window.
Knowledge graph11,897 topic, pattern, and source-evidence edgesAwaiting observed path spanEditorial relationship is distinct from a literal HTML link.
Recall / authorityExact canonical citationProbe status: not_runNo provider account is simulated.
03 / IDENTITY CONFIDENCE

A bot name is a claim until verified

Official CIDR ranges or forward-confirmed reverse DNS are checked privately. IPs, hostnames, raw User-Agents, and sessions never enter public files.

verified

Matched a supported provider verifier.

claimed_unverified

Named family, verifier mismatch or unavailable evidence.

not_verifiable

No supported public identity method.

Observed bot families by identity status
Claimed familyStatusRequestsCanonical nodes
Awaiting the first identity-aware production snapshot.
04 / JOURNEY DEPTH

How far each observed bot family travelled

Alternate HTML, Markdown, and JSON representations collapse into one material. Paths are privacy-safe family aggregates from short request windows, not persistent subjects or proof of crawler intent.

Observed crawler journeys for the selected window
Bot familyIdentitySessionsUnique materialsMaterial readsDepth median / maxObserved path
Awaiting the first journey-aware production snapshot.

“Path” means ordered successful server requests within a 30-minute inactivity window. IP is used only in memory to keep concurrent streams apart and is never published.

05 / PRE-REGISTRATION

Three ways this can fail usefully

Traffic growth is an external result, not a release criterion. The system succeeds by producing honest, reproducible evidence.

01 / manifest-gateway

Typed next_reads in the manifest increase verified downstream content reads within 30 minutes.

Primary window: 7d. Report at 7 days; keep 30-day collection running. Do not claim causality from request order.

  • manifest_sessions
  • manifest_followthrough_sessions
  • followthrough_percent
  • polling_only_sessions
02 / graph-discovery

Publishing the graph increases verified canonical-node coverage or reduces median discovery latency.

Primary window: 7d. Promising if coverage expands across two verified families or median latency improves by at least 25%.

  • eligible_nodes
  • covered_nodes
  • coverage_percent
  • median_discovery_latency_seconds
03 / authority-distribution

A public methods note creates attributable referrals, citations, or repeat verified fetches.

Primary window: 7d. External provider absence is not_run. Publication to social channels requires a separate owner approval and receipt.

  • referrals
  • verified_repeat_fetches
  • canonical_citations
  • ttfu_proxy_seconds
06 / EVIDENCE BOUNDARY

What the Lab proves — and what it cannot

These data can show

  • Whether a claimed crawler passed the available provider verifier.
  • Which canonical nodes and public representations it fetched.
  • Whether content reads followed a manifest read inside a bounded window.
  • How long first verified discovery took after publication.

These data do not show

  • Why a crawler chose a URL or whether the manifest caused the fetch.
  • Whether a page entered model training data or internal model memory.
  • That IndexNow submission became indexing, ranking, or referral traffic.
  • Any individual visitor, IP address, raw session, or private discovery source.
Who read this page?Machine requests, hidden until opened

Loading the privacy-safe route aggregate…

Open the JSON contract