Readership observatory / visitor dossiers

Who reads New Runtime by machine

Most of this site's readers are software. Here is a dossier on each one: who runs it, what it is built to do, whether it follows robots.txt, how to tell the real one from an impostor, and what it did on New Runtime.

Families profiled
93
Requests in window
93,955
New this week
10
Seen, not profiled
—

Baseline 2026-10-23, 93-day window. Loading live numbers…

01

AI search & answer engines 16

Fetch or index pages so an AI assistant or AI search engine can answer a person's question, usually with a link back.

  1. PerplexityBot / Perplexity-User

    Perplexity

    PerplexityBot surfaces and links websites in Perplexity search results and is not used to crawl content for AI foundation models. The separate Perplexity-User fetcher opens pages on user request to answer a question and link the page.

    • search index
    • live answer fetch
    Returning4,689req · 40d
  2. OAI-SearchBot

    OpenAI

    OpenAI's search crawler that surfaces and links websites in ChatGPT search features; not used for model training.

    • search index
    Regular4,407req · 86d
  3. ExaSearchBot / ExaUser

    Exa

    ExaSearchBot crawls public pages to index them for Exa's search, retrieval and citation API; ExaUser fetches a page live when a user or an app built on the Exa API asks for its current content.

    • search index
    • live answer fetch
    Regular2,234req · 21d
  4. SofyaBot

    Sofya

    Crawler for Sofya's own independent search index (served at search.sofya.co); Sofya sells a web search, fetch and extraction API to AI agents and developers.

    • search index
    Dormant876req · 1d
  5. ChatGPT-User

    OpenAI

    Fetches web pages for user-initiated actions in ChatGPT, Custom GPTs and GPT Actions; not an automatic crawler and not used to decide Search eligibility.

    • live answer fetch
    Regular812req · 78d
  6. ShapBot

    Parallel Web Systems

    Parallel Web Systems' crawler; it collects publicly available web content to build and maintain Parallel's search index (Parallel sells web search/research APIs for AI agents). A separate Shap-User agent handles user-triggered fetches and does not crawl automatically.

    • search index
    Returning602req · 3d
  7. KeenableBot

    Keenable

    Builds and maintains the Keenable search index, which answers queries from AI agents and applications and cites source pages with links. A separate Keenable-User agent fetches single pages on behalf of a person or their agent.

    • search index
    Regular252req · 20d
  8. Claude-User

    Anthropic

    Retrieves websites when a Claude user asks a question that requires them (user-initiated fetches).

    • live answer fetch
    Returning118req · 14d
  9. Claude-SearchBot

    Anthropic

    Navigates the web to improve the relevance and accuracy of Claude's search results.

    • search index
    Returning43req · 4d
  10. Kimi bots

    Moonshot AI

    Moonshot AI's Kimi crawlers: KimiBot crawls content potentially used to train Kimi foundation models; Kimi-SearchBot builds the index for Kimi search; Kimi-User fetches pages for user-initiated actions. 'MoonshotBot' is listed only by third-party directories and is not on Moonshot's crawler page.

    • model training
    • search index
    • live answer fetch
    Returning17req · 2d
  11. GrokBot / xAI-SearchBot

    xAI (unconfirmed; UA claims x.ai)

    User agents that claim to be xAI (GrokBot/1.0, xAI-SearchBot/1.0 with +https://x.ai). Third-party directories describe xAI-SearchBot as fetching/indexing pages for Grok search and answers; GrokBot's purpose is undisclosed. xAI publishes no crawler documentation.

    • unknown
    Returning8req · 3d
  12. YouBot

    You.com

    You.com's web crawler that discovers and indexes public pages to return real-time results in You.com's AI search.

    • search index
    Returning8req · 2d
  13. DuckAssistBot

    DuckDuckGo

    Crawls pages in real time to supply DuckDuckGo's AI-assisted answers (DuckAssist), which cite their sources. DuckDuckGo states the data is not used to train AI models.

    • live answer fetch
    Returning6req · 4d
  14. Google-CloudVertexBot

    Google

    Google crawler that performs crawls requested by site owners to build Vertex AI Agents; Google says it has no effect on Google Search or other products.

    • research
    • unknown
    Dormant5req · 1d
  15. cohere-ai

    Cohere

    Undocumented agent with UA 'cohere-ai; +https://cohere.com/crawler'. Secondary sources say it is possibly dispatched by Cohere's AI chat products to fetch content for user prompts; Cohere has not stated its purpose, and cohere.com/crawler returned 404 on 2026-10-23.

    • unknown
    Dormant4req · 1d
  16. Meta-ExternalFetcher

    Meta

    Fetches individual links at a user's request, including for agentic AI tasks in Meta AI products.

    • live answer fetch
    New this week2req · 1d
02

AI model crawlers 14

Collect pages for model training or open web datasets, plus the robots.txt control tokens that govern that use.

  1. Meta-ExternalAgent

    Meta

    Meta crawler for use cases such as training foundation AI models or improving products by indexing content directly.

    • model training
    • search index
    Returning7,984req · 12d
  2. ClaudeBot

    Anthropic

    Anthropic's crawler collecting public web content that may contribute to training its generative AI models.

    • model training
    Regular3,677req · 62d
  3. Reflectionbot

    Reflection AI

    Reflection AI's web crawler. It visits publicly available web pages to help build and improve the company's (open) AI models; it is currently the only user agent Reflection AI crawls with.

    • model training
    Regular2,345req · 23d
  4. Amazonbot

    Amazon

    Amazon crawler used to improve Amazon products and services; Amazon states its data may be used to train Amazon AI models. Related agents: Amzn-SearchBot (search, e.g. Alexa; no AI training) and Amzn-User (live user-request fetches).

    • model training
    • search index
    Regular2,267req · 24d
  5. GPTBot

    OpenAI

    OpenAI's crawler for content that may be used to train its generative AI foundation models.

    • model training
    Returning2,236req · 43d
  6. Google-Extended

    Google

    Not a crawler: a standalone robots.txt product token that lets publishers control whether content crawled by Google may be used to train future Gemini models and for grounding in Gemini Apps and Vertex AI. It has no separate HTTP user-agent string; crawling uses existing Google user agents.

    • model training
    Dormant106req · 7d
  7. Bytespider

    ByteDance

    ByteDance crawler described by secondary sources as an AI data scraper, allegedly used to build training data for ByteDance LLMs such as Doubao. No operator documentation was opened.

    • model training
    Returning29req · 4d
  8. CCBot

    Common Crawl

    Crawler of the non-profit Common Crawl, which builds an open web archive; the corpus is widely reused, including as AI training data.

    • model training
    • research
    Returning29req · 8d
  9. DeepSeekBot

    DeepSeek

    Reported as a DeepSeek crawler collecting web content for AI model training (UA 'DeepSeekBot/1.0; +https://www.deepseek.com/bot'). The referenced deepseek.com/bot page returned 404 on 2026-10-23, so there is no confirmed first-party documentation.

    • model training
    Returning15req · 5d
  10. ChatGLM-Spider

    Zhipu AI (unconfirmed; UA claims zhipuai.cn)

    Claims to be Zhipu AI's crawler (UA 'ChatGLM-Spider/1.0; +https://zhipuai.cn/'); third-party directories say it collects data to train and evaluate ChatGLM/GLM models. No Zhipu documentation found.

    • model training
    Returning7req · 2d
  11. Qwenbot

    Alibaba (unconfirmed; UA claims Qwen)

    Claims to be Alibaba's Qwen crawler (UA 'Qwenbot/1.0; +https://qwen.alibaba.com/'); third-party directories say it collects content for Qwen training and answers. The doc URL they cite (tongyi.aliyun.com/bot) redirects to a 404; no Alibaba documentation found.

    • unknown
    Returning7req · 2d
  12. Applebot-Extended

    Apple

    Secondary user agent that does not crawl; it only governs whether data already crawled by Applebot may be used to train Apple's foundation models.

    • model training
    Dormant5req · 2d
  13. YiBot

    01.AI (unconfirmed; UA claims 01.ai)

    Claims to be 01.AI's crawler (UA 'YiBot/1.0; +https://01.ai/'); a third-party directory describes it as collecting training data. No 01.AI documentation found.

    • unknown
    Dormant4req · 1d
  14. PanguBot

    Huawei (unconfirmed; UA contact pangubot@huawei.com)

    Claims to be Huawei's crawler (UA 'compatible;PanguBot;pangubot@huawei.com'); directories say it collects training data for Huawei's PanGu models. No Huawei documentation found.

    • model training
    Dormant3req · 1d
04

Agentic web & MCP registries 21

Directories, liveness monitors and verifiers for MCP servers, agent cards and machine entry points. The newest and fastest-growing group.

  1. mcpbeat

    mcpbeat (theevoq)

    Independent monitor of public MCP servers. Every 15 minutes it sends an MCP initialize handshake followed by tools/list, resources/list and prompts/list (never tools/call, never authenticates), and it also reviews registry, npm/PyPI and source metadata; results feed public uptime and ranking pages.

    • monitoring
    • registry / directory
    Regular3,075req · 28d
  2. rokmcp-collector

    Operator unknown

    Self-described in its user-agent as "rokmcp-collector/0.2", an MCP-related collector; rokmcp.com exists but returned HTTP 503 and no description could be retrieved. No operator page or public documentation could be found as of 2026-10-23.

    • registry / directory
    Regular591req · 24d
  3. agent-tools.cloud crawler

    agent-tools.cloud

    Crawler for agent-tools.cloud, an open discovery directory of x402 paid APIs, remote MCP servers and A2A agents. Per its about page it runs initialize + tools/list conformance checks on MCP servers, fetches and parses A2A agent cards, and re-probes availability daily.

    • registry / directory
    • monitoring
    Regular395req · 26d
  4. BrickBlueBot

    brick.blue

    Crawler for brick.blue's registry of agents and tools. It reads DNS TXT (_agent/_mcp), .well-known agent/MCP cards, llms.txt, openapi.json and agents.json, runs MCP initialize and tools/list (plus up to three no-argument tools/call on read-only-looking tools), and reads x402 402 responses without paying.

    • registry / directory
    Regular355req · 20d
  5. OraBot

    Ora

    Ora's on-demand scanner that scores how ready a website is for AI agents. When someone requests a score for a domain it fetches robots.txt, llms.txt, OpenAPI specs, MCP manifests, structured data and /.well-known/poppy.json in a bounded burst; during a scan it also requests the homepage under other crawlers' UAs (GPTBot, ClaudeBot, PerplexityBot).

    • monitoring
    Dormant164req · 1d
  6. GolemreachTrustBot

    Golemreach

    Outside-in trust prober for remote MCP servers listed in the official MCP registry: one GET, one initialize and one tools/list per endpoint at most every 6 hours (never calls tools or sends credentials), plus fetching A2A agent cards; results are published at golemreach.com/trust.

    • monitoring
    • registry / directory
    Regular160req · 22d
  7. WellknownBot

    Operator unknown

    Self-described in its user-agent as "WellknownBot/0.1" with listing and claim URLs, i.e. a directory crawler that creates a listing for the site and offers a claim link. No operator page or public documentation could be found as of 2026-10-23.

    • registry / directory
    Regular104req · 20d
  8. TalandorBot

    Talandor

    Likely the checker behind Talandor, an MCP server checker and monitoring tool that indexes Official MCP Registry entries, negotiates protocol versions and reads public tools/list results without calling tools. Talandor's pages do not document the TalandorBot user-agent.

    • monitoring
    Returning96req · 7d
  9. AgenstryBot

    Agenstry

    Crawler for Agenstry, an AI agent directory for A2A and MCP that publishes measured evidence about public agent infrastructure. It fetches .well-known A2A cards and MCP manifests, sitemaps and metadata, may send a JSON-RPC ping or MCP initialize/tools/list, and rechecks live agents roughly hourly.

    • registry / directory
    • monitoring
    Returning73req · 9d
  10. 7Maps-bot

    7IT (7IT Solutions)

    Crawler behind 7Maps, a live map of public MCP servers. It visits servers listed in public MCP registries and sends only initialize and tools/list (plus a daily server/discover), never calling tools; it publishes reachability, latency, tool counts and an estimated per-tool risk level.

    • registry / directory
    • monitoring
    Returning56req · 7d
  11. histor

    Operator unknown

    Self-described in its user-agent as "histor/0.1.0" making read-only MCP probes (initialize + tools/list only). No operator page or public documentation could be found as of 2026-10-23.

    • monitoring
    Returning21req · 4d
  12. KnownGood-Verifier

    Operator unknown

    Self-described in its user-agent as "KnownGood-Verifier/0.3", a verification bot (by name, verifying known-good endpoints). No operator page or public documentation could be found as of 2026-10-23.

    • security scan
    Dormant20req · 1d
  13. MCPHereBot

    Operator unknown

    Self-described in its user-agent as "MCPHereBot/1.0", an MCP-focused bot. No operator page or public documentation could be found as of 2026-10-23.

    • registry / directory
    Returning15req · 3d
  14. StealthStackCrawler

    StealthStack

    Crawler for StealthStack, 'the index of the agentic web', which crawls Agentic Resource Discovery (ARD) catalogs and indexes the skills, MCP servers, A2A agents and catalogs they publish (about 48.9k resources from 4.8k domains as of 2026-10-22). It identifies itself with did:web:stealthstack.ai, whose DID document resolves at stealthstack.ai.

    • registry / directory
    Returning13req · 5d
  15. GlintbaseResearch

    Glintbase

    Research agent of Glintbase, an applied AI research lab building agent-readiness infrastructure (Agent Readiness Standard, scanner). Its research page describes fetching sites 'as a declared research agent' for agent-readiness studies; no crawler documentation is published. The UA's contact address (research@glintbase.dev) ties it to Glintbase.

    • research
    Returning9req · 1d
  16. AgentTrustBot

    agenttru.st

    Crawler for the agenttru.st directory of AI agents. It discovers hostnames from public Certificate Transparency logs and checks well-known discovery paths (agent cards, MCP server cards) and DNS records (ANS, DNS-AID) for published agents, plus icon fetches and re-checks of listed agents. Formerly AgentTrust-Collector.

    • registry / directory
    Dormant6req · 1d
  17. GAIP-Observatory

    GAIP (gaipagents.com)

    The GAIP Agent Observatory records dated, observed facts about public MCP servers and A2A agent cards, checking each at most once a day and keeping a hash-chained history of manifests and changes. It reads agent cards, registry server.json and the MCP discovery handshake (initialize, tools/list, resources/list); it never calls tools or authenticates and gives no scores.

    • registry / directory
    • monitoring
    Returning6req · 6d
  18. MCPStationBot

    Protocol Holdings Co., Ltd. (MCP Station)

    Checker for MCP Station, which lists remote MCP servers published in the Official MCP Registry and checks each daily for reachability, required authentication, MCP version and offered tools.

    • registry / directory
    • monitoring
    New this week6req · 2d
  19. x402-router-crawler

    Operator unknown

    Self-described in its user-agent as "x402-router-crawler/4.0", a crawler for an x402 payment-routing service. No operator page or public documentation could be found as of 2026-10-23.

    • registry / directory
    New this week6req · 3d
  20. AgentIndexBot

    Operator unknown

    Self-described in its user-agent as "AgentIndexBot/0.1", a 'polite ARD crawler' (Agentic Resource Discovery catalogs). No operator page or public documentation could be found as of 2026-10-23.

    • registry / directory
    New this week4req · 3d
  21. mcpcensusbot

    The MCP Census

    Crawler for The MCP Census, a service for checking MCP servers before install. When a search finds no census hit for a brand, it looks for an off-registry server with GET /.well-known/mcp.json, GET /.well-known/mcp-registry-auth and a DoH lookup for mcp.<domain>; GET only, results cached 7–30 days.

    • registry / directory
    New this week4req · 2d
05

SEO & backlink crawlers 11

Build link graphs and page databases for SEO platforms.

  1. AhrefsBot

    Ahrefs

    Ahrefs' crawler that builds the link/page database behind the Ahrefs SEO platform and the Yep search engine. A separate AhrefsSiteAudit agent runs site audits.

    • SEO backlink index
    • search index
    Regular13,472req · 89d
  2. SemrushBot

    Semrush

    Semrush's crawler that discovers links and revisits pages for its Backlink Analytics index and many tools (Site Audit, Backlink Audit, writing/plagiarism checks, etc.).

    • SEO backlink index
    Regular9,333req · 84d
  3. MJ12bot

    Majestic

    Majestic's distributed crawler that maps link relationships between websites for the Majestic Site Explorer backlink index; it does not cache page content.

    • SEO backlink index
    Regular4,177req · 48d
  4. DataForSeoBot

    DataForSEO

    DataForSEO's backlink-checker crawler; adds newly found links to its backlink database and rechecks existing ones, which is resold via API to SEO tools.

    • SEO backlink index
    Dormant2,638req · 1d
  5. serpstatbot

    Serpstat

    Serpstat's crawler that discovers new links and tracks changes for the Serpstat backlink index.

    • SEO backlink index
    Returning1,152req · 2d
  6. DotBot

    Moz

    Moz's web crawler that gathers link and site data for Moz's link index and SEO tools (UA references opensiteexplorer.org/dotbot).

    • SEO backlink index
    Regular338req · 25d
  7. SERankingBacklinksBot

    SE Ranking

    SE Ranking's crawler for its backlink database used in the SE Ranking SEO platform.

    • SEO backlink index
    Regular315req · 19d
  8. Barkrowler

    Babbar

    Babbar's crawler that builds a graph of the public web (links and page metadata, not content) to power SEO and online-marketing metrics.

    • SEO backlink index
    Returning61req · 2d
  9. CrawlCheck

    CrawlCheck (crawlcheck.io)

    Free site audit that checks whether AI and search crawlers can read a site. The CrawlCheck/1.0 UA fetches machine files and control paths (robots.txt, sitemaps, llms.txt, API catalog); the homepage is additionally requested once under ~12 other crawler identities (GPTBot, ClaudeBot, Googlebot, etc.).

    • monitoring
    Returning12req · 4d
  10. AgentReadyBot

    Robots Drinking Tea LLC

    On-demand AI-readiness scanner: when someone requests a report it reads up to 25 pages plus robots.txt and the homepage, compares what humans and machines get, and sends test requests with seven AI crawler names to see whether the site blocks them.

    • research
    Dormant10req · 1d
  11. Screaming Frog SEO Spider

    Screaming Frog

    Desktop SEO crawler run by individual users (agencies, site owners) to audit sites on demand; not a central index.

    • research
    Dormant2req · 1d
07

Scanners & probes 9

Internet-wide security scanners and probes. Excluded from the observatory's reader totals.

  1. CensysInspect

    Censys

    Censys's internet-wide scanner; HTTP scans identify as CensysInspect and collect service information for the Censys search/attack-surface dataset without attempting authentication.

    • security scan
    • research
    Regular779req · 67d
  2. zgrab

    Operator unknown

    ZGrab/ZGrab2 is an open-source application-layer scanner from the ZMap project for Internet-wide surveys; its default UA (e.g. 'Mozilla/5.0 zgrab/0.x') is sent by academics, scan indexers and attackers alike.

    • security scan
    • research
    Regular638req · 86d
  3. Infrawatch

    Infrawatch Limited

    UK internet-measurement and security-research scanner that routinely observes publicly exposed hosts and services; says it does not exploit or brute-force.

    • security scan
    • research
    Returning195req · 33d
  4. ModatScanner

    Modat

    Modat's internet-wide scanner that indexes hosts at the machine level for its Internet search product; scanning only, no intrusion.

    • security scan
    Returning53req · 37d
  5. CyberConvoyScout

    CyberConvoy

    Internet-wide scanner (single SYN probes and light banner grabs on a few ports) collecting banners, HTTP headers and TLS certificates for threat intelligence, per a third-party catalog.

    • security scan
    Returning43req · 35d
  6. GenomeCrawlerd

    Nokia (Deepfield)

    Crawler for Nokia Deepfield Secure Genome, a 'security map' of the internet used for traffic analysis and DDoS detection/mitigation in Nokia Deepfield Defender. It systematically crawls the IPv4 and active IPv6 address spaces on specific ports to identify the services hosts provide.

    • security scan
    Returning17req · 3d
  7. GoScanner

    Operator unknown

    Unidentified scanner UA seen in our logs; no public documentation found for this name.

    • unknown
    Dormant8req · 1d
  8. Shodan

    Shodan

    Search engine for Internet-connected devices; its crawlers continuously query hosts and collect service banners for the Shodan index.

    • security scan
    • research
    Returning4req · 4d
  9. OAI-AdsBot

    OpenAI

    Validates the safety of landing pages submitted as ads on ChatGPT; only visits pages submitted as ads and its data is not used for model training.

    • security scan
    New this week2req · 1d
08

Unidentified 4

Clients with no product name. We describe only what they did.

  1. Unnamed Linux-Chrome automation

    Operator unknown

    Clients presenting a desktop Chrome on Linux user agent that the observatory classifies as automation rather than a human browser. No product name, operator or documentation is visible in the published label.

    • unknown
    Regular1,025req · 8d
  2. DocSpider

    Operator unknown

    Self-described in its user-agent as "Mozilla/5.0 (compatible; DocSpider/0.1; +URL)"; no public documentation or operator found (not the same as DocSpotBot).

    • unknown
    Returning21req · 1d
  3. IntentRouteBot

    Operator unknown

    Self-described in its user-agent as "IntentRouteBot/0.1 (+URL)"; no public documentation or operator found.

    • unknown
    New this week10req · 1d
  4. ZeroBot

    Operator unknown

    Self-described in its user-agent as "Mozilla/5.0 (compatible; ZeroBot/1.0; +URL)"; no public crawler documentation found (unrelated to the ZeroBot bot-protection vendor or ZumBot).

    • unknown
    New this week8req · 2d

How to read a dossier

Claimed vs verified

A name is what a client claims in its user agent; anyone can claim to be GPTBot. A family counts as verified only when its requests came from the operator's published IP ranges or passed forward-confirmed reverse DNS. Those checks run per family in /crawler-journeys.json.

Sourced vs ours

Operator, purpose, robots.txt behaviour and verification come from the operator's own documentation or a credible independent page, listed on each dossier. "Why it matters" is New Runtime's interpretation and is marked as such. Facts were researched on 2026-10-23.

Observed, live

Requests, active days and recency are recomputed in your browser from the public aggregates (history, journeys). Family-level counts only: no IP addresses, visitor IDs, raw user agents or query strings.

Who read this page?Machine requests, hidden until opened

Loading the privacy-safe route aggregate…

Open the JSON contract