SEO & backlink crawlers

CrawlCheck

Free site audit that checks whether AI and search crawlers can read a site. The CrawlCheck/1.0 UA fetches machine files and control paths (robots.txt, sitemaps, llms.txt, API catalog); the homepage is additionally requested once under ~12 other crawler identities (GPTBot, ClaudeBot, Googlebot, etc.).

Dossier

Operator: CrawlCheck (crawlcheck.io)
Documentation: crawlcheck.io
robots.txt token: CrawlCheck
Feeds: monitoring
Follows robots.txt: Yes, per its documentationDocs: 'User-agent: CrawlCheck' Disallow: / or 'CrawlCheck: none' opts out on next scan. CrawlCheck/1.0 requests are Web Bot Auth-signed; measurement probes under other crawlers' UAs are unsigned. No owned IP range (egress via Cloudflare).
How to verify: No published method
Names in our logs: crawlcheck
Researched: 23 Oct 2026

Observed on New Runtime

Baseline 2026-10-23 · 93-day window · loading live numbers…

Requests: 12
Days active: 4 / 93
First seen: 10 Oct 2026
Last seen: 22 Oct 2026

Returningother_bot

Daily requests across the observed window, square-root scale.

Identity check claimed vs verified

No per-family identity check is published for this reader; its name is a claim.

What it reads crawler journeys · current window

The crawler-journeys aggregate tracks reading depth for the busiest named crawlers only; this family is not among them.

Sources

crawlcheck.iocrawlcheck.io/policy
crawlcheck.iocrawlcheck.io/