AI model crawlers
CCBot
Crawler of the non-profit Common Crawl, which builds an open web archive; the corpus is widely reused, including as AI training data.
Dossier
- Operator
- Common Crawl
- Documentation
- commoncrawl.org
- robots.txt token
CCBot- Feeds
- model training, research
- Follows robots.txt
- Yes, per its documentationOpt out with User-agent: CCBot / Disallow: /. Common Crawl warns that other crawlers spoof CCBot.
- How to verify
- Published IP ranges and reverse DNSindex.commoncrawl.org
- Names in our logs
ccbot- Researched
- 23 Oct 2026
Observed on New Runtime
Baseline 2026-10-23 · 93-day window · loading live numbers…
- Requests
- 29
- Days active
- 8 / 93
- First seen
- 23 Jul 2026
- Last seen
- 21 Oct 2026
Returningai_crawler
Identity check claimed vs verified
- not verifiable2 reqprovider not published
What it reads crawler journeys · current window
- Content pages0
- Feeds & discovery0
- Machine-consumption0
0 unique materials · 0 canonical pages
Enters at · 7 days
Leaves from · 7 days
Sources
- commoncrawl.orgcommoncrawl.org/ccbot