---
schema_version: "newruntime-reader-dossiers-v1"
type: "reader_dossier"
slug: "ccbot"
kind: "ai_training"
operator: "Common Crawl"
researched_at: "2026-10-23"
confidence: "high"
json_url: "https://newruntime.com/readers/ccbot.json"
canonical_url: "https://newruntime.com/readers/ccbot/"
---

# CCBot

> Crawler of the non-profit Common Crawl, which builds an open web archive; the corpus is widely reused, including as AI training data.

## Dossier

- Kind: AI model crawlers
- Operator: Common Crawl (https://commoncrawl.org/)
- Documentation: https://commoncrawl.org/ccbot
- robots.txt tokens: CCBot
- Feeds: model training, research
- Respects robots.txt: Yes, per its documentation. Opt out with User-agent: CCBot / Disallow: /. Common Crawl warns that other crawlers spoof CCBot.
- Verification: Published IP ranges and reverse DNS (https://index.commoncrawl.org/ccbot.json)
- Labels matched in our logs: ccbot

## Why it matters to New Runtime (our interpretation)

Common Crawl snapshots are downstream inputs for many open and commercial models — one crawl, many future readers.

## Observed behaviour

- Baseline 2026-10-23 (93-day window): 29 requests on 8 days; first seen 2026-07-23, last seen 2026-10-21; Returning.
- Reading mix (crawler journeys): 0 content reads, 0 feed or discovery reads, 0 machine-consumption reads; 0 unique materials.
- Identity check: not verifiable 2 requests.
- Live numbers: https://newruntime.com/bot-traffic-history.json and https://newruntime.com/crawler-journeys.json

## Sources

- https://commoncrawl.org/ccbot

Identity: A family name is what a client claims in its user agent. It is verified only when its request came from the operator's published IP ranges or passed forward-confirmed reverse DNS, as reported per family in /crawler-journeys.json. Anyone can claim any name.

All dossiers: https://newruntime.com/readers.md