AI model crawlers

Bytespider

ByteDancemedium confidence

ByteDance crawler described by secondary sources as an AI data scraper, allegedly used to build training data for ByteDance LLMs such as Doubao. No operator documentation was opened.

Dossier

Operator: ByteDance
Documentation: None published
robots.txt token: Bytespider
Feeds: model training
Follows robots.txt: Not documentedSecondary source lists it as expected to follow robots.txt, but ByteDance publishes no verification method; compliance unconfirmed.
How to verify: No published method
Names in our logs: bytespider
Researched: 23 Oct 2026

Observed on New Runtime

Baseline 2026-10-23 · 93-day window · loading live numbers…

Requests: 29
Days active: 4 / 93
First seen: 11 Sept 2026
Last seen: 21 Oct 2026

Returningother_bot

Daily requests across the observed window, square-root scale.

Identity check claimed vs verified

No per-family identity check is published for this reader; its name is a claim.

What it reads crawler journeys · current window

The crawler-journeys aggregate tracks reading depth for the busiest named crawlers only; this family is not among them.

Sources

knownagents.comknownagents.com/agents/bytespider