SEO & backlink crawlers
Screaming Frog SEO Spider
Desktop SEO crawler run by individual users (agencies, site owners) to audit sites on demand; not a central index.
Dossier
- Operator
- Screaming Frog
- Documentation
- screamingfrog.co.uk
- robots.txt token
Screaming Frog SEO Spider- Feeds
- research
- Follows robots.txt
- Partly, per its documentationFAQ: robots.txt compliant by default, but users can ignore robots.txt and change the user-agent.
- How to verify
- No published method
- Names in our logs
screaming frog seo spider- Researched
- 23 Oct 2026
Observed on New Runtime
Baseline 2026-10-23 · 93-day window · loading live numbers…
- Requests
- 2
- Days active
- 1 / 93
- First seen
- 18 Aug 2026
- Last seen
- 18 Aug 2026
Dormantother_bot
Identity check claimed vs verified
- No per-family identity check is published for this reader; its name is a claim.
What it reads crawler journeys · current window
- The crawler-journeys aggregate tracks reading depth for the busiest named crawlers only; this family is not among them.
Enters at · 7 days
Leaves from · 7 days
Sources
- screamingfrog.co.ukwww.screamingfrog.co.uk/seo-spider/faq/