---
schema_version: "newruntime-reader-dossiers-v1"
type: "reader_dossier"
slug: "screaming-frog-seo-spider"
kind: "seo"
operator: "Screaming Frog"
researched_at: "2026-10-23"
confidence: "high"
json_url: "https://newruntime.com/readers/screaming-frog-seo-spider.json"
canonical_url: "https://newruntime.com/readers/screaming-frog-seo-spider/"
---

# Screaming Frog SEO Spider

> Desktop SEO crawler run by individual users (agencies, site owners) to audit sites on demand; not a central index.

## Dossier

- Kind: SEO & backlink crawlers
- Operator: Screaming Frog (https://www.screamingfrog.co.uk/)
- Documentation: https://www.screamingfrog.co.uk/seo-spider/faq/
- robots.txt tokens: Screaming Frog SEO Spider
- Feeds: research
- Respects robots.txt: Partly, per its documentation. FAQ: robots.txt compliant by default, but users can ignore robots.txt and change the user-agent.
- Verification: No published method
- Labels matched in our logs: screaming frog seo spider

## Why it matters to New Runtime (our interpretation)

A hit means a person (often an SEO/competitor) ran an audit of our site; signals human interest, not machine readership.

## Observed behaviour

- Baseline 2026-10-24 (94-day window): 4 requests on 2 days; first seen 2026-08-18, last seen 2026-10-02; Dormant.
- Live numbers: https://newruntime.com/bot-traffic-history.json and https://newruntime.com/crawler-journeys.json

## Sources

- https://www.screamingfrog.co.uk/seo-spider/faq/

Identity: A family name is what a client claims in its user agent. It is verified only when its request came from the operator's published IP ranges or passed forward-confirmed reverse DNS, as reported per family in /crawler-journeys.json. Anyone can claim any name.

All dossiers: https://newruntime.com/readers.md