AI model crawlers
Meta-ExternalAgent
Meta crawler for use cases such as training foundation AI models or improving products by indexing content directly.
Dossier
- Operator
- Meta
- Documentation
- developers.facebook.com
- robots.txt token
meta-externalagent- Feeds
- model training, search index
- Follows robots.txt
- Yes, per its documentationBlocked by a robots.txt disallow for its user agent; changes may take up to 24h.
- How to verify
- No published method
- Names in our logs
meta-externalagent- Researched
- 23 Oct 2026
Observed on New Runtime
Baseline 2026-10-23 · 93-day window · loading live numbers…
- Requests
- 7,984
- Days active
- 12 / 93
- First seen
- 25 Sept 2026
- Last seen
- 23 Oct 2026
Returningai_crawler · other_bot
Identity check claimed vs verified
- No per-family identity check is published for this reader; its name is a claim.
What it reads crawler journeys · current window
- The crawler-journeys aggregate tracks reading depth for the busiest named crawlers only; this family is not among them.
Enters at · 7 days
Leaves from · 7 days
Sources
- developers.facebook.comdevelopers.facebook.com/docs/sharing/webmasters/web-crawlers