Inside Firecrawl: scrape, crawl, map, extract and agent

What it does

• Scrape. Turns any URL into clean Markdown or structured data for AI agents, including JS-heavy pages and dynamic apps, and can output markdown, structured data, screenshots, or HTML. Each scrape consumes 1 credit, with extra credits for JSON, question, highlights, PII redaction, PDF parsing, and audio or video. How: POST https://api.firecrawl.dev/v2/scrape with JSON {"url":"https://firecrawl.dev","formats":"markdown","html"} or firecrawl.scrape(url, formats=...). source
• Crawl. Submits a starting URL, then recursively discovers and scrapes reachable subpages via sitemap and link traversal, with path filters, depth limits, and subdomain/external-link controls. Each crawled page costs 1 credit; default limit is 10,000 pages. How: POST https://api.firecrawl.dev/v2/crawl with {"url":"https://docs.firecrawl.dev","limit":10}; poll GET /v2/crawl/{jobId}. SDK: firecrawl.crawl(url=..., limit=10) or startcrawl then getcrawlstatus. Pass scrape options on every page via scrapeOptions / scrapeoptions. source
• Map. Takes a website URL and returns most links on the site, discovered primarily from the sitemap plus SERP results and previously crawled pages, without scraping page content. Each map request consumes 1 credit per call regardless of how many URLs are returned. How: POST https://api.firecrawl.dev/v2/map with {"url":"https://firecrawl.dev"}; optional search, limit, sitemap (include/skip/only). SDK: firecrawl.map(url=..., limit=50, sitemap="include"). CLI: firecrawl map https://firecrawl.dev. source
• Extract. Collects structured data from one or many URLs (optional / wildcards) using a prompt and/or JSON schema. Firecrawl crawls, parses, and collates results. Agent is documented as the successor. Extract billing uses credits; each credit is worth 15 tokens. How: POST https://api.firecrawl.dev/v2/extract with urls, prompt, and optional schema; poll GET /v2/extract/{jobId}. SDK: firecrawl.extract(urls=..., prompt=..., schema=...). Optional enableWebSearch and agent.model FIRE-1. source
• Agent. Autonomous web research: searches, navigates, and gathers data from a prompt without requiring URLs. Runs asynchronously on spark-2. Default maxCredits is 2,500. All users get 5 free daily runs; further usage is dynamic credit billing. How: POST https://api.firecrawl.dev/v2/agent with prompt (max 10,000 characters), optional urls, schema, model spark-2, effort low|medium|high, maxCredits. Poll GET /v2/agent/{jobId}. SDK: firecrawl.agent(...) or startagent / getagentstatus. source
• Scrape Formats. Choose one or more output shapes: markdown, summary, html, rawHtml, rawBase64, screenshot, links, json, images, branding, product, audio, video, query/question, and highlights. Screenshot URLs expire after 24 hours; audio and video return signed GCS URLs that expire after 1 hour. How: Set formats on POST /v2/scrape, e.g. formats:"markdown","html" or formats:{"type":"json","schema":{...}} or {"type":"question","question":"..."}. source
• actions. Run browser steps on a page before scraping (wait, click, write, press, scroll, screenshot, and similar) for dynamic or interaction-gated content. Docs recommend Interact over extending the actions array for richer, stateful sessions. How: On POST /v2/scrape include an actions array, e.g. {"type":"click","selector":"buttontype=\"submit\""} then {"type":"wait","milliseconds":1500}. Almost always wait before/after other actions. source
• Self-hosting Firecrawl. Run the open-source stack with Docker Compose when you need source or infrastructure control. The pinned guide uses v2.11.162, serves the API at http://localhost:3002, and covers core scrape, crawl, map, and search. Agent, Browser, Interact, screenshots, and page actions are not in the default stack. How: git clone https://github.com/firecrawl/firecrawl.git && git checkout v2.11.162; write .env with USEDBAUTHENTICATION=false and Postgres settings; docker compose up --build -d; POST http://localhost:3002/v2/scrape. source

Modes

• /v2/scrape. Synchronous single-URL scrape into markdown, HTML, JSON, screenshots, and other formats. Cheaper than Agent when the URL is known. source
• /v2/crawl. Asynchronous recursive crawl from a start URL; returns a job ID. Use limit (default 10,000), includePaths/excludePaths, sitemap, and scrapeOptions. source
• /v2/map. Fast URL discovery without scraping bodies. Optional search ranks links. Prefer /crawl when you need thorough recursive discovery and error records. source
• /v2/extract. Legacy LLM extraction over URLs or domain wildcards, with optional schema, enableWebSearch, and FIRE-1. Job results via API for 24 hours. Documented as Beta. source
• /v2/agent. Autonomous multi-page research from a prompt. No URLs required. Poll GET /v2/agent/{jobId}. Results available via the API for 24 hours. source
• extract. Agent mode (default): returns the complete structured result in data every turn. source
• chat. Agent mode: a follow-up that asks for no new data can get a short reply in message. Required for exchange.requireApproval. source
• spark-2. Default Agent model; every run executes on spark-2. Spark 1 names are deprecated and route to spark-2. Optional effort: low, medium, or high. source

Pricing and limits

• Free: $0 /month. 1,000 credits / month; 500 searches or 1,000 pages scraped; 2 concurrent requests; low rate limits; no credit card. Rate limits 10 / min on /scrape, /map and /search and 2 / min on /crawl and /agent. Pay-as-you-go not available. source
• Hobby: $19/month billed monthly, or $16/month billed annually. 5,000 credits/month (2,500 searches or 5,000 pages scraped); 5 concurrent requests; 1,000 extra credits per $5; 100 / min on /scrape, /map and /search; 20 / min on /crawl and /agent. source
• Standard: $99/month billed monthly ($83/month billed annually). 100,000 credits/month (50,000 searches or 100,000 pages scraped); 25 concurrent requests; 2,000 extra credits per $5; 500 / min on /scrape, /map and /search; 100 / min on /crawl and /agent. source
• Growth: $399/month billed monthly ($333/month billed annually). 500,000 credits/month (250,000 searches or 500,000 pages scraped); 50 concurrent requests; 2,500 extra credits per $5; 5,000 / min on /scrape, /map and /search; 1,000 / min on /crawl and /agent. source
• Scale: $749/month billed monthly ($599/month billed annually). 1,000,000 credits/month (500,000 searches or 1,000,000 pages scraped); 100 concurrent requests; 5,000 extra credits per $5; 10,000 / min on /scrape, /map and /search; 2,000 / min on /crawl and /agent. Credit rollover: one month (Scale). source
• Scrape / Crawl / Map (base): 1 credit per page (scrape or crawl); Map 1 credit / call. JSON format (LLM extraction) +4 credits / page; question or highlights +4 credits / page; PDF parsing +1 credit / PDF page; ZDR +1 credit / page. A scrape that returns no result is not charged; a 403 or 404 document still costs 1 credit. Effective September 4, 2026. Prices in USD. source
• Agent: 5 daily runs free. Dynamic pricing. maxCredits defaults to 2,500 if unset; dashboard supports up to 2,500; values above 2,500 via API are always billed as paid requests. Parallel Agents with Spark-2 Fast: 10 credits per cell. Failed credit-limit runs report creditsUsed: 0. source
• Pay-as-you-go increment: 5 USD per increment. Hobby 1,000 credits per 5 USD; Standard 2,000; Growth 2,500; Scale 5,000. Paid plans only. Free plan requests return HTTP 402 when the allotment runs out. source

Recent changes

• 2026-09-04: Current list prices took effect. Prices in USD. Hobby $19/month monthly or $16/month annual; Standard $99 / $83; Growth $399 / $333; Scale $749 / $599. source
• 2026-06-24: v2.11.0: Firecrawl Research Index; keyless access for /scrape, /search, /interact, and /parse; redactPII; deterministicJson format; video discovery on any page. source
• 2026-06-19: GitHub v2.11.0: PDF download/scrape size cap raised from 30 MB to 50 MB; experimental pii format replaced by redactPII; deterministicJson cannot be combined with json; crawl/batch status gained createdAt, completedAt, duration. source

Alternatives

• Reader API. Jina converts a URL to LLM-friendly text by prepending https://r.jina.ai (GET/POST). Billing is output tokens, not Firecrawl page credits. Without an API key: 20 RPM on r.jina.ai; with a free or paid key: 500 RPM; premium: 5000 RPM. Search at https://s.jina.ai costs a fixed token amount starting from 10,000 tokens per request and is blocked without a key. It does not bypass anti-bot systems. source
• Crawl4AI. Open-source Python crawler you run yourself (pip install -U crawl4ai; AsyncWebCrawler.arun), free forever, or Crawl4AI Cloud with the same engine. Cloud free tier: 1,000 credits every month, 10 a minute, 2 at a time, no card. Paid: 120 a minute, 10 at a time, from $10 a month. Library/self-host has no hosted /search, /serp, or /answer; those are Cloud. source
• Apify. Actor platform billed in prepaid USD usage plus compute units, not Firecrawl credits. Free: $0 with $5 included usage / month, $0.2 / CU, 5 concurrent runs. Starter: $19 / month ($17 / month billed annually) with $19 included usage, $0.2 / CU, 32 concurrent runs. Scale: $199 / month ($179 annually) with $199 included, $0.16 / CU. Unused prepaid usage expires at cycle end. source

Traps

• Map prioritizes speed and may not capture all website links. It relies mainly on the sitemap plus cached crawl data and search results. Use /crawl for thorough recursive discovery, or the scrape links format for one page. source
• /extract is still in Beta: full coverage of massive sites in one request is not supported; complex logical queries can miss rows; results can vary across runs. Domain wildcards (example.com/) are experimental. source
• If you set crawl limit higher than remaining credits, the job returns HTTP 402 instead of silently shrinking. Staff confirmed that is intended; omit limit or set it below the remaining balance. Default limit is 10,000 pages. source
• Default crawl only follows children of the start URL (website.com/other-parent/blog-1 is skipped when crawling website.com/blogs/). Set crawlEntireDomain for siblings/parents, allowSubdomains for subdomains. robots.txt is respected unless ignoreRobotsTxt (Enterprise only). source
• Self-host default stack has no screenshots or page actions (both need Fire-engine). Agent, Browser, Interact, and specialized product/audio/video formats are Cloud (or need extra external services). The Compose quickstart disables auth and is not a production architecture. source
• A page returned with 403 or 404 is still a billed document (1 credit). Only scrapes that return no document are free. Check metadata.statusCode; HTTP 200 from Firecrawl does not mean the target page was 2xx. source
• If an Agent run hits maxCredits it reports status failed with stopReason creditlimitreached and does not set data (partial may be present and incomplete). Cancellation is cooperative, so credits can still accrue after you cancel. Job results expire from the API after 24 hours. source
• Agent cost is dynamic. Official guidance: most runs consume a few hundred credits; a single run returns roughly 150-200 rows; split large jobs. Firecrawl recommends JSON mode on /scrape when the URL is known because it is cheaper and synchronous. source

AuthorAndrey Reshetnikov, AI product and AI engineering