What happened
Firecrawl’s Legal Information Retrieval Benchmark 2026 evaluates search systems on requests for official legal material: statutes, regulations, court opinions, and agency guidance. The page says five systems were tested on the same 581 requests and ranks them by “gold@1,” whether the official document is the first result. It was last updated November 3, 2026.
The five compared systems are Firecrawl Government & Legal Index, Exa, Parallel, Claude Code with built-in web search, and Claude Code without tools. Firecrawl’s index put the official page first on 76.2% of requests. Claude Code using its built-in web search reached 71.4%; Exa reached 59.6%; Parallel reached 54.2%; and Claude Code answering without tools reached 47.8%. Firecrawl says its lead over the other search engines on this measure holds at 95% confidence.
For “gold@10,” whether the official result appeared anywhere in the first ten, Firecrawl scored 96.6%, compared with 90.5% for Parallel and 89.5% for Exa. The page says the lead over each of those engines also holds at 95% confidence. Claude Code with web search scored 74.4% on this measure, while the no-tools setup scored 48.2%. The latter two setups return the one or two addresses the agent settles on, rather than a conventional ranked list of ten results.
The benchmark also measures “bad@10”: whether a page contradicting the official document appears in the top ten. Firecrawl and Exa each scored 9.6%, while Parallel scored 9.3%; Firecrawl describes the three figures as statistically tied. A contradicting page appeared first in at most four of the 581 requests for any engine. The Claude Code setups show 0.2% and 0.0% on this metric, but Firecrawl explicitly says those figures are not comparable because those setups do not return a top-ten list.
The page reports cost per request using each engine’s list price for one search call: Exa at $0.007, Parallel at $0.005, and Firecrawl’s Government & Legal Index free, with no credits used and no API key required. For Claude Code, the reported cost includes model tokens and search fees. Latency was measured with all three engines timed in the same run from one client location; Claude Code timings run from request to answer. The supplied material does not state a universal availability guarantee, service limit, or latency figure for the index.
What the archive adds
Firecrawl’s free Government & Legal Index for ChatGPT and Claude covered the index itself. This is a new fact on that known product story: it adds comparative retrieval results and methodology claims, rather than announcing a new index. The archive also contains a later Field Note about Firecrawl Directory, but that professional-profile product is not evidence about legal retrieval quality.
Trend signal
The benchmark supports the idea that agent systems should be judged on externally observable task outcomes, not simply on a model’s general capability. Here, the useful question is whether a workflow surfaces the authoritative document, not whether it can produce a plausible legal-sounding answer. This is adjacent to Agent economics moves to completed work, although this evaluation measures retrieval accuracy rather than cost per verified completed task.
Connections to other events
The archive’s GDPx benchmark note described an agentic evaluation spanning 206 tasks across 11 high-value domains. Firecrawl’s release is much narrower: it tests a retrieval component on official legal sources rather than an end-to-end agent. Together, the two items distinguish domain task evaluation from measurement of a specific tool inside the agent workflow.
Why it matters
Legal retrieval is a high-consequence use case where a plausible but unofficial or contradictory page can mislead a downstream answer. An agent may summarize, cite, or trigger a review based on what search ranks highly. That makes source identity and rank operational concerns, rather than merely search-quality details. A system that reaches an official page somewhere in a result set may still impose more verification work than one that puts it first.
The benchmark offers teams a starting point for separating “found something relevant” from “surfaced the authoritative source first,” while keeping contradiction risk visible. It also gives a concrete cost claim: Firecrawl lists its index as free with no credits or API key, versus the stated per-call prices for Exa and Parallel. Those are inputs to evaluation, not enough information to establish total workflow cost, since review time, retries, prompts, and model usage may vary.
The practical signal is not simply that Firecrawl wins. Its index leads on the benchmark’s stated ranking measures, while the contradiction metric is statistically tied with Exa and Parallel. Teams building a legal-research feature should turn that distinction into their own acceptance tests: can the system find the controlling document, preserve the source URL, and make uncertainty visible before a model drafts an answer? A vendor leaderboard can identify a candidate tool; it cannot remove the need for that verification layer.
Limits and uncertainty
This is a vendor-published benchmark by the provider whose index ranks first. The supplied material does not include enough detail to independently reproduce the dataset, query construction, relevance judgments, source selection, or confidence-interval calculations. It reports 581 requests, but does not establish that the sample represents every jurisdiction, legal task, language, or changing source collection.
The comparison also mixes full search engines returning a top ten with Claude Code reference setups returning one or two addresses. Firecrawl correctly notes that this makes bad@10 comparisons with those setups invalid. List prices per search call are not necessarily the cost of a complete research workflow, and timing from one client location does not establish performance elsewhere. Finally, the scores do not show that any system is suitable for legal advice, that retrieved material is current or complete, or that an agent will interpret the document correctly.
Skill to build now
Build a small evaluation set from the legal requests your product actually receives. For each request, record the authoritative target, whether it appeared first, whether it appeared in the returned set, and whether a contradictory source was also surfaced. Run the same set through the candidate retrieval tools and retain the URLs alongside generated answers. This makes the benchmark’s gold@1, gold@10, and bad@10 framing testable in a workflow your team owns.
Reader takeaway
Firecrawl’s index performed strongly at finding and ranking official legal sources in its own 581-query evaluation, and it is listed as free to call. But it was not better on the reported contradiction rate than Exa or Parallel, and the publisher ran the test. Treat the figures as a useful evaluation template and test your own legal queries before relying on any retrieval layer in production.