What happened
Huntsman is a public Apache-2.0 project from TulipFarm for turning web pages into clean Markdown for LLMs and search systems. It is designed to be self-hosted and is distributed as a Go binary and container image. Its central compatibility claim is a Firecrawl-shaped /v1/scrape API: a client can point most existing scrape requests at a different base URL while keeping the endpoint shape. The project publishes images for Linux amd64 and arm64, and its production guidance is to pin a release version such as 0.1.0 or an image digest rather than use latest.
The implementation is intentionally lean. Huntsman uses Lightpanda to render JavaScript, rather than bundling Playwright or Chrome, and it says it requires neither Redis nor a browser stack. It attempts to keep primary page content while removing navigation, footers, ads and cookie banners. The README says this uses Firecrawl's onlyMainContent rules and EasyList filters. PDFs are fetched directly; when a page blocks headless browsers, Huntsman retries it as plain HTML. The service waits for a single-page application to become quiet before returning the result.
The project exposes more than one-page scraping. POST /v1/batch queues between one and 1,000 URLs, while POST /v1/crawl initiates a site crawl with a URL limit and maximum depth. A SQLite-backed queue manages batches and crawls, with a per-host concurrency limit and a result cache. Other documented controls include Markdown-only output, waits by milliseconds or selector, timeout, cache age, and an onlyMainContent flag. Unknown request fields are rejected rather than silently ignored. The response for a scrape includes Markdown, links and metadata such as title, source URL, HTTP status, fetch time and engine.
Its security defaults are relevant for a service that agents may call with arbitrary URLs. Huntsman says it blocks private, loopback and link-local addresses at dial time, observes robots.txt, and runs as a non-root user. Operators can set one bearer token or a named collection of tokens. Requests are logged by key name rather than token value, which the README positions as a way to distinguish clients and revoke one client key without replacing the others.
The project makes performance claims, but they are project-authored benchmark results rather than independent measurements. On an AWS Graviton 8-vCPU, 16-GB test machine, its table reports 129 concurrent pages, 11.54 pages per second and a 90.8% success rate for Huntsman; the corresponding Firecrawl figure is 43 pages, 2.90 pages per second and 92.3%. The README summarizes that as four times as many pages per second on the same 8-vCPU machine. On a 2-vCPU, 4-GB machine, the table reports 45 concurrent pages at 4.58 pages per second and a 90.9% success rate. Those figures are useful hypotheses for an evaluation, not proof of production performance on a team's own target sites.
The compatibility boundary is also explicit. Huntsman returns Markdown and links only. It does not provide screenshots, raw HTML, JSON extraction, browser actions or search. The README suggests pairing it with a search engine such as SearXNG when search is required. That makes it a focused substitution for web reading, rather than a full replacement for every Firecrawl workflow.
What the archive adds
This is a new event on a known web-data story. Introducing Universal Scrape examined Firecrawl expanding an endpoint with selected external data-provider calls; Huntsman moves in the opposite direction by offering a self-hosted, deliberately narrower compatible surface. kwant launches as an open-source, self-hosted single binary also documented the appeal of compact, self-operated agent infrastructure, though it addressed multi-agent coordination rather than web retrieval.
Trend signal
Huntsman supports Agent-ready software exposes capabilities. Its useful boundary is not a general browsing product but observable endpoints for scrape, batch, crawl, job status and health, paired with tokens and URL restrictions. Its reduced output contract also illustrates that a composable capability can be valuable precisely because it omits adjacent features that would make operational control harder.
Connections to other events
The earlier Universal Scrape note described Firecrawl making data-provider integrations part of its product proposition. Huntsman is not evidence that the broader platform approach is failing; it is a different choice for operators whose immediate need is a deployable, compatible Markdown ingestion service rather than an expanding managed retrieval surface.
Why it matters
Web retrieval is often an unglamorous but consequential dependency in an agent stack. A team may need to convert internal or public URLs into model-ready context, schedule crawls, isolate credentials and control where data flows. Huntsman provides a narrowly specified component for that job and gives teams an API migration path if their client already speaks the documented Firecrawl scrape shape.
The trade-off is important. Replacing a richer hosted service with a local scraper does not eliminate operational work: someone still owns deployment, upgrades, capacity, access keys, outbound-network policy and evaluation against real pages. Nor does API compatibility create feature compatibility. Teams using screenshots, structured JSON extraction, raw HTML, search or browser actions would need another component or should retain a fuller product. The practical gain is choice over the retrieval layer, not a universal scraping solution.
What happens next
- Teams evaluating self-hosted web ingestion will test Huntsman against their own target domains and concurrency profile. (3 months, 70%). Watch for: reproducible reports beyond the repository benchmark table.
- The project will face requests for richer outputs or integrations as users test Firecrawl-client compatibility. (6 months, 65%). Watch for: issues or releases adding features outside Markdown and links.
Limits and uncertainty
The supplied source is the project's repository and README, so its architecture, benchmark setup and performance results are maintainer claims. It does not establish independent throughput, extraction quality, reliability on difficult sites, security-review results, support commitments or the maturity of the project under production load. The stated benchmark samples 1,000 URLs from the project's own dataset and uses a defined passing threshold; it cannot predict outcomes for paywalled pages, anti-bot protections, unusual JavaScript applications or a particular network environment. Firecrawl compatibility is presented around the documented endpoint shapes, while the feature boundary remains deliberately narrower.
Skill to build now
Make the web-reading boundary in one agent workflow explicit. List which consumers need only URL-to-Markdown and links, then run the same representative URLs through the current provider and a locally deployed Huntsman instance. Compare success rate, latency, content cleanliness, handling of JavaScript pages and the missing features your workflow actually uses. Put the service behind a bearer token, use separate named keys for development and production, and test that private-address blocking behaves as expected before allowing agent-originated URLs.
Reader takeaway
Huntsman is a focused self-hosted alternative for the Markdown-reading portion of a web-agent pipeline, not a complete Firecrawl clone. Its value is a small deployable API, batch and crawl support, and explicit operational controls. Evaluate the project's promising but unverified performance claims against your own sites—and do not migrate workflows that rely on the capabilities it intentionally leaves out.