New Runtime data
What we tested
Tools New Runtime ran on the same task, with the method, the checked results, what each run cost and our impressions. No sponsorship.
New Runtime runs the same task through several tools, checks every answer by hand and publishes the numbers. No sponsorship: no vendor paid for, saw or approved these results. Free tiers unless an entry states otherwise; each entry names its cost basis. Last updated October 28, 2026.
/tested.jsonThe same entries as JSONTested October 28, 2026 · by New Runtime · link
Five search APIs as a source of candidate websites for the same business records
The same records as the first entry (a registered business name with a city and state), but a narrower question: when a plain web-search API is given one fixed query per record, how often does the business's verified official website appear in the first ten results, and how high? This measures each API as a source of candidates for a verifier, not as a finished answer.
Method
- Reference set n = 55 Records from the first entry's batch whose official website one reviewer had verified by hand in that run. Every tool was also queried on the remaining records of the batch (100 queries per tool in total) for latency and cost, but only the 55 with a verified site are scored.
- Inputs
- One query per record built from one template: the registered name, the city and the state, with no quotes or operators. Ten results requested from every API.
- Judging
- Strict hit: a result whose host (minus www.) is the verified domain or a subdomain of it, counted at rank 1, within the top 3 and within the top 10 of the organic list. A second, looser count (in the misses column) credits a site of the same business under another domain, such as the parent company's corporate site, a product site or a domain alias, decided by hand after the run by the same reviewer. For SerpAPI the knowledge panel and local pack fields were also read, because its organic list often omitted a site that those fields carried.
- Runs
- One call per record and tool, all five tools run in parallel within the same hour. Same query string, byte for byte, across tools.
- Cost basis
- Free allowances on every tool: Serper's 2,500 free queries, Jina's 10,000,000 free tokens, You.com's $10 sign-up credits, Linkup's 4,000 free queries and SerpAPI's 250 free searches a month. Per-query prices in the table are the vendors' published list prices on the day of the test, not billed amounts.
Results
Reference set (55)| Tool | Official site in top 10 | At rank 1 / in top 3 | Misses and errors | Cost | Time |
|---|
| Serper | 46/55 | 35/55 at rank 1; 40/55 in top 3 | 9 misses, 6 of which returned another site of the same business (52/55 loosely); 0 errors | 100 of 2,500 free queries; then $1.00 per 1,000 | median 1.7 s, p90 2.6 s |
|---|
| Jina Search | 47/55 | 38/55 at rank 1; 40/55 in top 3 | 8 misses, 6 of which returned another site of the same business (53/55 loosely); 0 errors | 1,000,000 of 10,000,000 free tokens (a flat 10,000 tokens per call); then $0.05 per 1,000,000 tokens, about $0.50 per 1,000 calls | median 1.4 s, p90 2.2 s |
|---|
| You.com Web Search API | 36/55 | 14/55 at rank 1; 24/55 in top 3 | 19 misses, 7 of which returned another site of the same business (43/55 loosely); 0 errors | about $0.50 of the $10 sign-up credits; $5 per 1,000 calls | median 1.2 s, p90 1.6 s |
|---|
| Linkup Search | 28/55 | 6/55 at rank 1; 12/55 in top 3 | 27 misses, 3 of which returned another site of the same business (31/55 loosely); 0 errors | 100 of 4,000 free queries; then $5 to $6 per 1,000 standard searches | median 1.9 s, p90 2.8 s |
|---|
| SerpAPI | 29/55 in the organic list; 40/55 counting the knowledge panel and local pack | 19/55 at rank 1; 25/55 in top 3 | 26 organic misses; 11 of them had the site in the knowledge panel or local pack and 4 more another site of the same business (44/55 loosely); 0 errors, one call returned 9 results | 100 of 250 free searches a month; the first paid plan is $25 per 1,000 | median 1.2 s, p90 4.5 s |
|---|
At list prices, 1,000 queries cost about $0.50 on Jina, $1 on Serper, $5 on You.com, $5 to $6 on Linkup and $25 on SerpAPI. Every tool's free allowance covered this whole test.
Impressions
Serper
- Cheapest paid price per 1,000 queries here and the largest free allowance.
- Returns the credits used with every response.
- Rank 1 was a directory, map or LinkedIn page on 20 of 55 records, so a verifier is still needed.
- For brand and subsidiary records it returns the corporate parent's site, which may not be the site you want.
Jina Search
- Best strict top-10 and rank-1 counts, and the cleanest JSON to consume.
- The same key covers reading pages, so one vendor can do search and fetch.
- Charges a flat 10,000 tokens per search call even with page content suppressed, so the free allowance is 1,000 searches, not millions of tokens of reading.
- Its lists were so close to Serper's (same first result on 78 of 100 queries) that it is not a second opinion.
You.com Web Search API
- Fastest median here and never errored.
- Snippets are longer than a Google SERP's, which helps an agent read without a second fetch.
- The official site led the list on only 14 of 55 records; map and directory sites usually came first.
- No usage or cost field in the response; credits are visible only in the dashboard.
Linkup Search
- Every result carries a text extract, which the Google-based tools do not give at this price.
- Largest free allowance after Serper.
- The weakest candidate source in this test: the official site was in the top ten on about half of the records and first on 6 of 55.
- Leans towards people-data and company-directory sites for a business-name query.
SerpAPI
- Exposes the knowledge panel and local pack as structured fields; reading them lifted coverage from 29 to 40 of 55.
- Rich result types beyond organic links, useful when the task is the SERP itself.
- Its organic list alone was the weakest of the Google-based tools and drifted to look-alike names on short business names.
- The smallest free allowance (250 searches a month) and the highest list price here, 25 times Serper's.
Verdict
As a source of candidate sites for your own verifier, a plain Google results API is the right building block: Serper and Jina put the verified site in the top ten on about 85% of records and first on roughly two thirds, at $0.50 to $1 per 1,000 queries. SerpAPI reaches similar coverage only if you also read its knowledge-panel and local-pack fields, at 25 times the price. You.com and Linkup are built for agents that read, not for entity lookup: they surfaced the verified site on 65% and 51% of records and rarely at rank one. None of the five replaces verification: on every tool a directory, map or data-broker page outranked the official site more often than not, and a shared miss across tools does not mean the business has no website.
Caveats
- The reference comes from the first entry's hand check, and the same reviewer decided the same-business alternates after seeing the lists; this is a careful check, not a blind benchmark.
- One query template and one run per record; another template (quoted name, no city, a site: operator) could reorder the tools.
- Latency is the full round trip from one client on one afternoon; the vendors' own reported times were lower.
- Google-based tools can return different layouts for the same query, so Serper, Jina and SerpAPI are not measuring the same page.
- Prices and free allowances are those published on the day of the test and change often.
No sponsorship; every run stayed inside a free allowance or sign-up credits. No vendor saw or approved these results.
Tested October 28, 2026 · by New Runtime · link
Open models on NVIDIA's free API, run on the newsroom's compact triage step and scored against the production operator
The newsroom's compact triage step: given a batch of up to twenty stored material views (title, canonical URL, excerpt), return one JSON item per material with a decision (post, skip, watch or deep research), a content kind (news, release, research, profile, job, landing page, advertisement, other), a confidence, a reason code and a flag for deep analysis. In production this step runs on the newsroom's operator model (OpenAI GPT-5.6 through Codex), and its stored decisions are the reference here. The question is how far open models hosted for free get on exactly the same batches, under exactly the same prompt and schema contract.
Method
- Production batches n = 314 Twenty intake batches from the last three weeks of production, 314 materials, replayed with the operator's stored decisions as the reference: post 174, skip 85, watch 36, deep research 19. Items from a batch that a model failed twice are not scored, so the scored count differs by model.
- Inputs
- The production classification prompt, byte for byte, with the same stored material views the operator saw. Temperature 0, 300 output tokens per item, one repair attempt after a schema failure, as in production. Reasoning was switched off where the model exposes a switch, because with it on the smaller models spent the whole output budget thinking and returned nothing.
- Judging
- Per material: the model's decision equals the operator's (four-way), and the model's post / not-post call equals the operator's (two-way). Per batch: accepted on the first try, accepted after one repair, or rejected by the same schema and cardinality checks production applies. Twenty-five materials on which most models disagreed with the operator were then re-read by one reviewer to decide who was right.
- Runs
- One run per batch and model on one afternoon, three batches in flight at a time to stay under the trial's rate limit. Three further models in the account's callable list did not answer that day and are listed below as such.
- Cost basis
- NVIDIA's build.nvidia.com trial API: free to prototype, 40 requests a minute, no per-token billing and no production use. Every call in this entry was made on that trial, so the cost of the whole run was zero.
Results
Production batches (314)| Tool | Same decision as the operator | Same post / not-post call | Batches rejected and kind agreement | Cost | Time |
|---|
| NVIDIA Nemotron 3 Ultra (550B, 55B active) | 221/314 (70%) | 249/314 (79%); 33 extra posts, 32 missed posts | 0 of 20 batches rejected (19 accepted first try, 1 after repair); content kind agreed on 221/314 (70%) | $0 on the trial; 140,000 input and 18,500 output tokens | median 20 s per batch, p90 35 s |
|---|
| MiniMax M3.1 Flash | 180/265 (68%) | 204/265 (77%); 40 extra posts, 21 missed posts | 3 of 20 batches rejected (six empty responses with status 200, two invalid JSON, one miscounted batch); content kind agreed on 195/265 (74%) | $0 on the trial; 147,000 input and 111,000 output tokens, most of them reasoning | median 202 s per batch, p90 268 s |
|---|
| NVIDIA Nemotron 3.5 Bolt (4B) | 129/224 (58%) | 160/224 (71%); 50 extra posts, 14 missed posts | 6 of 20 batches rejected, 14 of 20 first attempts dropped or added an item; content kind agreed on 91/224 (41%) | $0 on the trial; 195,000 input and 24,000 output tokens | median 18 s per batch, p90 23 s |
|---|
| NVIDIA Nemotron 3.5 Lightning (30B, 3B active) | 67/143 (47%) | 79/143 (55%); 14 extra posts, 50 missed posts | 9 of 20 batches rejected, almost all for dropping or adding an item; content kind agreed on 61/143 (43%) | $0 on the trial; 221,000 input and 42,000 output tokens | median 15 s per batch, p90 21 s |
|---|
| Z.ai GLM 5.3 Flash | no answer | no answer | 3 batches attempted, every call timed out at 300 s, including a one-line probe; run stopped | $0 | none within 300 s |
|---|
| StepFun Step 5 preview | no answer | no answer | 3 batches attempted, every call timed out at 300 s; a one-line probe answered with empty content | $0 | none within 300 s |
|---|
| OpenAI gpt-oss-20b | not run | not run | a one-line probe timed out twice at 90 s on the test day, so no batch was sent | $0 | none within 90 s |
|---|
On the trial the cost is zero and the ceiling is 40 requests a minute: the twenty production batches took about seven minutes of wall time for Nemotron Ultra. The trial excludes production use; the same open weights are sold as downloadable containers for your own GPUs and are hosted elsewhere at per-token prices.
Impressions
NVIDIA Nemotron 3 Ultra (550B, 55B active)
- The only model here that held the batch contract: every batch came back with the right number of items and valid JSON.
- Errors are symmetric, so it is neither a spammer nor a censor; its post / not-post call matched the operator on four of five materials.
- Twenty seconds for a batch of twenty, with reasoning on, is close to the operator's own speed.
- It promotes vendor marketing, pricing pages and third-party explainers to post where the operator skips them.
- It almost never chooses deep research: one of nineteen cases, the rest became post or skip.
MiniMax M3.1 Flash
- Accuracy within two points of Nemotron Ultra on both measures and the best content-kind agreement of the four.
- Three to five minutes per batch and six empty responses with a success status: the slowest and least reliable of the models that answered.
- Reasoning tokens outnumbered answer tokens five to one; there was no switch to turn them off.
NVIDIA Nemotron 3.5 Bolt (4B)
- Fastest answers and the lowest miss rate on real posts: it flagged 14 of the operator's posts as not post.
- It cannot count: fourteen first attempts returned one item too few or too many, which the schema check rejects.
- Fifty extra posts, mostly materials the operator skipped as marketing or duplicates.
NVIDIA Nemotron 3.5 Lightning (30B, 3B active)
- Fast and cheap on tokens.
- The weakest of the four: under half the batches accepted and it skipped 50 of the operator's posts.
- Same counting problem as Bolt, plus one invalid reason code and one item with a missing key.
Z.ai GLM 5.3 Flash
- Listed as callable for the account and answered on an earlier probe.
- Did not answer anything on the test day, which cost forty minutes of retries per batch before the run was stopped.
StepFun Step 5 preview
- Answered an earlier short probe in under twenty seconds.
- Did not answer any production-size batch within 300 s.
OpenAI gpt-oss-20b
- Exposes the reasoning-effort switch that production would need.
- Unreachable on the test day, so nothing could be measured.
Verdict
On the newsroom's own triage prompt, Nemotron 3 Ultra served from NVIDIA's free trial reproduced the production operator's decision on 70% of materials and its post / not-post call on 79%, with every batch valid and about twenty seconds per batch. That is good enough to prototype a cheaper first pass, not to replace the operator: where the reviewer re-read the disagreements, the operator was right on 16 of 25, the open model on 2, and the rest were judgment calls. The open models' systematic errors are promoting vendor blogs, pricing pages and third-party explainers to post, and posting research and benchmark items the operator routes to deep research. The two Nemotron 3.5 models fail before judgment matters: they return a batch of twenty with nineteen or twenty-one items, which the contract rejects. MiniMax matches Ultra's accuracy but takes minutes per batch. Three of the seven callable models did not answer at all that day, which is the real shape of a free trial endpoint.
Caveats
- The reference is the production operator's stored output, not ground truth; the hand re-read covered only the 25 materials where most models disagreed, and the same reviewer wrote the production prompt.
- The prompt was tuned for the operator over months; an open model with its own prompt and examples could score higher.
- Reasoning was disabled on the Nemotron 3.5 models; with it on and a larger output budget their accuracy might differ.
- One run on one afternoon: three models were unreachable that day and may answer on another, and the trial's model list changes without notice.
- The trial terms page could not be retrieved on the test day; these are agreement rates with our own operator on our own task, not a model benchmark, and the numbers say nothing about the models on other tasks.
No sponsorship; NVIDIA did not see or approve these results. The models are third-party open weights that NVIDIA hosts; this entry measures them as served on the trial endpoint, not the weights themselves.
Tested October 20, 2026 · by New Runtime · link
Finding a company's official website from its name: six research agents compared
Given a business record (registered name, optionally a street address and a few registry fields), return the official website, the type of link between the site and the business (the business itself, one of its brands, or its parent company) and evidence pages. A site counts only when the evidence ties it to that exact business; a manufacturer of the same product or a similarly named firm does not count.
Method
- Diagnostic set n = 10 Chosen on purpose to include ambiguous cases (brands, parents, renamed firms).
- Hard cases n = 3 The three records no mode solved on the diagnostic set, rerun with the deepest modes.
- Residual set n = 45 Records the cheapest mode left without an accepted site in a larger batch.
- Input-field test n = 10 The diagnostic set three times on the cheapest mode: name only, name plus address, all fields.
- Inputs
- Business records with a name and, depending on the run, an address and registry fields.
- Judging
- One reviewer checked each returned site against its evidence pages and our own extra fetches, and accepted only links proven to the exact business. Not a blind gold standard: the reviewer saw the answers.
- Runs
- One run per record and mode. The instruction and the JSON answer schema were identical, byte for byte, across tools.
- Cost basis
- Paid API usage for Parallel and Exa (Parallel costs are estimates from its published per-task prices, Exa costs are the amounts its API returned); Firecrawl Agent ran inside its free daily agent runs; the Codex check ran on a ChatGPT subscription and its cost was not measured.
Results
Diagnostic set (10)| Tool | Verified sites | Link type right | Misses and errors | Cost | Time |
|---|
| Parallel Task API, Base | 7/10 | 7/7 | 3 not found | about $0.10 | median 82 s per record |
|---|
| Parallel Task API, Core | 7/10 | 5/7 | 3 not found; once left an unrelated domain filled in while saying not found | about $0.25 | median 105 s per record |
|---|
| Parallel Task API, Pro | 7/10 | 7/7 | 3 not found | about $1.00 | not compared |
|---|
| Exa Agent, Low | 7/10 | 5/7 | 3 not found | $0.25 | median 25 s per record |
|---|
| Exa Agent, Medium | 7/10 | 5/7 | 3 not found | $1.00 | not compared |
|---|
| Codex built-in web search | 7/10 | not scored | 3 not found | not measured (subscription) | not measured |
|---|
Hard cases (3)| Tool | Verified sites | Link type right | Misses and errors | Cost | Time |
|---|
| Parallel Task API, Ultra8x | 0/3 | n/a | 3 not found, no wrong site | about $7.20 | not compared |
|---|
| Firecrawl Agent | 0/3 | n/a | 1 'resolved' we rejected (a product maker, not the business) and 2 'ambiguous' | 0 credits charged (free daily agent runs); 103 credits of work in its execution traces | not compared |
|---|
Residual set (45)| Tool | Verified sites | Link type right | Misses and errors | Cost | Time |
|---|
| Parallel Task API, Core | 3 new of 45 | n/a | 15 not found, 3 ambiguous | about $1.10 | not compared |
|---|
| Parallel Task API, Pro | no new site beyond Core | n/a | 21 not found | about $4.50 | not compared |
|---|
| Exa Agent, Medium | 1 new of 45 that no other mode found | n/a | 15 not found, 1 ambiguous | $4.50 | not compared |
|---|
| Parallel Search API | 0 new | n/a | mostly business directories and aggregators | not returned | not compared |
|---|
Input-field test (10)| Tool | Verified sites | Link type right | Misses and errors | Cost | Time |
|---|
| Parallel Task API, Base | 7/10 with name only, 7/10 with address, 7/10 with all fields | 6/7, 7/7, 7/7 | one unsupported 'resolved' in the name-plus-address variant | about $0.10 per variant | not compared |
|---|
At the per-task prices used here, a first pass over 1,000 records would cost about $10 on Parallel Base, $25 on Parallel Core or Exa Low, $100 on Parallel Pro or Exa Medium and $2,400 on Parallel Ultra8x, before verification and retries.
Impressions
Parallel Task API, Base
- Cheapest mode with the same coverage as every deeper mode on the diagnostic set.
- Named the type of link correctly on all seven finds.
- The raw output lacked our record ids, and a failed record came back as empty strings instead of null.
Parallel Task API, Core
- Best value as a second pass: 3 of the 4 new sites on the residual set for about $1.10.
- Once filled in an unrelated domain while reporting not found, so the status alone cannot be trusted.
Parallel Task API, Pro
- Precise link types, no stray candidates.
- Four times the cost of Core for no extra coverage in any set.
Parallel Task API, Ultra8x
- Careful refusals: it separated look-alike firms instead of guessing.
- About $2.40 per record and no new site on the hard cases.
Exa Agent, Low
- Fastest by far (median 25 s) and returns its own cost with each answer.
- Less precise about whether a site is the business, a brand or a parent.
Exa Agent, Medium
- Stronger evidence pages on several records; the only mode to find one site on the residual set.
- Same price as Parallel Pro; still mislabelled some link types.
Firecrawl Agent
- Read pages that other tools' fetches came back empty on.
- Ran inside the free daily agent runs.
- Returned one confident 'resolved' that did not hold up.
- Its credit reporting needs care: 0 credits charged against 103 credits of work in the traces.
Codex built-in web search
- Matched the paid agents on the diagnostic set without any external research API.
- Not a blind test: the session already knew earlier answers, so treat it as a reproduction, not a score.
Parallel Search API
- Confirmed already known domains.
- Returned mostly directories and aggregators; added nothing as a source of candidates.
Verdict
For bulk website finding, start with the cheapest agent mode plus your own verifier, rerun only the misses on a mid mode, and keep a second vendor's mid mode for what is still missing. The deepest modes (Parallel Pro, Exa Medium, Parallel Ultra8x) cost 10 to 24 times more per record than the cheapest and added one site in total that a cheaper pass had missed. Never accept a 'resolved' status without checking that the evidence names the exact business, and do not read a shared failure across tools as proof that no website exists.
Caveats
- Small sets and one run per record: differences of one or two records are noise.
- One instruction and one kind of record; other record types may rank the tools differently.
- The reviewer saw the answers, so this is a careful check, not a blind benchmark.
- Parallel costs are estimates from published prices, not billed amounts.
No sponsorship; free tiers unless stated. This entry used paid API usage for Parallel and Exa.