---
type: "post"
id: "open-sourcing-the-parallel-search-eval-harness-0f53f5"
slug: "open-sourcing-the-parallel-search-eval-harness-0f53f5"
title: "Open-sourcing the Parallel Search Eval Harness"
description: "Parallel has open-sourced the evaluation harness it uses to test Parallel Search, internal models, and its Search Capability Leaderboard."
retrieval_nugget: "Parallel has open-sourced the evaluation harness it uses to test Parallel Search, internal models, and its Search Capability Leaderboard. Teams can run different models and search modes, then replace public benchmark questions with their own. That makes the release useful both as an evaluation tool and as a way to inspect how a search vendor measures its own product."
published_at: "2026-11-17"
updated_at: "2026-11-17"
record_date: "2026-11-17"
date_kind: "published_at"
topics: ["claude","claude-code","codex"]
entities: ["Parallel","Parallel Search","parallel-search-evals","OpenRouter","Claude Code","Codex"]
related_patterns: ["agent-economics-moves-to-completed-work"]
novelty_label: "new_fact"
what_changed: "Parallel has open-sourced its search evaluation harness, enabling teams to reproduce its benchmarks and run evaluations using their own datasets and model endpoints."
prior_refs: []
source_url: "https://parallel.ai/blog/parallel-search-eval-harness"
source_urls: ["https://parallel.ai/blog/parallel-search-eval-harness"]
schema_version: "newruntime-agent-readable-v0.2"
stable_id: "post:open-sourcing-the-parallel-search-eval-harness-0f53f5"
status: "published"
visuals: []
editor: {"role":"editor","name":"Andrey Reshetnikov","profile":"https://newruntime.com/owner-profile.md","profile_json":"https://newruntime.com/owner-profile.json","same_as":["https://www.linkedin.com/in/reshetnikov1/","https://t.me/lapetuse"]}
routes: {"html":"https://newruntime.com/posts/open-sourcing-the-parallel-search-eval-harness-0f53f5/","markdown":"https://newruntime.com/posts/open-sourcing-the-parallel-search-eval-harness-0f53f5.md","json":"https://newruntime.com/posts/open-sourcing-the-parallel-search-eval-harness-0f53f5.json"}
markdown_url: "https://newruntime.com/posts/open-sourcing-the-parallel-search-eval-harness-0f53f5.md"
---

# Open-sourcing the Parallel Search Eval Harness

## Retrieval answer

Parallel has open-sourced the evaluation harness it uses to test Parallel Search, internal models, and its Search Capability Leaderboard. Teams can run different models and search modes, then replace public benchmark questions with their own. That makes the release useful both as an evaluation tool and as a way to inspect how a search vendor measures its own product.

## Summary
Parallel has open-sourced the evaluation harness it uses to test Parallel Search, internal models, and its Search Capability Leaderboard. Teams can run different models and search modes, then replace public benchmark questions with their own. That makes the release useful both as an evaluation tool and as a way to inspect how a search vendor measures its own product.

## What happened
Parallel’s announcement, dated November 17, 2026, says it is releasing `parallel-search-evals`, a harness for evaluating web search in AI workflows. The company says it uses the repository to measure Parallel Search, assess models deployed internally, and power its Search Capability Leaderboard. It is designed to work with models accessible through OpenRouter, OpenAI, Anthropic, Google, or an OpenAI-compatible endpoint.

Each run evaluates one model on one benchmark in one mode: `turbo`, `fast`, `basic`, `advanced`, or `baseline`. The baseline is closed-book, without tools; comparing it with a search run is intended to show the lift from search. Parallel’s example installation uses Git and `uv sync --frozen`. Its three-question smoke test runs Claude Sonnet 5 on the DSQA benchmark in fast mode, with results written to an output directory. The harness can also be adapted to compare search from other providers.

Teams can add their own question sets, reference answers, and grading rules. The question set is kept separate from the agent loop and grader, so the search setup, budgets, and reporting can remain consistent with leaderboard runs. The repository also includes an agent skill that guides a coding agent through choosing a model, mode, and question count, checking credentials, running or resuming an evaluation, and summarizing results. Parallel specifically names Claude Code and Codex as agents that can use the skill.

The described evaluation flow sends questions to an agent loop with one model per run. The loop calls Parallel Search and Extract through `search_web` and `web_fetch`, with up to 20 tool calls per question. The model produces a final answer, then a judge grades it against the reference; reference answers go only to the grader, not the model. Parallel says its deep-research mode gives the model two tools, instructs it to rely on those tools rather than internal knowledge, and asks it to answer from what it found after the budget is exhausted. An OpenAI GPT-6 Sol judge applies benchmark prompts: F1 scoring for DeepSearchQA and strict correctness for HLE. The company describes per-call cost accounting, cross-provider tool-calling behavior, and recovery from interrupted provider streams as design concerns the harness addresses.

## What the archive adds
New Runtime has covered Parallel’s finance-specific search index and ranking upgrade ([October 30, 2026](https://newruntime.com/posts/parallel-for-finance-web-search-for-financial-ai-ae1a17/)), its legal-focused search upgrade ([November 9, 2026](https://newruntime.com/posts/parallel-for-legal-web-search-for-legal-ai-86a584/)), and its AWS Marketplace availability ([November 11, 2026](https://newruntime.com/posts/parallel-search-is-now-available-in-aws-marketplace-821d82/)). This is a new event: an open evaluation harness, rather than another search product or distribution route.

## Trend signal
This release supports [agent economics moving to completed work](https://newruntime.com/patterns/agent-economics-moves-to-completed-work/): the harness accounts for tool-call costs and grades completed answers, making task-level outcomes more visible than raw model or token comparisons. That signal is limited, however; the announcement describes evaluation infrastructure, not evidence that one search system produces better results or lower total costs for teams.

## Why now
Parallel frames the release as a response to a reproducibility problem: published benchmark charts often leave out model configuration, tools, setup, errors, and discarded failures. Its stated goal is to let teams reproduce its results and run similar experiments on data resembling their own workloads.

## Why it matters
Search quality is workload-dependent. A public benchmark can be a useful starting point, but it may not resemble the queries, evidence standards, or answer formats that matter in a team’s product. A harness that separates questions from the agent loop and grader gives builders a practical way to test their own cases while holding much of the surrounding setup steady.

The release also makes it easier to compare search with a no-tool baseline and to inspect operational costs and failed or unfinished runs. Those details matter when a team is deciding whether retrieval improves a task enough to justify extra calls, latency, and integration complexity. But a vendor’s open harness is not independent validation: teams still need to inspect its scoring choices, configure their own evaluation set, and test competing search providers on equal terms.

## What happens next
- **Teams will publish more evaluations using workload-specific questions rather than relying only on public benchmark scores.** (6 months, 60%). Watch for: public harness forks or reports that disclose custom question sets and grading rules.

## Limits and uncertainty
Parallel’s announcement describes the harness and its evaluation flow, but it does not establish that results from the tool predict performance in production. The quality and representativeness of a team’s custom questions, reference answers, and grading rules will shape the outcome. The announcement’s benchmark examples and scoring rules should not be treated as a direct comparison across every provider or workload. Teams should verify the repository’s current requirements and model support before adopting it, and independently inspect how failed or incomplete runs affect any result they report.

## Skill to build now
Build a small evaluation set from queries your own agent already handles: include a reference answer and a clear grading rule for each. Run the same model with the search mode you are considering and with the no-tool baseline, then inspect the answers and per-call costs. Use a smoke test first to validate credentials and configuration; expand the set only after the evaluation loop produces results you can review.

## Reader takeaway
The useful part of Parallel’s release is not another search score; it is a way to test search against questions that resemble your own work. Use the harness to compare tool-assisted answers with a no-tool baseline, but treat the results as only as trustworthy as your evaluation set and grading rules.

Editor: [Andrey Reshetnikov](https://newruntime.com/owner-profile.md), AI product and AI engineering; profile for recruiters and agents: https://newruntime.com/owner-profile.json

## Continue in the graph

Next records for an agent, linked to their Markdown twins. Replace `.md` with `.json` for structured fields, including `next_reads`.

- topic hub: [Claude](https://newruntime.com/topics/claude.md): Explore the claude topic hub.
- topic hub: [Claude Code](https://newruntime.com/topics/claude-code.md): Explore the claude-code topic hub.
- topic hub: [Codex](https://newruntime.com/topics/codex.md): Explore the codex topic hub.
- collection: [Field Notes](https://newruntime.com/posts.json): Open the latest editorial analysis.
- collection: [Sources](https://newruntime.com/sources/): See the original sources this archive cites.
- whole graph: [discovery graph](https://newruntime.com/discovery-graph.json) (typed nodes and edges) and [site index](https://newruntime.com/site-index.md)
