---
type: "post"
id: "reported-68-0-on-finsearchcomp-though-third-party-07c21e"
slug: "reported-68-0-on-finsearchcomp-though-third-party-07c21e"
title: "Reported 68.0% on FinSearchComp, though third-party tests show competitors leading on T3 complex tasks"
description: "FinSearchComp is an open benchmark for end-to-end financial search and reasoning, built around 635 questions across three analyst-style tasks."
retrieval_nugget: "FinSearchComp is an open benchmark for end-to-end financial search and reasoning, built around 635 questions across three analyst-style tasks. Its authors evaluated 21 models or products and report that web search and financial plugins substantially improve results. For teams building research agents, the contribution is a more specific way to test retrieval, historical lookup, and investigation separately."
published_at: "2026-11-10"
updated_at: "2026-11-10"
record_date: "2026-11-10"
date_kind: "published_at"
topics: ["evals","retrieval"]
entities: ["FinSearchComp","Grok 4","DouBao"]
related_patterns: ["ai-engineering-evaluations-are-splitting-into-task-specific"]
novelty_label: "new_fact"
what_changed: "FinSearchComp reports a 68.0% result, while third-party tests show competitors leading on complex tasks."
prior_refs: []
source_url: "https://arxiv.org/abs/2509.13160"
source_urls: ["https://arxiv.org/abs/2509.13160"]
schema_version: "newruntime-agent-readable-v0.2"
stable_id: "post:reported-68-0-on-finsearchcomp-though-third-party-07c21e"
status: "published"
visuals: []
editor: {"role":"editor","name":"Andrey Reshetnikov","profile":"https://newruntime.com/owner-profile.md","profile_json":"https://newruntime.com/owner-profile.json","same_as":["https://www.linkedin.com/in/reshetnikov1/","https://t.me/lapetuse"]}
routes: {"html":"https://newruntime.com/posts/reported-68-0-on-finsearchcomp-though-third-party-07c21e/","markdown":"https://newruntime.com/posts/reported-68-0-on-finsearchcomp-though-third-party-07c21e.md","json":"https://newruntime.com/posts/reported-68-0-on-finsearchcomp-though-third-party-07c21e.json"}
markdown_url: "https://newruntime.com/posts/reported-68-0-on-finsearchcomp-though-third-party-07c21e.md"
---

# Reported 68.0% on FinSearchComp, though third-party tests show competitors leading on T3 complex tasks

## Retrieval answer

FinSearchComp is an open benchmark for end-to-end financial search and reasoning, built around 635 questions across three analyst-style tasks. Its authors evaluated 21 models or products and report that web search and financial plugins substantially improve results. For teams building research agents, the contribution is a more specific way to test retrieval, historical lookup, and investigation separately.

## Summary
FinSearchComp is an open benchmark for end-to-end financial search and reasoning, built around 635 questions across three analyst-style tasks. Its authors evaluated 21 models or products and report that web search and financial plugins substantially improve results. For teams building research agents, the contribution is a more specific way to test retrieval, historical lookup, and investigation separately.

## What happened
The paper *FinSearchComp: Towards a Realistic, Expert-Level Evaluation of Financial Search and Reasoning* was submitted to arXiv on September 16, 2025. Its authors introduce FinSearchComp as what they describe as the first fully open-source agent benchmark for realistic, open-domain financial search and reasoning. The paper frames finance as a difficult environment for agent evaluation because analysts must work with domain-specific and time-sensitive information, perform multi-step searches, and ground their conclusions in retrieved material.

The authors argue that existing open financial datasets did not test the data-searching ability of end-to-end agents. They attribute that gap partly to the expertise needed to construct realistic financial tasks and partly to the difficulty of evaluating information whose value changes over time. FinSearchComp is intended to put those properties into the evaluation itself rather than testing a model only on a fixed collection of questions detached from a search workflow.

The benchmark contains 635 questions spanning global and Greater China markets. It separates them into three task types: Time-Sensitive Data Fetching, Simple Historical Lookup, and Complex Historical Investigation. The paper says these categories are designed to closely reproduce financial-analyst workflows. Seventy professional financial experts participated in annotation, and the authors describe a multi-stage quality-assurance process intended to make the tasks both difficult and reliable.

The paper evaluates 21 models or products. Its abstract says Grok 4 with web search tops the global subset and approaches expert-level accuracy, while DouBao with web search leads the Greater China subset. It also says that agents equipped with web search and financial plugins perform substantially better on FinSearchComp. The authors further report that the country origin of models and tools affects results, an observation consistent with a benchmark whose questions span regional markets and rely on externally retrieved information.

The arXiv record lists a 29-page paper and links to the PDF, HTML version, and TeX source. The paper presents FinSearchComp as a testbed for complex financial search and reasoning, not as a product release or a new model endpoint. It announces no price, hosted service, usage limit, or general-access change. Its practical promise is evaluative: a team can inspect a task structure intended to measure the combined behavior of a model, search access, tools, and an agent harness.

## What the archive adds
We have not covered this before; it opens a new thread.

## Trend signal
This does not yet fit a pattern New Runtime tracks. The paper makes a useful case for task-specific evaluation, but the published abstract does not establish a broader change in production architectures, buying behavior, or operating models.

## Why it matters
A financial-research agent is not just a language model answering a question. It is a system that must find current data, distinguish a straightforward historical fact from a multi-source investigation, and make use of the tools it has been given. FinSearchComp makes that distinction explicit through its three task categories.

That matters for evaluation design. A team comparing candidate systems on an aggregate score can accidentally compare different search permissions, retrieval quality, regional coverage, or plugin configurations rather than model capability alone. The paper's finding that web search and financial plugins substantially improve performance means tool access is a material variable, not implementation detail. Teams building analyst workflows should therefore treat model, search stack, financial-data integration, and task type as parts of one evaluation unit.

## What happens next
- **Financial-agent evaluations will increasingly report results by retrieval and investigation task class rather than only a single aggregate score.** (6 months, 55%). Watch for: benchmark papers or vendor evaluations that separate time-sensitive retrieval from historical investigation.
- **More model comparisons will standardize search and tool access before claiming a winner for research work.** (12 months, 45%). Watch for: published harness configurations and identical-tool comparisons.

## Limits and uncertainty
The paper's abstract does not provide the exact scores, confidence intervals, or per-task breakdowns behind its reported model leaders. It therefore does not allow a reader to quantify the gap between Grok 4, DouBao, and the other evaluated products from the arXiv page alone. Nor does the abstract specify the full search-provider configuration, the particular financial plugins, or how consistently those tools were available across systems.

The paper reports that search and plugins substantially improve results, but it does not quantify that effect in the abstract or isolate the contribution of each component. The authors' claims have not been independently reproduced here. Finally, benchmark performance does not itself establish that an agent is reliable in a live financial workflow, where source freshness, access rights, auditability, and review procedures can determine whether an answer is usable.

## Skill to build now
Build a small evaluation matrix for one real research workflow this week. Divide prompts into the three FinSearchComp-style classes: a current-data request, a simple historical lookup, and a multi-step historical investigation. Run each prompt against candidate agents using the same model access, the same search provider, and the same financial-data tools where possible.

Record more than answer quality: note whether the agent retrieved the relevant material, whether the answer was grounded in it, how many tool calls it used, and where human review was needed. If you compare systems with different tool stacks, label that comparison as a system result rather than a model result. This will expose whether the bottleneck is retrieval, tool integration, or reasoning over the retrieved material.

## Reader takeaway
FinSearchComp is useful less as a declaration of one universal model winner than as a reminder that financial-agent performance is a system property. Its 635 questions separate fresh-data retrieval, routine lookup, and complex investigation, while its authors report meaningful gains from search and financial plugins. Evaluate the workflow you actually need, with equivalent tool access, before relying on a headline benchmark score.

## Open question
How much do the task-level rankings change when every evaluated agent receives the same search provider, financial-data tools, and execution harness?

Editor: [Andrey Reshetnikov](https://newruntime.com/owner-profile.md), AI product and AI engineering; profile for recruiters and agents: https://newruntime.com/owner-profile.json

## Continue in the graph

Next records for an agent, linked to their Markdown twins. Replace `.md` with `.json` for structured fields, including `next_reads`.

- topic hub: [Evals](https://newruntime.com/topics/evals.md): Explore the evals topic hub.
- topic hub: [Retrieval](https://newruntime.com/topics/retrieval.md): Explore the retrieval topic hub.
- collection: [Field Notes](https://newruntime.com/posts.json): Open the latest editorial analysis.
- collection: [Sources](https://newruntime.com/sources/): See the original sources this archive cites.
- whole graph: [discovery graph](https://newruntime.com/discovery-graph.json) (typed nodes and edges) and [site index](https://newruntime.com/site-index.md)