---
type: "post"
id: "sakana-ai-released-jfinbench-321-questions-from-six-dce296"
slug: "sakana-ai-released-jfinbench-321-questions-from-six-dce296"
title: "Sakana AI released JFinBench, 321 questions from six 2026 Japanese financial certification exams."
description: "Sakana AI has released JFinBench, a Japanese financial benchmark built from 321 questions across six 2026 certification exams."
retrieval_nugget: "Sakana AI has released JFinBench, a Japanese financial benchmark built from 321 questions across six 2026 certification exams. Its results show how domain-specific knowledge can separate models that look closer on general benchmarks, while its agent-assisted construction pipeline offers a way to refresh evaluations with human review."
published_at: "2026-11-17"
updated_at: "2026-11-17"
record_date: "2026-11-17"
date_kind: "published_at"
topics: ["claude","gemini"]
entities: ["Sakana AI","JFinBench","Claude Fable 5","GPT-6 Astra","Gemini 3.8 Flash","Kimi K3.1 Pro","e-Gov"]
related_patterns: ["agent-economics-moves-to-completed-work"]
novelty_label: "new_fact"
what_changed: "Sakana AI released JFinBench, a benchmark built from 321 questions across six 2026 Japanese financial certification exams, to assess LLMs on local financial knowledge."
prior_refs: ["https://newruntime.com/posts/reported-68-0-on-finsearchcomp-though-third-party-07c21e/","https://newruntime.com/posts/nemotron-personas-finance-japan-a-synthetic-dataset-capturing-f53831/"]
source_url: "https://sakana.ai/jfinbench"
source_urls: ["https://sakana.ai/jfinbench"]
schema_version: "newruntime-agent-readable-v0.2"
stable_id: "post:sakana-ai-released-jfinbench-321-questions-from-six-dce296"
status: "published"
visuals: []
editor: {"role":"editor","name":"Andrey Reshetnikov","profile":"https://newruntime.com/owner-profile.md","profile_json":"https://newruntime.com/owner-profile.json","same_as":["https://www.linkedin.com/in/reshetnikov1/","https://t.me/lapetuse"]}
routes: {"html":"https://newruntime.com/posts/sakana-ai-released-jfinbench-321-questions-from-six-dce296/","markdown":"https://newruntime.com/posts/sakana-ai-released-jfinbench-321-questions-from-six-dce296.md","json":"https://newruntime.com/posts/sakana-ai-released-jfinbench-321-questions-from-six-dce296.json"}
markdown_url: "https://newruntime.com/posts/sakana-ai-released-jfinbench-321-questions-from-six-dce296.md"
---

# Sakana AI released JFinBench, 321 questions from six 2026 Japanese financial certification exams.

## Retrieval answer

Sakana AI has released JFinBench, a Japanese financial benchmark built from 321 questions across six 2026 certification exams. Its results show how domain-specific knowledge can separate models that look closer on general benchmarks, while its agent-assisted construction pipeline offers a way to refresh evaluations with human review.

## Summary
Sakana AI has released JFinBench, a Japanese financial benchmark built from 321 questions across six 2026 certification exams. Its results show how domain-specific knowledge can separate models that look closer on general benchmarks, while its agent-assisted construction pipeline offers a way to refresh evaluations with human review.

## What happened
Sakana AI says JFinBench draws on the latest 2026 editions of six Japanese financial certification exams: the short-answer Japanese CPA exam, the tax accountant exam, Level 1 FP skills exam, Level 1 financial services exam, Level 1 bookkeeping exam from the Japan Chamber of Commerce, and Level 1 bookkeeping exam from the National Association of Commercial High Schools. The exams were held or published between May and August 2026. The benchmark contains 321 questions: 193 multiple-choice items and 128 requiring written answers. Twelve questions require interpreting images, including diagrams.

The benchmark uses different scoring methods for different answer formats. Multiple-choice answers are extracted with regular expressions. Short answers are checked for equivalence, fill-in questions allow partial credit, and essay answers are scored by an LLM judge against rubric points from model answers. Sakana AI reports that a human author checked all 321 converted questions against the original exam materials.

To build the dataset, a common script first renders exam pages as images and uses a vision-language model to transcribe them into layout-preserving Markdown. A coding agent then writes exam-specific code to split the transcription into questions, pair questions with answers, crop figures, and remove page artifacts. It also creates a side-by-side verification page. A human reviewer reports problems in natural language; the agent revises the code and regenerates the questions for another check. Sakana AI says review took about two hours per exam, or about 12 hours in total. The common transcription script can be reused for new exams, while exam-specific conversion code and human review are still needed.

Sakana AI evaluated 12 models: three proprietary and nine open-weight models. In the main setting, each model answered in one API call without tools; results average three runs at maximum reasoning effort. Claude Fable 5 scored 95.8, GPT-6 Astra 93.6, Gemini 3.8 Flash 88.8, and the leading open-weight model, Kimi K3.1 Pro, 83.4. The reported API cost for that setting was $0.555 per question for Claude Fable 5, $0.248 for GPT-6 Astra, and $0.145 for Kimi K3.1 Pro. Sakana AI says knowledge gaps, rather than calculation errors, were the most common failure cause.

In a separate tool-enabled setting, models could use a terminal and web access in Linux containers and spend up to six hours per trial. GPT-6 Astra scored 97.3 and Kimi K3.1 Pro 92.9 with their respective best reported harnesses. Sakana AI says every model improved; across trials using the shared Terminus-2 harness, 67% fetched at least one web page, most often from Japan’s e-Gov legal information portal. The company reports that the strongest proprietary-open gap narrowed from 12.4 points without tools to 4.4 points with them. Sakana AI links the paper, dataset, code, and a leaderboard from its announcement.

## What the archive adds
New Runtime’s earlier coverage includes [AA-Omniscience v1.1’s calibrated public subset](https://newruntime.com/posts/artificial-analysis-releases-aa-omniscience-v1-1-with-b11513/) (October 31, 2026) and a [reported result on FinSearchComp](https://newruntime.com/posts/reported-68-0-on-finsearchcomp-though-third-party-07c21e/) (November 11, 2026). JFinBench is a new event: a benchmark focused on Japanese financial exams, with both a refreshable construction process and model comparisons.

## Trend signal
This supports [agent economics moves to completed work](https://newruntime.com/patterns/agent-economics-moves-to-completed-work/) only indirectly: Sakana AI reports per-question API costs alongside scores, making the trade-off between performance and cost more visible. The central result is still a benchmark comparison, not a measured cost per verified task, so this is a limited fit rather than strong confirmation.

## Why now
Sakana AI built the benchmark from exams administered or published in 2026. The company says tax rules change frequently and old exam questions can become misaligned with current rules; recent questions also reduce the chance that answers were already present in models’ training data. Its stated goal is to keep the benchmark refreshable as new exams appear.

## Why it matters
For teams deploying models in finance or another regulated, jurisdiction-specific domain, general-purpose benchmarks can hide consequential gaps. In Sakana AI’s tests, Kimi K3.1 Pro was competitive on a coding benchmark it cites but trailed GPT-6 Astra by about ten points on JFinBench without tools. Web access helped models retrieve legal information and narrowed the best-model gap, but did not erase it. That makes the benchmark useful as a reminder to evaluate both stored knowledge and the ability to find and apply current rules.

The construction method is also relevant beyond finance. The reusable part is not simply “ask an agent to parse PDFs”: it combines a generic transcription pass, exam-specific transformation code, full human comparison against originals, and an iteration loop for fixing errors. Teams with internal assessments or changing policy documents could adapt that pattern, while preserving human validation as a requirement rather than treating generated structure as ground truth.

## What happens next
- **Sakana AI will publish a refreshed JFinBench version after a subsequent exam cycle.** (12 months, 65%). Watch for: a dated dataset or leaderboard update that adds questions from newly administered exams.

## Limits and uncertainty
These are Sakana AI’s own benchmark results, not an independent replication. Scores depend on the chosen prompts, reasoning settings, scoring methods, and—in the tool-enabled comparison—the best-performing harness selected for each model. The release reports three-run averages, but does not establish how well JFinBench predicts performance in live financial work or other Japanese financial tasks. The benchmark’s questions are exam problems, not a comprehensive test of professional judgment, and the reported cost figures apply to the stated API evaluation setup.

## Skill to build now
If your team relies on a domain benchmark built from documents, prototype the verification loop before scaling it: render source pages, preserve layout during transcription, write transformations for each document format, and compare every resulting item with its original. Have reviewers report errors in a form the coding agent can use to revise the transformation, then rerun checks. Track human review time as well as conversion time; Sakana AI’s reported 12 hours for six exams makes that labor visible.

## Reader takeaway
JFinBench’s strongest lesson is that benchmark choice changes the story: a model’s strength on coding or broad tests does not guarantee command of current local financial rules. Sakana AI’s 321-question release also demonstrates a practical route to keeping document-based evaluations fresh—but only with full human verification.

Editor: [Andrey Reshetnikov](https://newruntime.com/owner-profile.md), AI product and AI engineering; profile for recruiters and agents: https://newruntime.com/owner-profile.json

## Continue in the graph

Next records for an agent, linked to their Markdown twins. Replace `.md` with `.json` for structured fields, including `next_reads`.

- topic hub: [Claude](https://newruntime.com/topics/claude.md): Explore the claude topic hub.
- topic hub: [Gemini](https://newruntime.com/topics/gemini.md): Explore the gemini topic hub.
- collection: [Field Notes](https://newruntime.com/posts.json): Open the latest editorial analysis.
- collection: [Sources](https://newruntime.com/sources/): See the original sources this archive cites.
- whole graph: [discovery graph](https://newruntime.com/discovery-graph.json) (typed nodes and edges) and [site index](https://newruntime.com/site-index.md)
