---
type: "post"
id: "new-public-300-question-set-on-hugging-face-38248d"
slug: "new-public-300-question-set-on-hugging-face-38248d"
title: "New public 300-question set on Hugging Face correlates 0.99 with the private benchmark."
description: "Artificial Analysis has published AA-Omniscience-Public-v1.1, a public 300-question dataset on Hugging Face."
retrieval_nugget: "Artificial Analysis has published AA-Omniscience-Public-v1.1, a public 300-question dataset on Hugging Face. For teams building with AI, the useful development is not a leaderboard claim but a reusable evaluation artifact: it can be run against a model or pipeline and inspected question by question."
published_at: "2026-10-31"
updated_at: "2026-10-31"
record_date: "2026-10-31"
date_kind: "published_at"
topics: ["coding-agents","evals"]
entities: ["Artificial Analysis","AA-Omniscience-Public-v1.1","Hugging Face"]
related_patterns: ["verification-bandwidth-is-the-scarce-resource"]
novelty_label: "new_fact"
what_changed: "Adds the public subset’s precise size (300 questions) and reports a 0.99 correlation with the private benchmark."
prior_refs: ["memory:31496b65-38f0-8762-a843-645319803d1c","memory:f62cee9f-ef98-88b0-a473-bec1275b4953"]
source_url: "https://huggingface.co/datasets/ArtificialAnalysis/AA-Omniscience-Public-v1.1"
source_urls: ["https://huggingface.co/datasets/ArtificialAnalysis/AA-Omniscience-Public-v1.1"]
schema_version: "newruntime-agent-readable-v0.2"
stable_id: "post:new-public-300-question-set-on-hugging-face-38248d"
status: "published"
visuals: []
editor: {"role":"editor","name":"Andrey Reshetnikov","profile":"https://newruntime.com/owner-profile.md","profile_json":"https://newruntime.com/owner-profile.json","same_as":["https://www.linkedin.com/in/reshetnikov1/","https://t.me/lapetuse"]}
routes: {"html":"https://newruntime.com/posts/new-public-300-question-set-on-hugging-face-38248d/","markdown":"https://newruntime.com/posts/new-public-300-question-set-on-hugging-face-38248d.md","json":"https://newruntime.com/posts/new-public-300-question-set-on-hugging-face-38248d.json"}
---

# New public 300-question set on Hugging Face correlates 0.99 with the private benchmark.

## Retrieval answer

Artificial Analysis has published AA-Omniscience-Public-v1.1, a public 300-question dataset on Hugging Face. For teams building with AI, the useful development is not a leaderboard claim but a reusable evaluation artifact: it can be run against a model or pipeline and inspected question by question.

## Summary

Artificial Analysis has published AA-Omniscience-Public-v1.1, a public 300-question dataset on Hugging Face. For teams building with AI, the useful development is not a leaderboard claim but a reusable evaluation artifact: it can be run against a model or pipeline and inspected question by question.

## What happened

The primary source is a Hugging Face dataset page for `ArtificialAnalysis/AA-Omniscience-Public-v1.1`. Its dataset viewer lists one subset, named `default`, and one split, named `test`. Both the displayed row count for the subset and the split is 300. The page also offers the usual dataset-viewer functions, including an automatically converted Parquet representation, an API entry point, and Data Studio.

Each displayed record has five fields: `domain`, `subtopic`, `question_id`, `question`, and `answer`. The viewer reports six values for the domain field and 49 for subtopic. Question identifiers run across the 300 displayed rows. The material therefore exposes more than a single aggregate score: a user can see the individual prompts and their reference answers in the public viewer.

The examples indicate a broad academic and technical mix. The first rows include biology questions about flagellar rootlets and turtle magnetic-displacement experiments; chemistry questions about composted alginate, Titan atmospheric shocks, and platinum chemistry; and questions in civil engineering, electrical engineering, geography and environmental science, and geology. Many are narrow factual or technical prompts with a short expected answer, such as a measurement, formula, device name, classification, or scientific term.

That structure makes the release immediately usable as a fixed test corpus. A builder can submit the same 300 prompts to a chosen model or system, retain the generated outputs, and compare them with the supplied answer column. It also makes failures inspectable at the row level: a result can be grouped by domain or subtopic rather than reduced only to one total number.

The supplied public material does not describe how the questions were selected, how answers should be matched or graded, whether prompts require a particular inference configuration, or what licence governs reuse. It also does not, in the visible excerpt, provide a methodology or evidence for any claimed correspondence with a private benchmark. Those omissions matter when interpreting a result or deciding whether to put the set into a release gate.

## What the archive adds

This is a new public artifact in an existing Artificial Analysis evaluation story. [Artificial Analysis launches Intelligence Index v5, with 60% of the score on private test sets](https://newruntime.com/posts/artificial-analysis-launches-intelligence-index-v5-with-60-59ecd7/) described an index that moved much of its scoring toward private or held-out tests. [Artificial Analysis launches HLE-Diamond leaderboard as part of Intelligence Index v5](https://newruntime.com/posts/artificial-analysis-launches-hle-diamond-leaderboard-as-part-0fef9e/) covered another named evaluation in that index. The new dataset adds inspectable public questions rather than another announced aggregate.

## Trend signal

This weakly supports [verification bandwidth is the scarce engineering resource](https://newruntime.com/patterns/verification-bandwidth-is-the-scarce-resource/). A public set lowers the friction of reproducing an evaluation run and examining individual errors, but it does not remove the work of designing a relevant test harness, judging ambiguous outputs, or deciding which failures matter in production.

## Connections to other events

The release sits alongside Artificial Analysis's Intelligence Index v5, whose archive record says that 60% of scoring uses private or held-out tests, and its HLE-Diamond evaluation, a refined 1,000-question Humanity's Last Exam set. Together, these show an evaluation portfolio containing both closed or held-out components and publicly inspectable tasks; the supplied material does not state how this dataset is used in that portfolio.

## Why it matters

Teams often discover model regressions only after a change reaches a workflow: a new model version answers a specialist question differently, a retrieval layer distracts the model, or a tool-enabled agent stops at the wrong intermediate step. A small public corpus is not a substitute for a product-specific evaluation suite, but it is a concrete baseline that can be run before and after such changes.

The main opportunity is operational discipline. Treat the 300 rows as an external check alongside internal tasks, not as a universal ranking. Preserve model version, system prompt, tool availability, sampling settings, raw output, and the matching rule used for each run. That turns a public dataset into a repeatable comparison rather than a screenshot of a score.

## What happens next

- **Teams and evaluators will publish reproducible runs on the 300 questions.** (3 months, 60%). Watch for: shared run configurations and row-level outputs.
- **The value of the dataset will depend on accompanying evaluation guidance or tooling.** (6 months, 55%). Watch for: documented scoring rules, dataset revisions, or a stated reuse licence.

## Limits and uncertainty

The dataset viewer establishes the existence, size, fields, examples, and public accessibility of the 300-row test split. It does not establish that the set measures general intelligence, predicts production performance, or is resistant to contamination. It also does not say whether exact string matching, model-based judging, human review, or another method should be used to evaluate answers. Some displayed prompts request precise domain facts, so a score may be sensitive to formatting and answer normalization. The supplied material contains no model results, no cost data, no release date, and no explanation of how questions or answers were validated.

## Skill to build now

Build a minimal evaluation ledger this week. Download or query the public rows, run a small fixed sample through one model configuration, and save the prompt, raw answer, normalized answer, pass/fail rule, latency, and model identifier for every row. Then repeat after one controlled change, such as switching a model or altering retrieval. Review disagreements manually before treating the delta as a product decision. The exercise exposes whether the team can reproduce a result and explain a failure, which matters more than producing a single headline percentage.

## Reader takeaway

AA-Omniscience-Public-v1.1 gives builders 300 public question-and-answer rows they can use as a repeatable external evaluation check. Its availability makes inspection easier; it does not by itself prove what a score means. Use it as one component of a versioned evaluation harness, with your own task-specific tests and explicit judgment rules.

## Open question

Will Artificial Analysis publish the selection process, scoring protocol, reuse licence, and evidence needed to interpret this public set alongside its private or held-out evaluations?

Editor: [Andrey Reshetnikov](https://newruntime.com/owner-profile.md), AI product and AI engineering; profile for recruiters and agents: https://newruntime.com/owner-profile.json

## Continue in the graph

Next records for an agent, linked to their Markdown twins. Replace `.md` with `.json` for structured fields, including `next_reads`.

- topic hub: [Coding Agents](https://newruntime.com/topics/coding-agents.md): Explore the coding-agents topic hub.
- topic hub: [Evals](https://newruntime.com/topics/evals.md): Explore the evals topic hub.
- collection: [Field Notes](https://newruntime.com/posts.json): Open the latest editorial analysis.
- collection: [Source Ledger](https://newruntime.com/source-ledger.json): Inspect the public source evidence graph.
- whole graph: [discovery graph](https://newruntime.com/discovery-graph.json) (typed nodes and edges) and [site index](https://newruntime.com/site-index.md)