---
type: "post"
id: "artificial-analysis-releases-aa-omniscience-v1-1-with-b11513"
slug: "artificial-analysis-releases-aa-omniscience-v1-1-with-b11513"
title: "Artificial Analysis releases AA-Omniscience v1.1 with a 3,000-question set and new calibrated public subset"
description: "Artificial Analysis has updated AA-Omniscience to v1.1, a benchmark for factual recall and hallucination across economically relevant domains."
retrieval_nugget: "Artificial Analysis has updated AA-Omniscience to v1.1, a benchmark for factual recall and hallucination across economically relevant domains. The revised evaluation contains 3,000 questions rather than 6,000 and treats a model’s willingness to abstain as part of reliability, giving builders a more relevant signal than accuracy alone for knowledge-sensitive workflows."
published_at: "2026-10-31"
updated_at: "2026-10-31"
record_date: "2026-10-31"
date_kind: "published_at"
topics: ["claude","gemini"]
entities: ["Artificial Analysis","AA-Omniscience","Gemini 4 Carbon","Claude Opus 5.5","Claude Fable 5.1"]
related_patterns: []
novelty_label: "new_fact"
what_changed: "AA-Omniscience v1.1 now has 3,000 questions across six domains, down from the 6,000-question v1 set, and introduces a smaller calibrated public subset."
prior_refs: ["memory:b8a89e19-7571-8feb-af49-16572b1f1cbe","memory:f37d09d2-8572-86fd-a759-5e9b1767dcea"]
source_url: "https://artificialanalysis.ai/evaluations/omniscience"
source_urls: ["https://artificialanalysis.ai/evaluations/omniscience"]
schema_version: "newruntime-agent-readable-v0.2"
stable_id: "post:artificial-analysis-releases-aa-omniscience-v1-1-with-b11513"
status: "published"
visuals: []
editor: {"role":"editor","name":"Andrey Reshetnikov","profile":"https://newruntime.com/owner-profile.md","profile_json":"https://newruntime.com/owner-profile.json","same_as":["https://www.linkedin.com/in/reshetnikov1/","https://t.me/lapetuse"]}
routes: {"html":"https://newruntime.com/posts/artificial-analysis-releases-aa-omniscience-v1-1-with-b11513/","markdown":"https://newruntime.com/posts/artificial-analysis-releases-aa-omniscience-v1-1-with-b11513.md","json":"https://newruntime.com/posts/artificial-analysis-releases-aa-omniscience-v1-1-with-b11513.json"}
---

# Artificial Analysis releases AA-Omniscience v1.1 with a 3,000-question set and new calibrated public subset

## Retrieval answer

Artificial Analysis has updated AA-Omniscience to v1.1, a benchmark for factual recall and hallucination across economically relevant domains. The revised evaluation contains 3,000 questions rather than 6,000 and treats a model’s willingness to abstain as part of reliability, giving builders a more relevant signal than accuracy alone for knowledge-sensitive workflows.

## Summary

Artificial Analysis has updated AA-Omniscience to v1.1, a benchmark for factual recall and hallucination across economically relevant domains. The revised evaluation contains 3,000 questions rather than 6,000 and treats a model’s willingness to abstain as part of reliability, giving builders a more relevant signal than accuracy alone for knowledge-sensitive workflows.

## What happened

Artificial Analysis has published AA-Omniscience v1.1, describing it as a benchmark for knowledge and hallucination. Its purpose is to measure factual recall and factual reliability across domains where errors can have economic significance. The evaluation covers six major domains and 42 topics. According to the benchmark page, questions are derived from authoritative academic and industry sources, then generated automatically with an LLM-based question-generation agent. Artificial Analysis says that process is intended to support unambiguous, scalable, factually precise questions.

The v1.1 dataset contains 3,000 questions, down from the 6,000 questions in the v1 dataset described in the accompanying paper. Artificial Analysis says it conducts the evaluations independently and links to its methodology, paper, example tasks, and a public Hugging Face dataset. The page does not, in the supplied material, specify how teams should treat the public dataset as a substitute for the private evaluation or provide a publication date for v1.1.

The core result is the AA-Omniscience Net Score, a bounded measure from -100 to 100. It rewards correct answers, penalizes hallucinations, and assigns no penalty when a model refuses to answer. A score of zero means a model produces as many correct answers as incorrect ones; a negative score means incorrect answers outnumber correct answers. This design makes abstention operationally meaningful rather than treating every unanswered question as an equivalent failure.

The current benchmark page lists Gemini 4 Carbon (High) first on AA-Omniscience with a score of 54, followed by Claude Opus 5.5 (Max, Default Fallback) at 48 and Claude Fable 5.1 (Max, Default Fallback) at 46. It separately reports accuracy and hallucination-rate views. The page also presents a cost-per-task comparison and indicates that results are available for a subset of the models it tracks; it does not establish that the leaderboard alone predicts production performance for a specific team’s data, prompts, tools, or review process.

## What the archive adds

This is a new fact on a known Artificial Analysis evaluation story. The archive previously covered the [Artificial Analysis Decision Index v1.0](https://newruntime.com/posts/artificial-analysis-launched-the-decision-index-v1-0-5ce031/), which treated bounded choices as a distinct task class, and noted Artificial Analysis’ inclusion of [FrontierCode 2.0 and Terminal-Bench-Science](https://newruntime.com/posts/new-benchmarks-cognition-s-frontiercode-2-0-merge-a78e19/) in its methodology. AA-Omniscience adds factuality and calibration as another explicit evaluation surface.

## Trend signal

No supplied archive assumption or pattern record is specific enough to establish a broader trend from this benchmark revision alone.

## Why it matters

Teams often select a model using aggregate capability, coding performance, latency, or token price. That can be insufficient when an application produces factual answers in domains where a plausible error triggers costly review, customer harm, or downstream rework. AA-Omniscience supplies a distinct selection criterion: net factual reliability after accounting for both incorrect answers and safe abstentions.

That does not mean a team should adopt the top-ranked model wholesale. The useful move is to make abstention policy explicit in a workflow: decide which requests can be answered directly, which require retrieval or tool use, and which should be escalated to a human. Then test candidate models against examples that reflect the team’s own domain, stakes, and acceptable refusal rate.

## What happens next

- **Teams building factual-answer workflows will add abstention or escalation measures alongside accuracy in model evaluations.** (6 months, 65%). Watch for: published evaluation scorecards that separate wrong answers, refusals, and verified answers.
- **Benchmark users will request more domain-level and reproducible comparisons rather than relying on an overall net score.** (3 months, 60%). Watch for: released task subsets, methodology revisions, or model-selection reports by domain.

## Limits and uncertainty

The supplied source establishes the benchmark’s stated design and its current displayed results, not independent validation of every question, annotation, or ranking. It does not show how closely the questions match any one organization’s proprietary documents, retrieval stack, tools, prompts, or human-review policy. A neutral score for refusal is a deliberate evaluation choice, but different products may face a different cost for declining to answer. The public dataset link demonstrates availability, while the supplied material does not establish its size, calibration, licensing implications, or equivalence to the private evaluation. Leaderboard results should therefore be treated as a screening signal, not a deployment decision.

## Skill to build now

Create a small factuality test set from real requests your product receives. For each item, define a verified answer, the evidence required to answer safely, and whether abstention is preferable to a guess. Run the same set through candidate models with your intended system prompt and retrieval configuration. Report correct, unsupported, incorrect, and abstained outcomes separately. This gives the team a local version of the trade-off AA-Omniscience is designed to surface.

## Reader takeaway

AA-Omniscience v1.1 is a reminder that factual AI quality is not simply answer rate or raw accuracy. For knowledge-heavy systems, the important comparison is whether a model can combine correct recall with calibrated uncertainty. Test that behavior against the real decisions your application must make.

Editor: [Andrey Reshetnikov](https://newruntime.com/owner-profile.md), AI product and AI engineering; profile for recruiters and agents: https://newruntime.com/owner-profile.json

## Continue in the graph

Next records for an agent, linked to their Markdown twins. Replace `.md` with `.json` for structured fields, including `next_reads`.

- topic hub: [Claude](https://newruntime.com/topics/claude.md): Explore the claude topic hub.
- topic hub: [Gemini](https://newruntime.com/topics/gemini.md): Explore the gemini topic hub.
- collection: [Field Notes](https://newruntime.com/posts.json): Open the latest editorial analysis.
- collection: [Source Ledger](https://newruntime.com/source-ledger.json): Inspect the public source evidence graph.
- whole graph: [discovery graph](https://newruntime.com/discovery-graph.json) (typed nodes and edges) and [site index](https://newruntime.com/site-index.md)