What happened
Artificial Analysis has published AA-Omniscience v1.1, describing it as a benchmark for knowledge and hallucination. Its purpose is to measure factual recall and factual reliability across domains where errors can have economic significance. The evaluation covers six major domains and 42 topics. According to the benchmark page, questions are derived from authoritative academic and industry sources, then generated automatically with an LLM-based question-generation agent. Artificial Analysis says that process is intended to support unambiguous, scalable, factually precise questions.
The v1.1 dataset contains 3,000 questions, down from the 6,000 questions in the v1 dataset described in the accompanying paper. Artificial Analysis says it conducts the evaluations independently and links to its methodology, paper, example tasks, and a public Hugging Face dataset. The page does not, in the supplied material, specify how teams should treat the public dataset as a substitute for the private evaluation or provide a publication date for v1.1.
The core result is the AA-Omniscience Net Score, a bounded measure from -100 to 100. It rewards correct answers, penalizes hallucinations, and assigns no penalty when a model refuses to answer. A score of zero means a model produces as many correct answers as incorrect ones; a negative score means incorrect answers outnumber correct answers. This design makes abstention operationally meaningful rather than treating every unanswered question as an equivalent failure.
The current benchmark page lists Gemini 4 Carbon (High) first on AA-Omniscience with a score of 54, followed by Claude Opus 5.5 (Max, Default Fallback) at 48 and Claude Fable 5.1 (Max, Default Fallback) at 46. It separately reports accuracy and hallucination-rate views. The page also presents a cost-per-task comparison and indicates that results are available for a subset of the models it tracks; it does not establish that the leaderboard alone predicts production performance for a specific team’s data, prompts, tools, or review process.
What the archive adds
This is a new fact on a known Artificial Analysis evaluation story. The archive previously covered the Artificial Analysis Decision Index v1.0, which treated bounded choices as a distinct task class, and noted Artificial Analysis’ inclusion of FrontierCode 2.0 and Terminal-Bench-Science in its methodology. AA-Omniscience adds factuality and calibration as another explicit evaluation surface.
Trend signal
No supplied archive assumption or pattern record is specific enough to establish a broader trend from this benchmark revision alone.
Why it matters
Teams often select a model using aggregate capability, coding performance, latency, or token price. That can be insufficient when an application produces factual answers in domains where a plausible error triggers costly review, customer harm, or downstream rework. AA-Omniscience supplies a distinct selection criterion: net factual reliability after accounting for both incorrect answers and safe abstentions.
That does not mean a team should adopt the top-ranked model wholesale. The useful move is to make abstention policy explicit in a workflow: decide which requests can be answered directly, which require retrieval or tool use, and which should be escalated to a human. Then test candidate models against examples that reflect the team’s own domain, stakes, and acceptable refusal rate.
What happens next
- Teams building factual-answer workflows will add abstention or escalation measures alongside accuracy in model evaluations. (6 months, 65%). Watch for: published evaluation scorecards that separate wrong answers, refusals, and verified answers.
- Benchmark users will request more domain-level and reproducible comparisons rather than relying on an overall net score. (3 months, 60%). Watch for: released task subsets, methodology revisions, or model-selection reports by domain.
Limits and uncertainty
The supplied source establishes the benchmark’s stated design and its current displayed results, not independent validation of every question, annotation, or ranking. It does not show how closely the questions match any one organization’s proprietary documents, retrieval stack, tools, prompts, or human-review policy. A neutral score for refusal is a deliberate evaluation choice, but different products may face a different cost for declining to answer. The public dataset link demonstrates availability, while the supplied material does not establish its size, calibration, licensing implications, or equivalence to the private evaluation. Leaderboard results should therefore be treated as a screening signal, not a deployment decision.
Skill to build now
Create a small factuality test set from real requests your product receives. For each item, define a verified answer, the evidence required to answer safely, and whether abstention is preferable to a guess. Run the same set through candidate models with your intended system prompt and retrieval configuration. Report correct, unsupported, incorrect, and abstained outcomes separately. This gives the team a local version of the trade-off AA-Omniscience is designed to surface.
Reader takeaway
AA-Omniscience v1.1 is a reminder that factual AI quality is not simply answer rate or raw accuracy. For knowledge-heavy systems, the important comparison is whether a model can combine correct recall with calibrated uncertainty. Test that behavior against the real decisions your application must make.