Field noteNov 20, 20265 min read1 source

Harvey LAB-AA v1.1 Benchmark

In one minute

Artificial Analysis has updated its implementation of Harvey’s Legal Agent Benchmark to v1.1, adding a source-document check that zeros a task’s headline score when the deliverable contains a material hallucination. For teams evaluating legal agents, the change makes factual grounding part of success rather than a separate quality metric—and makes v1.1 scores incompatible with v1.0.

What happened

Artificial Analysis says Harvey LAB-AA v1.1 tests agents on Harvey’s private dataset of 120 tasks across 24 legal practice areas. Each task provides partner-style instructions and case documents in a sandbox; the agent reads the materials and produces a legal deliverable. The tasks are designed to represent longer legal work rather than isolated question answering.

Artificial Analysis built v1.1 in collaboration with Harvey and runs the evaluation independently using its Stirrup agent harness. The scoring process uses a panel of three LLM judges, which grades each deliverable criterion by criterion against a task-specific rubric. A separate, two-stage check compares every deliverable with the task’s source documents. The company says Harvey’s human preference studies found hallucinations to be a primary factor when legal professionals choose between otherwise comprehensive responses, motivating the new check.

The headline measure is the Hallucination-Gated All-Pass Rate: a task earns credit only when the judges find every rubric criterion passed and the deliverable contains no material hallucination. Artificial Analysis averages the panel’s results across tasks. Its leaderboard lists Astra for Law 6.2 (Max) first at 20.8%, followed by GPT-6.2 Astra (Max) at 15.0% and GPT-6.2 Astra (High) at 13.3%. The page also reports a raw Criterion Pass Rate, averaged across the three judges and pooled across rubric criteria, alongside material hallucinations per checked task.

The distinction changes how leaderboard figures should be read. Muse Spark 1.3 (Max), for example, has a 96.0% Criterion Pass Rate but an 8.9% Hallucination-Gated All-Pass Rate, with 1.68 material hallucinations per task in the table. Astra for Law 6.2 (Max) has a 94.4% Criterion Pass Rate, a 20.8% gated rate, and 0.03 material hallucinations per task. The benchmark defines a material hallucination as an error that could mislead on a substantive point, such as a wrong party, amount, date or obligation. Minor hallucinations are reported but do not affect the headline score.

The release also adds Harvey’s latest private dataset, with improvements to tasks and criteria, and updates the judges to GPT-6 Sol, Grok 4.7 and Claude Opus 5.5. Artificial Analysis says it found minimal self-preference and averages all three judges to mitigate possible bias. The page notes that 27 of 45 models have results in several comparisons. It also gives task costs: the listed lowest is Gemini 3.5 Flash-Lite at $0.14, followed by Inkling Small at $0.15 and Claude Haiku 5.5 (Low) at $0.17. Artificial Analysis explicitly warns that v1.1 scores are not comparable with v1.0.

What the archive adds

New Runtime’s earlier coverage of Artificial Analysis’s per-benchmark results and methodology said that inspectable tasks and scoring make aggregate benchmarks more useful, while not proving that a benchmark matches a team’s deployment. LAB-AA v1.1 is a new fact on that broader measurement story: it makes material hallucinations a hard gate on task credit.

Trend signal

This supports scientific agents need artifact evidence: evaluation becomes more inspectable when another party can examine tasks, inputs, outputs and scoring. The benchmark’s public example tasks and source documents make parts of the evaluation concrete, while the main dataset remains private; this is evidence about evaluation design, not proof that results will reproduce in a team’s own legal workflow.

Why it matters

For teams assessing document-heavy agents, a high average rubric pass rate can conceal a failure mode that matters more than a missed formatting or coverage criterion. A wrong party, amount, date or obligation can make an otherwise complete legal deliverable unsafe to rely on. LAB-AA’s gated score expresses that asymmetry directly: full task credit requires both rubric completion and no material hallucination.

The benchmark also illustrates a practical evaluation choice: track criterion completion and source-grounded errors separately, then make severe errors consequential in the aggregate score. That does not mean every deployment should adopt the benchmark’s exact threshold or judge panel. Teams should define materiality against their own workflows, check outputs against the records that support them, and treat benchmark rankings as comparative signals rather than guarantees of performance on their contracts or jurisdictions.

Limits and uncertainty

The results cover Harvey’s 120-task private set across 24 practice areas, not the full range of legal work or any one organization’s documents. The headline depends on task rubrics, three LLM judges and the two-stage hallucination check; Artificial Analysis describes averaging the judges as a way to mitigate bias, not eliminate it. The source does not establish that benchmark rankings predict performance in production, and the private task set limits external inspection of the full evaluation. Costs are evaluation averages, not a promise of the same per-task expense elsewhere. Version 1.1 scores cannot be compared with v1.0.

Reader takeaway

The useful signal is not simply which model ranks first. It is that the benchmark treats a substantive, source-contradicted claim as disqualifying even when a response passes its other criteria. When evaluating agents for consequential document work, separate rubric coverage from grounded factual error—and make the latter visible in the decision rule.

Evidence / sources

  1. 1artificialanalysis.ai/evaluations/harvey-lab-aa
All Field Notes

Next reads

Read next

Open the graph
  1. 01topicEvalsExplore the evals topic hub.
  2. 02topicVerificationExplore the verification topic hub.
  3. 03archiveField NotesOpen the latest editorial analysis.
  4. 04sourcesSourcesSee the original sources this archive cites.
Who read this page?Machine requests, hidden until opened

Loading the privacy-safe route aggregate...

Open the JSON contract