---
type: "post"
id: "your-ai-passed-the-test-did-it-answer-e7da89"
slug: "your-ai-passed-the-test-did-it-answer-e7da89"
title: "Your AI Passed the Test. Did It Answer the Business Question?"
description: "Ted Tschopp argues that an AI evaluation can grade an answer as correct while missing whether the evidence supports the business conclusion."
retrieval_nugget: "Ted Tschopp argues that an AI evaluation can grade an answer as correct while missing whether the evidence supports the business conclusion. His cloud-spending example shows why teams need tests for scope, uncertainty and intended meaning—not just for whether a number appears in a source."
published_at: "2026-11-02"
updated_at: "2026-11-06"
record_date: "2026-11-02"
date_kind: "published_at"
topics: ["anthropic"]
entities: ["Ted Tschopp","TIFIN","Anthropic","OpenAI"]
related_patterns: ["ai-engineering-evaluations-are-splitting-into-task-specific"]
novelty_label: "new_angle"
what_changed: "It shifts AI-evaluation discussion from FAQ questions about references and trace review to whether evaluations establish that an answer is supported and addresses the business task."
prior_refs: []
source_url: "https://tedt.org/Your-AI-Passed-the-Test"
source_urls: ["https://tedt.org/Your-AI-Passed-the-Test"]
schema_version: "newruntime-agent-readable-v0.2"
stable_id: "post:your-ai-passed-the-test-did-it-answer-e7da89"
status: "published"
visuals: []
editor: {"role":"editor","name":"Andrey Reshetnikov","profile":"https://newruntime.com/owner-profile.md","profile_json":"https://newruntime.com/owner-profile.json","same_as":["https://www.linkedin.com/in/reshetnikov1/","https://t.me/lapetuse"]}
routes: {"html":"https://newruntime.com/posts/your-ai-passed-the-test-did-it-answer-e7da89/","markdown":"https://newruntime.com/posts/your-ai-passed-the-test-did-it-answer-e7da89.md","json":"https://newruntime.com/posts/your-ai-passed-the-test-did-it-answer-e7da89.json"}
related_shifts: ["verification-ownership"]
markdown_url: "https://newruntime.com/posts/your-ai-passed-the-test-did-it-answer-e7da89.md"
---

# Your AI Passed the Test. Did It Answer the Business Question?

## Retrieval answer

Ted Tschopp argues that an AI evaluation can grade an answer as correct while missing whether the evidence supports the business conclusion. His cloud-spending example shows why teams need tests for scope, uncertainty and intended meaning—not just for whether a number appears in a source.

## Summary
Ted Tschopp argues that an AI evaluation can grade an answer as correct while missing whether the evidence supports the business conclusion. His cloud-spending example shows why teams need tests for scope, uncertainty and intended meaning—not just for whether a number appears in a source.

## What happened
In an essay published November 1, 2026, Ted Tschopp explains evaluations as structured tests: give an AI system a task, observe its behavior, and compare the result with expectations defined in advance. Those expectations may concern a correct answer, a completed action, a necessary question, or recognition that the available information cannot support a conclusion. An evaluation can include conventional software tests and user acceptance testing, but it is not the same as either. It examines a particular combination of model, instructions, tools and data, across a collection of cases and conditions.

Tschopp’s central example is a fictional assistant for cloud spending. Reports say that $120,000 was spent on systems used by customers and employees, $30,000 on software development, and $10,000 on testing; together, they cover all company cloud costs for a quarter. Asked for total spending, the assistant should say $160,000. But if it receives only the production-cost report, it may answer that total company spending was $120,000. The figure appears in the report, yet the report explicitly excludes development and testing. A checker that merely confirms the number appears in the source would pass an unsupported company-wide claim.

The essay argues that this is not only a weakness in the grader. The team may have defined success too narrowly. A better grader cannot compensate for an evaluation that fails to say what the business needs. Teams should define representative cases, the information available in each, and successful behavior before running tests. A test runner submits the cases, captures answers or actions, applies grading rules and records results. After a change to the product, the team can repeat the cases and compare behavior. Grading may use ordinary software, a person or another AI; different parts of a task can use different methods. For an assistant that updates a record, the resulting record should be checked alongside its description of what it did.

Tschopp also describes related research reported in the essay. A TIFIN study of constructed Indian mutual-fund questions found that an AI judge, supplied with field definitions, identified 40 of 44 mislabeled figures. Without knowing the intended interpretation, it accepted 42 of 52 answers that silently selected one reading of an ambiguous question. The essay also mentions research finding numerical claims drawn from the wrong experiment, metric or run, and a study in which models performed worse on policy questions lacking enough information. These are controlled research cases, not estimates of enterprise failure rates. Tschopp recommends involving domain specialists to specify what counts as correct, testing the grader itself, and tracking consequential errors separately from unnecessary clarification.

## What the archive adds
[When AI Moves Faster Than the Organization](https://newruntime.com/posts/when-ai-moves-faster-than-the-organization-analysis-61d4a8/) argued that enterprise AI capability belongs to the whole working system, not to a model or agent count in isolation. This essay adds a concrete evaluation example to that concern: a score can improve while a business-critical behavior regresses. It is a new fact on a related theme, not a restatement.

## Trend signal
This supports the pattern that verification bandwidth is the scarce engineering resource. The difficult part of agentic delivery is not merely generating output, but specifying and checking whether it meets the task’s actual requirements. Tschopp’s example makes that burden concrete: someone must encode the distinction between a valid partial figure and an unsupported total, then test both incomplete and complete evidence conditions.

## Why now
The essay situates evaluation as an ongoing release practice rather than a one-time model benchmark. It says that changes to a model, instructions, search process or data fields can alter behavior, so teams should rerun relevant cases after changes. Its cited studies are discussed as evidence that narrow scoring rules can miss interpretive failures.

## Why it matters
A system that produces fluent answers and plausible numbers can still make a consequential mistake when it silently expands a source’s scope. For teams, the practical implication is to turn business intent into testable requirements: for example, “do not present production-only costs as a company-wide total.” That requires engineers and domain specialists to agree on meanings such as spending, revenue or a completed case.

Evaluation results should also inform a decision, not become a certificate of general reliability. A higher aggregate score can hide a regression that the product owner considers release-blocking. Conversely, making an assistant ask more questions may reduce some unsupported claims while burdening users when records are complete. Tests that include both missing-information and complete-information cases help teams detect that trade-off. The evaluation identifies behavior to investigate; it does not, by itself, locate the defect or fix the product.

## Limits and uncertainty
The cloud-spending scenario is fictional, and the essay does not provide production measurements for how often assistants make this kind of scope error. The cited mutual-fund and policy studies use constructed or bounded research cases; they do not establish enterprise-wide failure rates. The essay describes evaluation practices and possible checks, but does not report a controlled comparison showing that a particular design reliably improves deployed systems. A passing score remains evidence only about selected cases and grading criteria; it cannot guarantee behavior on unfamiliar tasks or after the system changes.

## Skill to build now
Write two paired evaluation cases for one consequential business question: one where the available records are incomplete and one where they are complete. Specify what the assistant must do in each, then have a domain owner review the expected behavior. For the incomplete case, reject an unsupported total; for the complete case, require a direct answer. Include a check that the grader accepts valid paraphrases and catches claims that overstate the evidence.

## Reader takeaway
Do not ask only whether the answer contains a correct fact. Ask whether the available evidence supports the conclusion the user actually needs. A useful evaluation makes that distinction testable—and treats its score as bounded evidence, not a guarantee.

Editor: [Andrey Reshetnikov](https://newruntime.com/owner-profile.md), AI product and AI engineering; profile for recruiters and agents: https://newruntime.com/owner-profile.json

## Continue in the graph

Next records for an agent, linked to their Markdown twins. Replace `.md` with `.json` for structured fields, including `next_reads`.

- topic hub: [Anthropic](https://newruntime.com/topics/anthropic.md): Explore the anthropic topic hub.
- collection: [Field Notes](https://newruntime.com/posts.json): Open the latest editorial analysis.
- collection: [Sources](https://newruntime.com/sources/): See the original sources this archive cites.
- whole graph: [discovery graph](https://newruntime.com/discovery-graph.json) (typed nodes and edges) and [site index](https://newruntime.com/site-index.md)