What happened
Vercel published “WTF are evals? How to test agents and the software they use,” by George Fahmy and Yury Selivanov, on November 12, 2026. The authors frame agents as users of software, including APIs, command-line interfaces, MCP servers, and SDKs. They describe a product funnel of discovery, selection, access, and success, then focus the guide on success: whether an agent completes a user’s task correctly.
The guide groups tests by the number of LLM calls they typically require. Deterministic tests use none and check expected software behavior. One-turn evaluations use one call to test a single decision, such as selecting a command and forming its arguments. End-to-end evaluations use two or more calls across an entire task, in an environment, and check the final result. Vercel says call count is a useful proxy for effort, runtime, and cost, so teams should use cheaper tests wherever they can catch the problem.
Its running example is SF Beans, a hypothetical coffee-ordering API exposed through a CLI. The user asks for a large iced latte with whole milk, ready by 9:00, under $8 including tip, in San Francisco’s Mission district. The example distinguishes cafes that fail one constraint from a cafe that meets them all. A deterministic test can check search results, tax and tip calculations, and order charges. A one-turn eval can check whether the agent searches the right area or quotes a price before ordering. An end-to-end eval checks whether it places exactly one order satisfying the drink, timing, and budget constraints.
For the end-to-end example, Vercel uses Harbor, an open-source framework from the creators of Terminal-Bench, to run agents in containers, check results, and record actions. It defines a task as a problem with a starting environment and success check; a variant as the agent setup being tested; a trial as one attempt; and a run as the set of variants tested across tasks and repeated attempts. The recorded trace includes messages, tool calls, logs, files, and final output.
Vercel recommends starting with one narrow outcome that matters to users, drawing candidate goals from observability, customer feedback, or the roadmap. The example chooses correct drink ordering, with Claude Code and Codex as target agents, and excludes cancellations, refunds, and accounts. Metrics include success rate, pass@k (at least one success in k trials), and pass^k (success in all k trials); efficiency metrics—steps, time, and cost—are measured on successful trials. In an illustrative five-trial run with Claude Code and Opus 5.5, three trials pass: 60% success, approximately 99% pass@5, and 8% pass^5. Vercel emphasizes that the relevant metric depends on the cost of failure, and that a small sample with no failures does not prove an agent never fails. The guide also calls for tasks based on real requests, consistent starting conditions, verifiers that score constraints, and common, edge, and no-action cases.
What the archive adds
New Runtime’s earlier coverage on January 12, 2026, described why agents need evaluation beyond ordinary LLM answer checks: they take multi-step actions and change an environment. Vercel adds a structured testing pyramid, explicit reliability metrics, and a worked task-design example. This is a new practical treatment of a known topic, not a wholly new thread.
Trend signal
This supports New Runtime’s verification-bandwidth thesis: as agents perform longer tasks, teams need repeatable ways to verify outcomes rather than relying on plausible-looking traces or a successful manual run. Vercel’s distinction between pass@k and pass^k makes the verification question concrete: an agent that can succeed eventually may still be too unreliable for a task where each failure reaches the user.
Why now
Vercel’s guide says agents are increasingly doing work on people’s behalf and using software interfaces such as APIs, CLIs, MCP servers, and SDKs. That makes successful task completion a product concern for both agent builders and the teams exposing capabilities to agents, rather than solely a model-evaluation problem.
Why it matters
A team can mistake a clean tool-call log or a single successful demo for reliability. The guide offers a way to separate those signals: test a bounded outcome repeatedly, score the final state against user constraints, and select a metric that reflects the cost of a miss. A retry may be acceptable when a coding agent can use test feedback before delivery; it is less acceptable when an agent places an irreversible order or sends an unreviewed action. The testing pyramid also helps manage evaluation cost: deterministic checks should handle ordinary software behavior, while expensive end-to-end runs are reserved for failures that require observing the whole workflow. This is a practical foundation for deciding what to automate, what to review, and where evidence is still thin.
Limits and uncertainty
The coffee-ordering results are illustrative, not a reported benchmark of production agents or customers. Five trials are too few to establish dependable failure rates, and the example’s pass@k calculation assumes independent trials with a stable 60% success rate. Results can vary with task selection, environment realism, model and prompt configuration, tools, and verifier quality. The guide’s metrics make those choices visible but do not remove the need to validate that the test set represents real user work or that its verifier catches consequential mistakes.
Skill to build now
Pick one agent task that users care about and define its success as observable constraints on the final state. Build a deterministic test for anything the product itself can guarantee, then add a repeated end-to-end task for failures that depend on a sequence of agent decisions. Include one impossible or unsafe request where the correct outcome is not to act, and report both the rate of any success and the rate of success on every trial if users cannot safely retry.
Reader takeaway
An agent evaluation should answer not only whether the agent can complete a task, but how reliably it does so under repeated attempts and what a failure costs. Build the cheapest test that catches the problem, and measure the final outcome users actually care about.