What happened
Visa announced the Visa Agentic Commerce Benchmark on November 5, 2026. The first round tested 22 AI models from 12 labs against 15 shopping challenges, including choosing a TV, laptop, or personal-care appliance. Visa frames shopping as a requirements-discovery task: a consumer may know the product category they want without knowing or stating all relevant specifications. An agent must use conversation to uncover needs, and should decline to buy if nothing meets them.
A run passes only when the model discovers every task-defined customer requirement and makes the correct purchase or no-purchase decision. Visa says this makes its benchmark distinct from one that scores only a product recommendation. In a TV example, the customer mentions a recommended brand and membership in a third-party supplier program, while leaving other needs unstated. The agent must elicit budget and additional criteria such as format compatibility, HDR10+ support for family videos, and gaming capabilities.
Visa reports an overall pass rate of 57.2%. The best-performing models passed 82.2% of attempts and the worst passed 28.9%. The five models that uncovered the most hidden buying factors averaged a 70.2% pass rate; the five that uncovered the least averaged 44.4%. The five models with the highest average cost per run passed 67.1%, compared with 50.2% for the five least expensive. These are group averages, not a claim that price alone caused the performance gap.
Each model attempted each scenario three times. Across models and scenarios, all three attempts passed in just 34.2% of cases. Visa also found that about 54% of failed runs still ended in a purchase: the recommendation either missed identified requirements or was made without uncovering all requirements. Visa says the benchmark and an interactive experience are publicly available through Visa Payments Frontier, where visitors can explore research, watch replays, and attempt the benchmark. The company plans to add shopping scenarios and product categories, then extend testing to checkout, payment, and order placement. It has shared the benchmark with leading model providers for feedback.
What the archive adds
New Runtime’s earlier coverage has examined other agent evaluations, including the Continual Learning Exam benchmark on progressive professional tasks and the Stanford Collaborative Intelligence Benchmark on agent teamwork. Visa adds a new, commerce-specific evaluation: it measures whether an agent elicits requirements and avoids a wrong purchase, not learning over time or collaboration.
Trend signal
This supports the pattern that AI-native work is shifting toward review and orchestration: AI-native organizations move toward review and orchestration. A shopping agent’s output is not useful merely because it produces a recommendation; someone must establish that the result meets the buyer’s needs and is repeatable. Visa’s benchmark makes that acceptance gap visible in a consumer task, though it does not establish how enterprises measure their own agents.
Why it matters
Shopping is a particularly clear test of an agent’s ability to ask before acting. A search system can return plausible products while missing a budget, compatibility requirement, or reason to buy nothing. Visa’s scoring rule captures that distinction, and the failed-run purchase rate shows why autonomous action needs more than a plausible answer.
For teams building commerce agents, the practical implication is to evaluate the whole decision path: what requirements were elicited, whether each was satisfied, whether the agent appropriately abstained, and whether the outcome holds across repeats. The benchmark’s public interface also gives developers and researchers a way to inspect attempts rather than relying only on aggregate scores. Visa’s findings do not prove that all shopping agents perform similarly in production, but they provide a structured baseline for asking what “ready” should mean.
Limits and uncertainty
Visa’s release describes the benchmark and reports aggregate results, but does not identify the 22 tested models or provide per-model scores in the announcement. It does not establish whether these scenario results predict performance in live commerce, where product catalogs, policies, user behavior, and fulfillment constraints can differ. The release says that Visa shared the benchmark with leading model providers for feedback, but does not describe the feedback received or independent replication. The benchmark is Visa’s own evaluation; the figures are not an independently audited industry-wide measurement.
Reader takeaway
A shopping agent should be judged on whether it discovers every material requirement, chooses correctly, and does so consistently—not simply on whether it returns a plausible product. Visa’s results put numbers on the gap: more than four in ten runs failed, and a failed run often still purchased something. That is a concrete reliability target for anyone building agents that can transact.
Open question
Will Visa publish model-level results and enough benchmark detail for independent teams to reproduce and compare the scores?