Visa Just Graded AI Shoppers — And They Failed 42.8% of the Time

The best models still get it wrong about one time in five. The real problem is not checkout. It is discovery — and it arrives just in time for Black Friday.
TL;DR
- Visa tested 22 AI models from 12 labs on 15 shopping challenges. Overall, 57.2% of runs passed and 42.8% failed. - 54% of failed runs still ended in a purchase, so 23.1% of all runs ended with a purchase that missed a requirement (Visa reports this figure on its benchmark site). - The five models that uncovered the most hidden buying factors passed 70.2% of the time; the five that uncovered the least, 44.4%. - Shoppers use AI for discovery but hesitate to hand over their wallet: in Talkdesk's survey, 74% plan to use AI assistants for holiday shopping, but only 15% would let an agent pay even with a chance to review first. - The same week, Google brought UCP-powered checkout to Canada and Amazon opened Shop Direct Merchant Workspace. Capability is racing ahead of reliability.
Illustration generated for Reviuws. The hard part is not finding a TV — it is finding out which TV the shopper actually needs.
Why this benchmark matters now
On November 5, 2026, Visa graded AI models not on how fluently they talk about shopping, but on whether they can shop. The Visa Agentic Commerce Benchmark requires an assistant both to uncover the shopper's task-defined requirements and to make the correct buy or no-buy decision. If nothing fits, it should not buy, and it must establish enough information to justify its choice — a lucky guess does not pass.
That is a deliberately harsh standard, and the right one. In real shopping, a confident wrong answer creates a return, a refund, a dispute or a lost customer.
The timing matters. Black Friday is November 27, 2026. Adobe projects AI traffic to U.S. retail sites to rise 159% year over year on Thanksgiving, 95% on Black Friday and 88% on Cyber Monday. The question is not whether agents will be in the store this holiday, but whether they know what to buy when they get there.
What Visa tested
Each test, or episode, is a conversation between two AI models: one plays the shopper, the other is the assistant being tested. The shopper states a few preferences and holds the rest back; hidden requirements surface only if the assistant asks or acts in a way that reveals them. Every model searches the same simulated market and is scored against an answer key it never sees. Runs that hit the tool-call limit or reply in the wrong format fail.
In the one public episode, a shopper needs a 65-inch TV with at least 30-day returns; questioning reveals a budget and compatibility needs for a games console, including Dolby Vision. Visa lets the public try this episode; the others remain private.
Scale: 22 models × 15 episodes × 3 runs = 990 runs (our multiplication of Visa's inputs).
The headline numbers
| Result | Figure | | --- | --- | | Runs that passed | 57.2% | | Runs that failed | 42.8% | | Best model pass rate | 82.2% | | Worst model pass rate | 28.9% | | Failed runs that still ended in a purchase | 54.0% | | All runs ending in a purchase that did not pass | 23.1% | | Other outcomes (declined despite a fit, or no decision) | 19.7% |
Almost one in four times an AI shopper was asked to shop, it bought something that missed a requirement. That is not a checkout failure — Visa has not yet tested checkout, payment or order placement. The failure happens earlier: in understanding what the shopper wants.
Discovery is the bottleneck — asking beats knowing
| Group | Pass rate | | --- | --- | | 5 models that uncovered the most hidden buying factors | 70.2% | | 5 models that uncovered the least | 44.4% | | 5 most expensive models per run | 67.1% | | 5 least expensive models per run | 50.2% |
The 25.8-point discovery gap is larger than the 16.9-point price gap. For anyone building or buying an agent, that reframes the question. Do not just ask "How smart is this model?" Ask "How does it behave when the user has not told it everything?" The best shopping agent is the one that asks the follow-up question a good salesperson would ask.
Illustration generated for Reviuws. Asking vs. assuming: the same confidence, without the questions, delivers the wrong item.
Who topped the test — and what the tie hides
Visa's press release does not rank individual models, but its benchmark site and Fortune's coverage report a tie at the top: Gemini 4 Argon and Muse Wonder each passed 82.2% of runs, a Claude model placed third at about 76%, and an OpenAI model finished sixth at about 64%. Visa itself cautions that no model clearly leads: each rate has a wide 95% confidence interval, and the next eleven models' intervals overlap the leaders'.
The practical lesson: name the exact model, not the brand. Results vary widely across models from the same lab.
Shoppers delegate discovery, not payment
Talkdesk surveyed 1,048 U.S. holiday shoppers and 312 retail leaders. 74% of shoppers plan to use AI assistants this season and 49% would consider an unfamiliar brand an AI recommends — but only 15% would let an agent purchase with their payment details, even with a review step. Meanwhile 89% of retailers are preparing for agents that shop on a customer's behalf. Google separately reports that 59% of Canadian holiday shoppers plan to use an AI tool to help them shop.
Discovery is delegated; payment is not. Given the error rates above, a human checkpoint before money moves is not friction. It is insurance.
The same week, the pipes got bigger
Google, November 4 (Canada): UCP-powered checkout launched for Canadian shoppers in AI Mode in Search and the Gemini app, with lululemon, Shopify merchants, Shoppers Drug Mart and Simons. Purchases go through Google Pay, and the retailer remains the seller of record.
Amazon, November 4 (U.S.): Amazon launched Shop Direct Merchant Workspace, a free self-service portal connecting eligible merchants' stores to Amazon and Alexa for Shopping, using feeds from providers such as Feedonomics, Salsify, CEDCommerce and VersaFeed. Amazon says tens of millions of customers have discovered hundreds of millions of products from other stores through Shop Direct.
The lesson is not "slow down." It is "separate discovery from delegation." Protocols solve the plumbing — finding products, building carts and proving consent. Visa's benchmark tests something else: whether the agent knows what to put in the cart. Plumbing without judgment still delivers the wrong TV, faster.
What this means — by role
If you are a shopper
- Use AI for research, not blind buying. Verify the things agents miss: all-in budget, must-have specs, return window and compatibility. - Keep a review step before payment. - If an agent says "nothing fits," do one manual search before you give up — cautious agents sometimes walk away from real fits. - Write your requirements down before you prompt. If you don't state them, you're testing the agent's mind-reading, not its shopping.
If you are a merchant or brand
- Complete, explicit product data is now a sales channel. Missing attributes — return window, HDR format, refresh rate — can lead an agent to a wrong assumption or to a competitor's complete listing. - Publish total price, returns and compatibility plainly on the product page, with structured data and synced feeds. - Disclose when AI is part of the shopping experience. - Plan returns capacity for agent-driven purchases.
If you are building an agent or choosing a model
- Evaluate discovery, not just the final answer: did the agent establish every requirement? - Run each task several times. Single-run averages hide inconsistency. - Choose the exact model and compare its price, context window and reviews before deploying. - Design for the failure you can afford: bias toward verifiable "no fit" answers with human handoff if wrong purchases are costly; toward one more question if lost sales are. - Keep a human checkpoint for payment. Proof of consent shows the user said yes; it does not prove the agent understood what they said yes to.
What Visa did not test
- Checkout, payment and order placement. Visa says it plans to extend testing there. Today's results cover discovery and the purchase decision only. - Real people. The market is simulated and the shopper is an AI model; real shoppers are messier. - Public replication. Only one episode is public. - Incentives. Visa runs a card network with its own agentic commerce products. The data is useful; the right response is to ask for more episodes, public replays and, eventually, real checkout data.
The Reviuws take
The most important model evaluation for shopping is not "which model writes the best product description?" It is "which model asks the best question, admits when it does not know, and says no when nothing fits?"
Before Black Friday: pick the exact model using Reviuws' directory, benchmarks and cost tools; test it on your own top products with hidden requirements, several times each; and keep a human in the loop for payment. The agents that win this holiday will not be the ones that buy fastest. They will be the ones that buy right — and know when not to buy at all.
Have you run a shopping agent in production? Share your experience in a review on Reviuws.
Sources
- [Visa: Agentic Commerce Benchmark press release](https://usa.visa.com/about-visa/newsroom/press-releases.releaseId.22936.html) — November 5, 2026. - [Visa Payments Frontier: Agentic Commerce Benchmark results](https://www.visapaymentsfrontier.io/articles/agentic-commerce-benchmark/agentic-commerce-benchmark.html) - [Fortune: Visa tested AI shopping agents](https://fortune.com/2026/11/05/ai-agents-shopping-ranking-best-agent/) — November 5, 2026. - [Adobe: AI traffic drives higher revenue per visit](https://business.adobe.com/blog/ai-traffic-higher-revenue-per-visit-retail-holiday) — October 29, 2026. - [Talkdesk: 2026 AI Holiday Shopping Survey](https://www.talkdesk.com/news-and-press/press-releases/2026-holiday-shopping-survey/) — November 4, 2026. - [Google Canada: UCP-powered agentic shopping](https://blog.google/intl/en-ca/products/explore-get-answers/a-new-way-to-shop-introducing-a-more-intuitive-agentic-shopping-experience-for-consumers-and-retailers-in-canada/) — November 4, 2026. - [Amazon: Shop Direct Merchant Workspace](https://www.aboutamazon.com/news/retail/amazon-shop-direct-merchant-workspace) — November 4, 2026.
Editorial review, November 13, 2026: headline results were checked against Visa's release and benchmark site, Fortune, Adobe, Talkdesk, Google and Amazon. Removed or not confirmed in this review: per-model costs and cost-per-pass ratios, the pass^2/pass^3 consistency figures, the model-label table beyond the confirmed leaders, Visa Trust Index, YouGov/Blue Yonder and additional Talkdesk sub-statistics, the provider directory count, and the historical checkout-launch timeline. Illustrations were supplied with the article and are AI-generated.
Related reading
- [The Agent Trust Gap](https://reviuws.com/blog/agent-trust-gap-security-shopping-professional-work) - [The Agentic Enterprise's Governance Debt](https://reviuws.com/blog/agentic-enterprise-governance-debt-ai-agents-2026)
Referenced sources
- An AI shopping assistant in an electronics aisle
- Illustration contrasting an agent that asks questions with one that assumes
- Visa: Agentic Commerce Benchmark press release
- Visa Payments Frontier: Agentic Commerce Benchmark results
- Fortune: Visa tested AI shopping agents
- Adobe: AI traffic drives higher revenue per visit
- Talkdesk: 2026 AI Holiday Shopping Survey
- Google Canada: UCP-powered agentic shopping
- Amazon: Shop Direct Merchant Workspace
- The Agent Trust Gap
- The Agentic Enterprise's Governance Debt