Method
I ran each search once through Shop MCP, as an anonymous US buyer, and saved the top 20 results. Every call to a model got that same saved list, in the catalog's order (except in the test where I reversed it), so the products never changed between models or between runs. Eight skincare searches were fetched on 15 October. Six more, backpack, coffee beans, running shoes, water bottle, wireless earbuds and yoga mat, were fetched on 24 October. I also fetched "scented candle" and "dog food" and dropped them, because one store filled all 20 candle results and three stores filled the dog food list.
For each product the model saw the title, the store, the price, the star rating and review count, the catalog's one-line summary and its first three listed features, numbered 1 to 20. It saw nothing else. The instruction, word for word:
You are a shopping agent. A person has asked you to buy something for them. You will see the products the Shopify catalog returned for their request. Choose the one product you would buy for them. Reply with JSON only, no other text: {"pick": <the product number>, "reason": "<one sentence>"}
The models were Claude Haiku 4.5, Sonnet 5 and Opus 5, run through Claude Code with default settings, no tools and no web access. Every pick was a fresh call with no memory of the others.
There were three sets of runs, 1,980 picks in all:
- Six product searches with only the search words as the request: 50 picks per model per search, 900 picks.
- One sunscreen search, asked two ways, first "mineral sunscreen SPF 50", then "mineral sunscreen SPF 50, under $25, for oily skin": 100 picks per model each way, 600 picks.
- Eight skincare searches with the list shown in the catalog's order and then reversed: 10 picks per model each way per search, 480 picks.
Results
Each model's most-picked product out of 50, on the six product searches:
| Search | Haiku 4.5 | Sonnet 5 | Opus 5 |
|---|---|---|---|
| Backpack | Roll Top 20L, 35 | Roll Top 20L, 50 | Everyday Backpack 20L, 50 |
| Coffee beans | EASE Dark Roast, 25 | EASE Dark Roast, 50 | FLOW Medium Roast, 48 |
| Running shoes | Brooks Ghost Max 3, 33 | adidas Adizero Evo, 44 | Brooks Ghost Max 3, 49 |
| Water bottle | nobl Steel Bottle, 45 | nobl Steel Bottle, 39 | Owala FreeSip, 50 |
| Wireless earbuds | soundcore Sport X20, 24 | soundcore Liberty 4 NC, 50 | soundcore Liberty 4 NC, 50 |
| Yoga mat | POPFLEX Suede Mat, 40 | Liforme Classic, 50 | Liforme Classic, 19 |
On the six product searches, the three models' most-picked product matched in none. Across the six searches, Haiku picked the product Shopify's catalog ranked first 8% of the time, Sonnet never, and Opus 20%.
In 222 of its 300 picks on the six product searches, Claude Opus 5's reason called its choice "safe" or "safest". Haiku did that once in 300 and Sonnet 11 times.
Across all eight skincare searches, out of 80 per version, the catalog's top product was picked with the list in catalog order, then reversed: Haiku went from 28 to 11, Sonnet from 15 to 11 and Opus from 24 to 10. In 480 reasons, not one mentioned where a product sat in the list.
The catalog's top-ranked sunscreen cost $48. With the plain request Haiku picked it 100 times out of 100, Sonnet 90 and Opus 33. With the budget and skin type in the request, all three picked it zero times.
What this test does not cover
- Three Claude models only. Haiku 4.5, Sonnet 5 and Opus 5 are all made by Anthropic. Other companies' models may pick differently, and this test does not show how.
- Not real shopping apps. Shopping features in apps such as ChatGPT, Gemini or Claude Duet run their own setup around the model, which this test leaves out. This test isolates the model.
- One instruction, no shopper context. No preferences, no history, and in all but one test, no stated need.
- Fixed lists from two days. Each list is what Shopify's catalog returned to an anonymous US buyer on 15 or 24 October 2026. A signed-in shopper may see different products.
- "Safe" was counted by searching each reason for the words "safe" or "safest". Every prompt and every reply is saved.
- Nothing was bought. The models picked. No order was placed.