Shopify Catalog Search Tests, October 2026

Method

All 1,305 searches in the article ran on 13 October 2026, at the keyless level every new agent starts on, mostly as a US buyer, with Shopify's command-line tool (ucp 0.9.0). Every search but one ran at least twice with identical settings, and every raw response is saved.

For each search's list of 50 products and each signal (the review count, the star rating, the length of the description and so on), I put the products in two orders, once by their position and once by that signal, and measured how closely the two orders match with a standard rank correlation (Spearman's). Above 0.1, products with more of that signal tended to come back higher, and I counted the list as leaning toward the top. Below -0.1, it leaned toward the bottom, and in between it was flat. Products with no review count were counted as zero.

The Shop MCP check, 15 October. 80 more searches: the 20 queries of the review-count test, each run four times in turn (command-line tool, Shop MCP, command-line tool, Shop MCP), 1.5 seconds apart, 50 results each, as a US buyer shipping to the US, with the same settings on both and without the language field the 13 October runs sent. No sign-in, no API key. Then 540 searches repeating the intent tests through Shop MCP alone, top 10, every condition twice, with language: en included as on 13 October.

The Male filter, 16 October. 80 more searches with the command-line tool: the 20 queries of the intent test, each with no filter, no filter again, Target gender Female and Target gender Male, top 10, as a US buyer shipping to the US. The plain search ran twice and each filter once. Every request and response is saved, and these 80 are not in either data file.

The data. Titles, sellers and links are left out so no store is named. The IDs are Shopify's own, so any product can be looked up. Three queries that typed a brand name show it as "[brand]". A repeat_of column links every repeat to the first search with identical settings: 688 links, one for each repeat.

Results

Across 80 searches in five categories, each returning 50 products:

Across every list in the tests, counting each distinct search once, the lean held in 78 of 91 searches that returned 50 products (it went the other way in 3) and in 176 of 276 searches that returned ten (the other way in 47).

Across 688 pairs of identical searches, the second run returned on average 97% of the same products. Shop MCP and Shopify's command-line tool returned 49.4 of the same 50 products on average, about what each one shares with itself when asked twice (49.6 and 49.7).

Shopify's documented search phrase, "trail running shoes" with the intent "Customer runs marathons and needs road shoes", returned the same ten products in the same order with and without the intent.

What these tests do not cover

• Verified agents. The spec allows a business to ignore context signals that conflict with more reliable ones, and verified agents may be handled differently. Verifying needed a photo ID and a selfie, which I did not submit.
• Saved catalogs and the other calls. intent also appears in the catalog's lookup and product calls. I tested search only.
• Personalized search. Shopify documents "personalized search_catalog results for a signed-in Shop customer", through a buyer-linked token. I did not test it.
• Cause. The review-count finding is a correlation. I own no store in the catalog to change and watch.
• Time. These results are what the catalog did on one day. Shopify runs the catalog and can change it without notice. Anyone relying on these numbers should run the same searches again.
• The big agents. I do not know whether Muse, ChatGPT or any other agent using Shopify's catalog sends intent, or uses a higher access level where it might work.