Shopify Products With More Reviews Come Back Higher When AI Agents Search

When an AI shopping agent asks Shopify's catalog for products, the ones with more reviews tend to come back higher on the list. In 78 of 91 of my searches that returned 50 products, more reviews leaned toward the top, and two thirds of the products in the top three spots had more reviews than the middle of their list. What the shopper told the agent about themselves, in the field Shopify built for it, made no measurable difference, including with Shopify's example search phrase. Filters such as gender and price, and the buyer's country with shipping set to match, did change the results. Shopify also documents a second way in: agents it invites can mix in promoted products and earn a commission on what shoppers buy. None of my searches returned one. That matters because the list is what an agent picks from.

I ran 1,305 searches against Shopify Catalog on 13 October 2026, at the access level every new agent starts on, with no key, and every search but one ran at least twice. Every search is in the data file: 1,305 searches and 20,410 product rows.

The same day, Shopify opened this search to any agent through Shop MCP, with no key, and let that agent build a cart and check out at Shopify stores without the shopper signing in. On 15 October I ran the searches again through Shop MCP, and the results were nearly identical to Shopify's command-line tool: on average 49.4 of the same 50 products, about what either one shares with itself when asked twice, and the top ten in the same order on 17 of 20 searches. What the shopper said still made no measurable difference.

Contents

• Shopify Catalog Gives AI Shopping Agents One Search Across Every Shopify Store
• Shop MCP Returns 49.4 Of The Same 50 Products As Shopify's Command-Line Tool
• A Shopify Catalog Search Returns Price, Stock And Reviews, With No Field For Ingredients Or Origin
• Shopify Catalog Returned A Review Count For 82% Of 5,567 Products And A SKU For None
• The Same Shopify Catalog Search Run Twice Returned 97% Of The Same Products
• Products With More Reviews Came Back Higher In 68 Of 80 Shopify Catalog Searches
• Shopify Offers AI Agents A 0.3% Commission To Blend Promoted Products Into Results
• Shopify's Documented Intent Example Returned The Same Ten Products With And Without The Intent
• Eight Shopper Statements In The Intent Field Changed Shopify Catalog Results About As Much As A Repeat Search
• A UK Or Portugal Buyer Shipping Home Shares About Half Of A US Buyer's Shopify Catalog Results
• Changing One Word In A Shopify Catalog Search Replaced Up To All Ten Top Results
• AI Shopping Agents Built On Shopify Catalog Should Say What They Could Not Check
• 1,305 Keyless Shopify Catalog Searches Ran On 13 October 2026, And 620 Through Shop MCP On 15 October

Shopify Catalog Gives AI Shopping Agents One Search Across Every Shopify Store

Shopify Catalog is a product search an AI agent can run across all Shopify stores at once. The agent sends a search phrase, some filters and some context about the buyer. It gets back products with prices, stock, ratings, seller details and a link into the store's own checkout. Shopify's documentation calls the all-stores version Global Catalog, and the single-store version Storefront Catalog.

The catalog speaks the Universal Commerce Protocol, UCP for short: a shared standard for how agents search for products, build carts and check out. UCP's website lists Google, Shopify, Etsy, Wayfair, Target, Walmart, Amazon, Microsoft, Meta, Salesforce and Stripe among the companies that co-developed it for shopping. When I say "a search" in this article, I mean UCP's search_catalog call against Shopify's catalog, sent with Shopify's command-line tool or, where I say so, through Shop MCP.

Shopify already decided which stores are in these results. As I wrote in September, Shopify enrols stores in its AI shopping channels by default. And Meta says its Muse agent launched with "access to the entire Shopify catalogue". None of Muse, ChatGPT or the other big agents has said publicly whether it queries this exact search, with these settings, at this access level. Shopify's merchant help page also says channels "often re-rank results based on their own logic", so the catalog's results are what an agent starts from, not necessarily what a shopper ends up seeing.

Getting in costs nothing. Shopify's catalog documentation lists the access for Global Catalog as "Agent profile (no API key needed)". Every search in this article ran that way. When I opened the Catalogs page in Shopify's Dev Dashboard, the option to identify my agent read "Agent identity, Locked, Verify to unlock", and verifying asks for a photo ID and a selfie. I did not submit them, so nothing here was tested as a verified agent.

How fast you can ask is not published. The documentation says "Catalog queries are subject to rate limits" and "Keyless catalog access doesn't support rate limit increases". It gives no numbers. Running about one search every 1.5 seconds, I was never blocked. A few calls returned a service error and went through on a retry.

Shop MCP Returns 49.4 Of The Same 50 Products As Shopify's Command-Line Tool

On 13 October, the day these tests ran, Shopify announced Shop MCP: "Shop MCP lets your agent or app search Shopify Catalog, build carts, and complete Shop Pay checkout through a single MCP connection." Searching needs no API key. Carts and checkout on Shopify stores work without the shopper signing in. For a store that does not run on Shopify, Shop's documentation says "the agent has to pay on the store's own checkout page": a shopper who has connected their Shop account approves the purchase, a saved card and a spending limit in Shop, and Shop gives the agent card details for that purchase. And for the store: "Orders placed through your agent appear in the merchant's Shopify admin like any other order."

So I ran the same 20 skincare searches from the review-count test both ways on 15 October, back to back: Shopify's command-line tool and Shop MCP, twice each, with the same settings and no sign-in. The two returned 49.4 of the same 50 products on average, about what each one shares with itself when asked twice (49.6 and 49.7). The top ten came back in the same order on 17 of 20 searches. More reviews leaned toward the top on 17 of 20 through both, against 18 of 20 on 13 October.

An agent searching through Shop MCP without a signed-in shopper gets the results this article describes. The intent tests, run again through Shop MCP, came out the same. What I did not test through it: a signed-in shopper, whose "saved preferences" Shopify says the search tools use, and the cart, checkout and payment steps. The changelog and the documentation do not say whether a store can opt out of agents checking out on it.

A Shopify Catalog Search Returns Price, Stock And Reviews, With No Field For Ingredients Or Origin

Each product a catalog search returns comes with a title, a description, a rating with its review count, one image, its options, and one variant with a price, stock status, seller, condition, a checkout link and a flag for whether the agent can check out without leaving to the store. A search can ask for up to 50 products at a time. Shopify also adds summaries it generates or enriches, in a metadata block: selling points, top features, technical specs.

The documented filters cover category, price, price tier within a category, stock, shop, condition, shipping origin and destination, minimum rating, and three attributes: Color, Size and Target gender. The context carries the buyer's country, region, postal code, language, currency and a free-text intent.

Ingredients and country of origin have no field, so a search can only match the words. Materials have a documented one, metadata.attributes ("Product attributes such as material, style, and occasion"), which never came back in my searches. Each of these ran twice, with the same result both times:

• "moisturizer without niacinamide or fragrance" returned ten moisturisers, and seven of them list niacinamide as an ingredient ("Formulated with a blend of ceramides, niacinamide…").
• "running shoes made in Europe" returned ten shoes, as a US buyer and as a Portugal buyer, and none states a European origin. Three or four of them contain the word "made".
• "something to carry my phone and keys on a 10k" returned no running belt or waist pack, for either buyer.
• "sunscreen safe for rosacea" returned two of ten that mention rosacea at all.

For skincare I checked where the ingredient list lives, on the 50 products of one search ("fragrance-free moisturizer for sensitive skin"). On the merchants' own pages, a full ingredient list was readable for 36 of the 50, without running any JavaScript. In the catalog's text for the same 50 products, 5 had one. The lists exist. Most merchants keep them in a tab or panel on the page, and the catalog does not carry the tab. The products where the list got through had it in the description itself.

Three more things an agent has to know before trusting what comes back:

• Search returns one variant per product, and it can be one marked unavailable. For 560 of 5,567 products (10%), the variant that came back was marked unavailable every time the product appeared, though most of these products list other sizes or colours. The top result for Shopify's road-shoes example phrase came back as a size 6.5 marked unavailable, while the same product listed twelve sizes in its options. An agent has to look the product up again before telling a shopper something is sold out.
• The match count is "an estimate". Shopify's Global Catalog documentation says so, and identical searches seconds apart returned different counts (221, then 230).
• Paging stops at 1,000 results, and the next-page cursor is a plain offset over a ranking that moves. Page two of "niacinamide serum" repeated a product from page one.

Shopify Catalog Returned A Review Count For 82% Of 5,567 Products And A SKU For None

Across the 5,567 different products my searches returned, 82% carried a rating with a review count, and none carried a SKU, barcode, category or tags.

Shopify's documentation marks several of these fields as "Inferred", meaning "generated or enriched by Shopify": the description, the options, the selling points, top features and technical specs, and the condition label. Its instruction to agents is to "Treat them as discovery and merchandising signals, not as merchant-authored source text." So the description an agent reads may be the store's words, Shopify's version of them, or a mix, and the response does not say which. Descriptions run a median of 93 words with the HTML removed, 102 products had fewer than 10, and 94% carry raw HTML in a field named plain. A typical one starts like this:

<p><strong>Bold, trail-inspired shoes with enhanced cushioning.</strong></p>Wherever your run takes you, channel the feeling of mountain vistas…

An agent that reads plain at its word gets the tags too. The UCP specification has optional fields for a product's categories, tags and page url, and for a variant's sku, barcodes ("Industry-standard product identifiers") and tags. Shopify's Global Catalog documentation shows some of them filled in, in an illustrative example. None of the 5,567 products came back with any of them, at any depth of the response (each variant does carry its own link). Optional means Shopify is allowed to leave them out. It also means an agent cannot match a product to a barcode or a store's own SKU from what search returns. Every product sends the buyer to the store's own checkout, tagged utm_source=shopify&utm_medium=catalog.

The Same Shopify Catalog Search Run Twice Returned 97% Of The Same Products

Across 688 pairs of identical searches, the second run returned on average 97% of the same products. Every one of the 1,305 searches of 13 October but one ran at least twice, with identical settings, the same day, seconds to a few hours apart. In 497 pairs (72%) every product came back, and in 445 (65%) they came back in the identical order.

Results that stable make the catalog behave like a search engine: ask the same thing and you get almost the same results. Two agents sending the same search with the same settings at this access level get almost the same list, so differences in the list the catalog returns start with what each agent sends. And when something in the tests below changes the results, or does not, the comparison is against a baseline that barely moves.

The exceptions were plain-English requests. "Easy 3-step skincare routine for men over 40" shared only two of its ten products with itself, and "shoes for running on cobblestones in Lisbon" six. Every other pair shared at least seven of ten.

If the catalog works like a search engine, the next questions are the ones a merchant asks of any search engine: how does it order the results, and how does a product get higher? Shopify does not publish the order. Its merchant help page names popularity, customer engagement, brand recognition and reviews as signals, without weights. From outside, the words of the search and the buyer's country, with shipping set to match, changed which products came back, and among them, the number of reviews lined up with a higher position in every category. The star rating and the length of the description barely lined up with position. That leaves a store three levers it can see: the words its buyers search with, in the title and description, the fields the filters read, and reviews. The review count is a correlation, and Shopify can change the order without notice.

Products With More Reviews Came Back Higher In 68 Of 80 Shopify Catalog Searches

Products with more reviews came back higher in 68 of 80 searches that returned 50 products, across skincare, home goods, running gear, supplements and clothing. Shopify publishes no ranking formula. Its merchant help page says search relevance "also depends on other signals such as popularity, customer engagement, and brand recognition", and that "Product reviews signal trust and popularity, which can influence how AI channels rank your products". It gives no weights. So I looked from outside: run a search for 50 products, and check which of the things visible in the response line up with a higher position. For each search's list of 50 products and each signal (the review count, the star rating, the length of the description and so on), I put the products in two orders, once by their position and once by that signal, and measured how closely the two orders match with a standard rank correlation (Spearman's). The score runs from 1, where the two orders are identical, through 0, where they have nothing to do with each other, to -1, where one is the other turned upside down. Above 0.1, products with more of that signal tended to come back higher, and I counted the list as leaning toward the top. Below -0.1, it leaned toward the bottom, and in between it was flat. That is a low bar for any one list, so the count shows which way lists lean, not how strongly. What carries the review-count finding is how consistent it is: of the 91 searches that returned 50 products, 78 leaned toward the top and 3 toward the bottom. Products with no review count were counted as zero.

The number of reviews leaned toward the top in every category. Across 80 searches in five categories, each returning 50 products:

Across every list in the tests, counting each distinct search once, the lean held in 78 of 91 searches that returned 50 products (it went the other way in 3) and in 176 of 276 searches that returned ten (the other way in 47). Short lists show it less clearly, which is what you would expect with fewer products to compare.

For every list, I took the product in the middle of that list by review count, and asked how often the products at each position had more reviews than it:

Two thirds of the products in the top three spots have more reviews than the middle of their list. At the bottom of a 50-product list, four in ten do. The lean is a tilt, not a rule: plenty of products with few reviews reached the top three, and the single most-reviewed product of a list reached the top three in 11 of the 91 searches that returned 50 products, about twice the rate chance would give, and in 115 of the 276 that returned ten, against about 86 by chance.

The star rating barely counted. It leaned toward the top in 5 to 9 searches per category and was flat in most of the rest. Description length did not line up with position in any category. Words from the search in the product title lined up a little with a higher position in skincare (12 of 20 searches) and not elsewhere.

Big brands have more reviews and more products, so this could be brand size showing up as review count. I checked it two ways. Holding constant how many of each category's results a seller took, the lean toward more reviews held in 10 of 14 clothing searches and in 12 to 14 of 15 in the other three categories. And comparing only products from the same seller in the same list, for sellers with three or more products in a list, better-reviewed products came back higher in 135 of 193 groups across clothing, home goods, running gear and supplements, and lower in 37. Within one seller, brand cannot explain it.

Where the review count comes from. Every number above is the product-level count, the one the catalog shows in search results. Each variant also carries a count, and the two rarely agree. I looked up 80 products with more than one colour, one colour at a time: the product's count matched the colours added up for none of them, matched a single colour's count for one, was larger than its colours' different counts added up for 45, and was smaller than at least one of its colours for 27. On 52 of the 80, several colours carried the identical count. Shopify's standard review fields explain how that can happen: "Product rating apps should write to this standard metafield", on the product and on each variant separately. Some apps also let a store pool reviews across products. Judge.me's Product Groups put "Reviews from products in the same group" on "all product pages within that group". From outside I cannot tell which applies to which product. So the count an agent sees is whatever the store's review app wrote, and it does not necessarily mean that many people reviewed that exact item.

The review-count finding is a correlation, measured from outside, on one day, at one access level. Review count may stand in for popularity or engagement, which Shopify says it uses and does not return. Proving cause needs a change on a store and a watch on its position, which I have not done.

Meta says it is adding retailers to make Muse "an even better shopper". The results an agent like that chooses from lean toward products with more reviews, and, as the intent sections below show, what you told the agent about yourself did not move them. Without a signed-in shopper, anything personal happens after that list arrives. And a new or small store with few reviews tends to start lower on it.

Shopify Offers AI Agents A 0.3% Commission To Blend Promoted Products Into Results

Shopify's promoted placement documentation describes a second way into the results, one that pays the agent: an agent "can opt into promoted placements and earn revenue on attributed purchases you drive". Promoted products are blended into the ranked results and carry a placement object whose type is "affiliate".

Shopify pays a 0.3% base commission "on the value of attributed purchases you drive through promoted placements", on every item in the order that is available through the catalog, not only the clicked one. The rate "might change before general availability". Merchants will be able to add their own commission "in the future", and Shopify's sample shows 1.5% combined. Attribution is last click within seven days, through shclid and shcgid parameters on the product links. In an approved agent's results, organic links carry those parameters too, and the page does not say whether purchases through them earn anything. Shopify tells agents to disclose the commission "where required by law" (its example: "We may earn a commission on purchases") and to label promoted products, and says it "doesn't prescribe the exact wording or placement".

Promoted placement is in Developer Preview, "limited and invite-led". Agents that are not approved get organic results only, and none of my searches returned a promoted product. So a shopper's agent may be choosing from results that lean toward products with more reviews and, if the agent is enrolled, include promoted products that earn it a commission. The labelling is up to the agent.

Shopify's Documented Intent Example Returned The Same Ten Products With And Without The Intent

On 13 October 2026, Shopify's documented example, "trail running shoes" with the intent "Customer runs marathons and needs road shoes", returned the same ten products in the same order with and without the intent. The intent field is where a shopper's own words are meant to go. The UCP specification defines it as "Background context describing buyer's intent (e.g., 'looking for a gift under $50', 'need something durable for outdoor use'). Informs relevance, recommendations, and personalization." Shopify's documentation lists it among the "buyer signals for relevance and localization", and its example search pairs "trail running shoes" with the intent "Customer runs marathons and needs road shoes". The instructions that ship with Shopify's UCP command-line tool describe the context as "soft signals that inform ranking", with an example pairing "marathon training shoes" and "daily trainer for marathon training".

Those examples show how to write a request. They are also the uses Shopify and the spec chose to show.

First, I sent the words alone. I sent Shopify's search phrase and intent with nothing else but shipping to the US, so the intent, which asks for road shoes, was the only difference between the two searches. It returned the exact same ten shoes, in the exact same order, as the search with no intent at all. I ran it twice with the intent and twice without, and got four identical lists. Four of the ten mentioned "road" either way. The command-line tool's example, sent the same way as words only, did the same, four times: same ten, same order.

Then I sent Shopify's full request, as published. Shopify's example also sets a stack of filters: shipping from the US or Canada, under $150, in stock, black, sizes 10 and 10.5, rated 4.5 or better with at least 10 reviews, low or medium price tier. Run exactly like that, the catalog did something it did nowhere else on 13 October: it alternated between two lists that differ by three products, with the intent and without it. Across 28 runs, the search without the intent came back as the first list 6 times and the second 8 times. With the intent, it came back as the first list 2 times and the second 12 times. The intent leans toward the second list, and on 28 runs that lean could still be chance. My first four runs happened to split cleanly by intent and looked like a three-product effect. The next 24 showed it was mostly the catalog switching lists.

The spec's examples made no difference either. "Looking for a gift under $50" on eight searches: seven result lists shared nine or ten of their ten products with no intent, and the number of products at $50 or under stayed the same on seven of the eight (perfume went from four to five). Two of the eight, "watch" and "headphones", had none at $50 or under with the intent and none without it. A price filter of $50 on the same searches brought their median prices to $49.99 and $40.50. "Need something durable for outdoor use" on six searches: all six result lists came back the same.

Through Shop MCP on 15 October, every example again made no difference, each run twice. Shopify's example and the command-line tool's example returned the same ten products with and without the intent, and Shopify's had "road" in four of them either way. "Looking for a gift under $50" left the number of products at $50 or under unchanged on all eight searches, while the $50 filter brought the watch results to a median of $44.99. Shopify's full request alternated between two lists again, not the same two as on 13 October: without the intent, 7 times one and 3 times the other, and with it, 5 and 5.

A "no difference" result can come from a mistake, so I checked the two likeliest ones. The first would be that the intent never reached Shopify. It did: the command-line tool can print the request instead of sending it, and the intent is in it. The second would be that the catalog ignores the whole part of the request the intent sits in, called the context. It does not: changing the buyer's country in that same context, along with where the order ships, changed about half the results (see the section on UK and Portugal buyers below). On a plain search, the intent itself changed nothing I could measure.

The UCP spec says when a business may ignore the intent and when it should use it. It "MAY ignore or down-rank" these signals "if inconsistent with higher-confidence signals (authenticated account, risk detection) or regulatory constraints". A keyless search sends neither an account nor anything else the intent could conflict with. It "SHOULD use these values when verified inputs (e.g., shipping address) are absent". If a keyless search counts as having no verified inputs, the spec says the catalog should use the intent. The spec does not say whether it counts.

What I could not test: whether the intent works for verified agents, with a saved catalog, or in the catalog's other calls.

Eight Shopper Statements In The Intent Field Changed Shopify Catalog Results About As Much As A Repeat Search

Across 20 searches, eight things a shopper might say about themselves kept 9.55 to 9.85 of ten products, against 9.75 for a plain repeat. The documented examples describe a shopping goal. I also tested what a shopper might say about themselves: 20 searches, ten for clothing and ten for general goods, each run with no intent, with no intent again, and with eight statements.

Every statement moved the results about as much as asking the same thing twice. The budget statement lowered the median price about as much as "money is no object" did: counting the products priced in dollars, the median of the per-search median prices was $142 with no intent, $134 with "tight budget" and $130 with "money no object". The luxury request came out cheapest, so the difference is noise. The under-$30 filter brought it to $23.

The filters are the control. They replaced eight or nine of the ten products on average and returned a full ten products every time, so the test could see a change when there was one. The Male filter was added on 16 October, with the plain search, a repeat and the Female filter run again the same day so the comparison is not mixed with three days of drift: the repeat kept 9.35 of 10 products, the Female filter 1.70 and the Male filter 1.05, and the Female and Male results shared 0.15 products on average. An agent that wants different results for a woman on a budget has to turn her sentence into Target gender and a price limit itself.

Through Shop MCP on 15 October, with every search run twice, the statements again moved the results about as much as a repeat did. A plain repeat kept 9.95 of 10 products, the eight statements 9.70 to 9.85, the gender filter 1.75 and the under-$30 filter 0.85. "I am on a tight budget" and "Money is no object" both left the median of the per-search median prices at $136.25, exactly where it was with no intent. The under-$30 filter took it to $24.99.

Shopify's example combines intent with filters, so I ran the same 20 searches again with a filter set (in stock, shipping to the US, under $150, rated 4 or better with at least 10 reviews), with no intent and with three of the statements, each twice. Against the no-intent list, a plain repeat shared 9.85 of 10 products, "I am a woman" 9.8, "Money is no object" 9.7 and "I am on a tight budget" 9.5. The budget statement moved slightly more than the others, but nothing came close to what a filter does.

A UK Or Portugal Buyer Shipping Home Shares About Half Of A US Buyer's Shopify Catalog Results

The buyer's country, with shipping set to match, changed the results where the intent did not. I ran 30 searches, 15 for clothing and 15 for home goods, as a US buyer, a US buyer again, a UK buyer and a Portugal buyer, each shipping to their own country.

Home goods moved most. "Scented candle", "doormat" and "storage basket" shared nothing at all between the US results and either European set. The country also does not fully settle the currency: UK and Portugal buyers saw their own currency on 7.6 and 7.7 of ten products, so two or three products in European results still came priced in something else, despite shipping set to the buyer's country.

Changing One Word In A Shopify Catalog Search Replaced Up To All Ten Top Results

Changing one word in a search replaced up to all ten of the top ten products, while a repeat of the same search kept all ten.

"My skin gets red and stings with most products, what can I use?" and "what can I use if most skincare makes my face red and sting" shared one of their ten products. Each of them, run again, returned the same ten as the first time. The two sentences describe the same person with the same need.

One-word edits to "fragrance-free moisturizer for sensitive skin", 50 products each, compared on the top ten:

Words that describe the product change the results. "Cheap" and "best" each replaced three of the top ten, as much as switching to the UK spelling, against none on a repeat. This was one skincare search. "Cheap" moved the median price of the top ten from $41.50 to $37. "Unscented" and "fragrance-free" mean the same thing to a shopper, and the catalog treated them as different searches.

AI Shopping Agents Built On Shopify Catalog Should Say What They Could Not Check

It is too early to build a shopping agent on what the documentation says. Build on what the catalog does on the day you test it, and test again before relying on it: the documented intent examples made no measurable difference, and the documented optional response fields never came back.

From these results, an agent on Shopify Catalog that wants to serve the person it shops for has to do three things the catalog does not do for it:

• Translate the person into filters and product words. "I am a woman on a tight budget" has to become Target gender: Female and a price limit. Putting it in intent made no measurable difference in my tests.
• Search more than one way. One word can swap every result, so a single phrasing shows a single set of products. Synonyms the shopper would treat as the same ("unscented", "fragrance-free") are different searches.
• Say what it could not check. Ingredients, origin and materials have no field in the response. A list that leans toward more reviews is not a recommendation, and a shopper deserves to know which of the agent's claims came from the catalog, which from the store's own page, and which it could not verify at all.

For a store, the levers I could see from outside are the ones the catalog reads: words in the title and description that match how people search, complete product data that the filters can use, and reviews. The fact a shopper cares about most, if it lives in a tab on the product page, is probably not in the catalog response. The stores whose ingredient lists got through had put them in the description. The protocols around the catalog, and what they ask of a small store's stack, are in an earlier piece.

1,305 Keyless Shopify Catalog Searches Ran On 13 October 2026, And 620 Through Shop MCP On 15 October

All 1,305 searches in this article ran on 13 October 2026, at the keyless level every new agent starts on, mostly as a US buyer, with Shopify's command-line tool (ucp 0.9.0). Every search but one ran at least twice with identical settings, and every raw response is saved.

The Shop MCP check, 15 October. 80 more searches: the 20 queries of the review-count test, each run four times in turn (command-line tool, Shop MCP, command-line tool, Shop MCP), 1.5 seconds apart, 50 results each, as a US buyer shipping to the US, with the same settings on both and without the language field the 13 October runs sent. No sign-in, no API key. Then 540 searches repeating the intent tests through Shop MCP alone, top 10, every condition twice, with language: en included as on 13 October. Two calls returned a catalog error and went through on a retry. Every request and response is saved. These 620 are in a second file, shop-mcp-tests-2026-10-15.csv: 620 searches and 9,400 product rows, the same columns as the file below plus channel (the command-line tool or Shop MCP). Shop MCP does not return the native-checkout flag, so that column is blank on its rows.

The Male filter, 16 October. 80 more searches with the command-line tool: the 20 queries of the intent test, each with no filter, no filter again, Target gender Female and Target gender Male, top 10, as a US buyer shipping to the US. The plain search ran twice and each filter once. Every request and response is saved, and these 80 are not in either data file.

The data. shopify-catalog-tests-2026-10.csv has 1,305 searches and one row for every product each of them returned, 20,410 rows: the test, the exact query, the buyer's country, language and currency, the intent if one was sent, the filters, how many products were asked for, the match count Shopify reported, the position, Shopify's product and variant IDs, the price and currency, the regular price where a sale price was shown, the rating and review count, stock and the native checkout flag. A repeat_of column links every repeat to the first search with identical settings: 688 links, one for each repeat. Titles, sellers and links are left out so no store is named. The IDs are Shopify's own, so any product can be looked up. Three queries that typed a brand name show it as "[brand]". One 10-product field check has no saved query and a single run.

What these tests do not cover:

• Verified agents. The spec allows a business to ignore context signals that conflict with more reliable ones, and verified agents may be handled differently. Verifying needed a photo ID and a selfie, which I did not submit.
• Saved catalogs and the other calls. intent also appears in the catalog's lookup and product calls. I tested search only.
• Personalized search. Shopify documents "personalized search_catalog results for a signed-in Shop customer", through a buyer-linked token. I did not test it.
• Cause. The review-count finding is a correlation. I own no store in the catalog to change and watch.
• Time. These results are what the catalog did on one day. Shopify runs the catalog and can change it without notice. Anyone relying on these numbers should run the same searches again.
• The big agents. I do not know whether Muse, ChatGPT or any other agent using Shopify's catalog sends intent, or uses a higher access level where it might work.

On 13 October I sent Shopify's press team four questions: whether context.intent is meant to affect ranking at the keyless level or only for verified agents, whether keyless search has a published rate limit, whether leaving out the SKU, barcode, category, tag and product-page fields is by design, and where a product's review count comes from. Shopify had not replied by 16 October, when this was published.