Shopify ShopGym: a sandbox for testing shopping agents

Shopify has detailed ShopGym, an open-source framework for testing AI shopping agents in repeatable sandbox stores. It gives developers a way to compare agents on the same catalogue, navigation and policies, without a live shop changing between test runs.

The Shopify Engineering article published on 1 October explains how the framework builds test stores from anonymised descriptions of real storefronts, then creates shopping tasks for agents to attempt. Its immediate audience is developers and researchers building shopping agents.

What ShopGym tests

The research project site groups tasks around exact-product search, finding alternatives, browsing collections, applying filters, locating shipping and returns information, and longer shopping journeys.

Those longer journeys can combine filtering, sorting, inspecting a product, checking a policy and adding an item to the cart. Testing each step in isolation can miss failures that appear only when the agent must carry information across several pages.

ShopArena builds the sandbox environment from a shop specification. ShopGuru generates tasks grounded in that environment and checks that referenced products, collections and filter options exist. Shopify says the specifications omit brand names, product names, people’s names and absolute URLs before the generation stage.

What the evidence supports

Shopify reports validation across 224 tasks and six sandbox shops: three using synthetic data and three using real data. Agent results in synthetic shops correlated with results on the live stores they mirrored, particularly for multi-step tasks.

The company also reports gaps on some focused tasks. The simulated stores have fewer connections between interface states because they omit external links, marketing pages and other auxiliary paths. That makes the result useful evidence for a testing approach, rather than proof that the sandbox reproduces every live-store difficulty.

The project site currently showcases ten sandbox shops. That larger demonstration collection should not be confused with the six-shop validation described in the engineering article.

Where developers can start

The repository README describes an explore → generate → serve workflow, followed by benchmark generation and evaluation. It lists Python 3.12 or later, Node 20 or later, pnpm 9 or later, and a supported coding-agent runtime among its requirements. Check the current instructions and model-provider requirements before setting it up; open-source code does not make model usage free.

A practical first experiment is to keep the store and tasks fixed, change one part of your agent, and compare both task success and the steps where it fails. Treat successful sandbox runs as a reason to investigate further, not permission to skip testing the live integration.

What merchants can take from it

Store owners do not need to install a research framework to act on the task list. As an editorial takeaway, check that shoppers can find a specific item, use consistent filters, locate delivery and returns information, select the right variant and change cart quantities without ambiguity.

These are sensible usability checks; Shopify’s research does not establish that passing them makes a store eligible for any particular AI channel. For the separate questions of channel settings, catalogue information and measurement, see our Shopify agentic commerce checklist.

COLM 2026 workshop update

On 5 October, Shopify Engineering shared ShopGym and SimPersona’s presentation at the COLM 2026 Workshop on Learning from Situated and Embodied Interaction. This is a research update; it does not change the framework’s merchant eligibility or establish sales uplift.

Read Shopify Engineering’s original post.