Common questions and situations when running a theme experiment.

This happens when the challenger theme is still sitting in your theme library. Shopify remembers which theme it last served each visitor, and stopping the experiment in Cooee does not clear that.

Publish the challenger as your live theme, or delete it. Either resolves it for everyone. Duplicate it first if you want to keep the design work.

Visitors assigned to the variant arm briefly see ?preview_theme_id=... on the first page they land on. Shopify removes it once the theme is served, and it does not persist as they browse.

It is not indexed by search engines and does not affect your SEO. You may see a small number of sessions attributed to that URL in analytics tools that key on the full address — this is expected and does not affect the experiment.

No. Cooee suppresses the preview bar, so the challenger theme looks like a normal storefront. If you do see the bar, you are previewing the theme from Shopify Admin rather than viewing it as an assigned visitor.

Assignment is per browser. The same person on a different device, a different browser, or a private window is treated as a new visitor and assigned independently.

This is normal for browser-based testing everywhere and does not distort your results, since it applies evenly to both arms. It does mean you cannot reliably force yourself into a particular arm — open private windows until you land in the one you want to inspect.

Expect some drift, especially early on. Assignment is decided per visitor, so a 50/50 split evens out over volume rather than matching exactly at every point.

Also note the variant arm normally shows around 3–5% fewer tracked sessions for a structural reason explained in Reading theme experiment results. A gap far larger than that, or one that keeps widening, is worth raising with us.

The challenger is a separate theme, so an edit to one is not reflected in the other. Editing either theme mid-test introduces a second difference between the arms and muddies the result.

If the change was cosmetic, note the date and keep going. If it touched anything on the path to purchase, restart the experiment so both arms are compared over the same period.

Below roughly 500 daily unique visitors, conversion-level results need four to six weeks to settle. Testing a bolder change helps — small refinements need far more traffic to detect than obvious ones.

If a test runs that long without separating, treat that as its own answer: the change is not moving the needle enough to matter.