Theme A/B Testing

Theme A/B Testing lets you compare two Shopify themes against each other — your current live theme (control) vs a challenger theme — to see which drives better business outcomes. Visitors are assigned to one arm automatically and consistently, and the correct theme loads silently without redirecting them to a different URL.

Common use cases

Testing a redesigned homepage or PDP experience before a full rollout
Comparing a conversion-optimised theme against your existing one
Validating a seasonal or campaign theme with real traffic before switching
Testing different layouts or trust-signal placements across the whole site

How it works

Every visitor who lands on your store goes through the following steps in under a second:

Visitor ID — Cooee reads a unique visitor ID stored in localStorage and a first-party cookie (cooee_vid). If no ID exists, one is generated. The cookie is the source of truth and survives browser data clears on Safari.
Bot exclusion — Automated traffic (crawlers, headless browsers, Playwright/Puppeteer) is detected and excluded from assignment. Bot sessions are never counted in experiment results.
Assignment — Cooee checks whether an active experiment exists for this store. If so, the visitor is assigned to the control or variant arm using a deterministic algorithm (MurmurHash3 on the visitor ID + experiment ID). The result is stored in localStorage so returning visitors always get the same arm — this "stickiness" is essential for clean data.
Redirect to challenger theme — Variant arm visitors are served the challenger theme using Shopify's built-in theme preview mechanism (?preview_theme_id=<id>&pb=0). The pb=0 parameter suppresses the Shopify preview bar so the experience looks identical to production.
Anti-flicker protection — The page is briefly hidden until the correct theme is confirmed to be loading, then revealed. This prevents a flash of the wrong theme. A hard timeout of 1,500 ms ensures the page is never hidden indefinitely.
Assignment recorded — A lightweight background request records the assignment server-side. This enables per-arm impression counts and downstream conversion metrics in your Cooee dashboard.

Before you start

Prerequisites

Cooee Shopify App installed with the advanced theme integration enabled (Enable advanced Cooee integration)
At least one additional theme in your Shopify theme library (Admin → Online Store → Themes) — this will be your challenger theme
The challenger theme fully configured and tested (it should look correct before the experiment runs — visitors assigned to it will see it as-is)

Setting up a Theme A/B Test

Step 1 — Create the experiment

In Cooee, navigate to Experiments and click New Experiment
Select Theme A/B Testing
Enter a clear Name (e.g. "Q3 Homepage Redesign")
Write a Hypothesis — state what you expect to happen and why:

"If we show the new minimalist theme, add-to-cart rate will increase because the simplified product layout reduces decision friction."

Choose a Primary Metric — the single KPI that defines success for this test (e.g. Add-to-Cart Rate, Conversion Rate, Revenue per Visitor)
Click Create

Step 2 — Configure the arms

Your experiment is created with two arms:

Arm	Role	Theme
Control	Baseline experience	Your current live theme (set automatically)
Variant A	Challenger experience	You select this

To set up the variant:

Click on Variant A
Select the challenger theme from your Shopify theme library
Optionally rename the arm to something descriptive (e.g. "Minimalist v2")

Traffic split: The default is 50% control / 50% variant. You can adjust this — for example 80/20 if you want a cautious initial rollout. All arms must sum to exactly 100%.

Tip: Allocating less than 30% traffic to a variant will significantly slow down how quickly you reach statistical significance. We recommend 50/50 splits unless you have a strong reason for an asymmetric split.

Step 3 — Publish

Review the configuration and click Publish
Cooee immediately starts routing new visitors to the correct theme
Returning visitors who already have an assignment are re-served their original arm

Reading your results

Open the experiment in Cooee to see live metrics broken down by arm. All metrics are per unique visitor:

Metric	What it measures
Impressions	Unique visitors assigned to this arm who started a real tracked session
Add to Cart Rate	% of visitors who added at least one product to cart
Conversion Rate (CVR)	% of visitors who completed a purchase
Revenue per Visitor (RPV)	Total revenue ÷ total visitors — the most complete single performance metric
Average Order Value (AOV)	Average order size among converting visitors
Session Duration	Average time spent on site per session
Bounce Rate	% of visitors who left without interacting
Pure Revenue	Revenue attributed only to visitors who converted within the same session they were first assigned

Recommendation: Always check guardrail metrics (CVR, RPV, Bounce Rate) even if your primary metric improves. A winning variant on ATC but a significant CVR drop is not a real win.

Results are updated daily. Allow at least 7–14 days (and ideally 2 full business cycles) before drawing conclusions, to account for day-of-week effects.

Automatic experiment safeguards

Cooee monitors your Shopify theme library and automatically responds to changes:

Event	Cooee response
Variant theme is published as the live theme on Shopify	Experiment is stopped automatically — publishing the variant would break the A/B assignment
Variant theme is deleted from Shopify	Experiment is stopped automatically
Control (live) theme is deleted	Experiment is paused automatically and your team is notified

Stopping the experiment and calling a winner

When you're ready to conclude:

Open the experiment and click Stop
Review results across primary and guardrail metrics
Make your rollout decision and take the corresponding action in Shopify Admin (Online Store → Themes):

Decision	Action in Shopify Admin
Variant wins	Publish the challenger theme as your new live theme
Control wins	Delete the challenger/draft theme
Inconclusive	Extend the runtime — or, if stopping entirely, delete the challenger/draft theme

Why is deleting the draft theme important when the variant doesn't win? When Shopify serves a visitor the variant theme via the preview mechanism, it stores a reference to that theme in a persistent browser cookie. This cookie is managed by Shopify and cannot be cleared by Cooee after the experiment stops. If the draft theme is left in your theme library, visitors who were in the variant arm will continue seeing it on return visits — indefinitely. Deleting the draft theme is the only reliable way to ensure those visitors revert to your live theme.
Recommended before deleting: Duplicate the theme first (Themes → ⋯ menu → Duplicate). This preserves all your design changes in a copy you can reference later, without affecting live visitor traffic.

Note: Stopping the experiment in Cooee does not automatically publish or delete the challenger theme. You take that action yourself in Shopify Admin.

Known limitations and gotchas

URL parameter on first load
Visitors assigned to the variant arm will see ?preview_theme_id=...&pb=0 appended to their first URL. Shopify removes this parameter after serving the theme preview — it does not appear on subsequent page navigations. This parameter is not indexed by search engines.

Cross-device assignment
Visitor assignment is stored locally per browser. A visitor switching from mobile to desktop (or using a different browser) will receive a fresh, independent assignment. This is standard behaviour for all browser-based experiments.

Shopify preview bar
The pb=0 parameter suppresses the preview bar, ensuring variant visitors see the same production-like experience as control visitors — no "You are previewing..." banner.

Sample size and runtime
Thin traffic stores may need 4–6 weeks to reach significance on conversion-level metrics. If your store has fewer than 500 daily unique visitors, consider running the experiment at least 4 weeks before evaluating results.