Q: Will visitors notice they're in an experiment?
No. For Theme A/B Testing, the variant theme loads via Shopify's built-in preview mechanism with the preview bar suppressed. The experience is indistinguishable from the live theme. For Split URL Testing, the redirect happens before the page renders — visitors land directly on the variant URL.

Q: Can the same visitor end up in different arms on different visits?
No. Once a visitor is assigned to an arm, their assignment is stored and reused on every subsequent visit — they are "sticky" to their arm. This is critical for data integrity: if visitors flipped arms between visits, the results would be unreliable.

Q: What if a visitor clears their cookies and localStorage?
If both localStorage and the cooee_vid cookie are cleared, Cooee generates a new visitor ID and the visitor may be assigned to a different arm. In practice, both are rarely cleared simultaneously. The cookie (cooee_vid) has a 2-year expiry and survives most browser data clears on Safari.

Q: Are visitors on mobile and desktop assigned separately?
Yes — a visitor using a different device or browser receives a fresh assignment on that device. This is standard for all browser-based experiments. Your results in Cooee can be segmented by device type.

Q: Can I target the experiment to only new or only returning visitors?
Audience targeting by visitor type is on the roadmap. Reach out to your Cooee contact to discuss targeting options for your specific experiment.

Q: How long should I run an experiment?
Run for at least 2 full business cycles (typically 14 days minimum). This accounts for day-of-week variation. For stores with fewer than 500 daily unique visitors, plan for 4–6 weeks on conversion-level metrics.

Q: What traffic split should I use?
A 50/50 split gives you the fastest path to statistical significance. Use a smaller variant split (e.g. 20%) only if you want to limit exposure of an unproven experience to visitors. Remember: a smaller variant allocation means more time needed to conclude the experiment.

Q: Can I run multiple experiments at the same time?
Cooee supports one active redirect experiment per store at a time (Theme A/B or Split URL). Running overlapping redirect experiments would create conflicting assignments. Campaign-type experiments (popups, banners, etc.) can run in parallel with redirect experiments.

Q: How many visitors do I need before I can call a winner?
This depends on your baseline conversion rate and the minimum lift you want to detect. As a rough guide: to detect a 5% relative lift on a 2% conversion rate with 80% power and 95% significance, you need approximately 10,000 visitors per arm. Use a standard A/B test sample size calculator before launching.

Q: When do results update?
Experiment results in Cooee update daily. Real-time impression counts are visible immediately after the experiment starts.

Q: What does "Pure Revenue" mean?
Pure Revenue counts only orders placed by visitors who converted within the same session they were first assigned to an arm. This is a conservative attribution model that eliminates cross-session noise. Standard revenue attribution includes all orders from assigned visitors regardless of when they converted.

Q: My variant has more impressions but lower conversion — what does this mean?
Impressions count unique visitors who started a real session in that arm. A discrepancy in impressions between arms could happen due to early rounding in a new experiment (expected to equalise over time) or because one arm attracts a slightly different mix of traffic (e.g. more organic vs paid). Always check that the traffic split is close to your configured percentage before drawing conclusions.

Q: Can I see results broken down by device type?
Yes — your Cooee dashboard shows per-arm metrics segmented by mobile and desktop. Always check device-level results; a variant that wins on desktop but loses on mobile (or vice versa) calls for a device-specific rollout decision.

Q: What happens if I pause and then resume an experiment?
Pausing removes the experiment from the active assignment list. New visitors during the pause period will not be assigned. When you resume, new visitors are assigned again; existing sticky assignments are preserved. Pause periods are not counted in the experiment runtime, so plan accordingly.

Q: What happens when I stop an experiment?
Stopping marks the experiment as concluded and no new visitors are assigned. Your experiment data remains accessible in the dashboard.

There is an important cleanup step required in Shopify. When a visitor was first assigned to the variant arm, Shopify stored a reference to the variant theme in a persistent browser cookie. This cookie is managed entirely by Shopify and cannot be cleared by Cooee after the experiment stops. As a result, visitors who were in the variant arm will continue to see the variant/draft theme on return visits — indefinitely — unless you take action on the theme itself.

The only reliable way to ensure those visitors revert to your live theme is to take one of the following actions in Shopify Admin (Online Store → Themes):

Recommended before deleting: Duplicate the theme first (Themes → ⋯ menu → Duplicate). This preserves all your design changes in a copy you can revisit later, without affecting live visitor traffic.

Stopping the experiment in Cooee does not automatically delete or publish the challenger theme. You take that action in Shopify Admin.

Q: Can I change the traffic split after the experiment starts?
Changing a live traffic split is possible but not recommended. It changes the assignment pool mid-experiment, which introduces bias and makes statistical analysis unreliable. If you need to adjust, consider stopping and restarting the experiment.

Q: The experiment stopped automatically — why?
Cooee automatically stops a Theme A/B experiment if: (a) the challenger theme is published as the live Shopify theme, or (b) the challenger theme is deleted from Shopify. In either case, the variant arm's theme is no longer available as a distinct preview, so the experiment cannot continue cleanly. You'll see an auto-stop notice in the experiment details.

Q: Does the experiment affect page speed?
The experiment-runtime snippet runs synchronously in <head> and takes less than 5 ms for assignment resolution on a warm cache. The assignment logic uses deterministic hashing with no network calls. The only network request is a single background beacon after assignment, which uses fetch with keepalive: true and does not block the page render.

Q: Will the preview_theme_id parameter show up in Google Analytics or my ad platform?
The parameter appears on the first URL a variant visitor lands on. Shopify removes it after the theme preview loads, so it does not persist to subsequent page navigations. Most analytics tools use the full URL including query parameters for landing page attribution, so you may see a small number of sessions attributed to the preview_theme_id URL. This is expected and does not affect experiment validity.

Q: Does Cooee A/B testing affect SEO?
Theme A/B Testing does not change the URL or page content as seen by search engines. Googlebot is detected and excluded from experiment assignment — it will always see your control (live) theme. Split URL Testing can affect indexation of the variant URL if the experiment runs for an extended period; see the Split URL Testing documentation for SEO guidance.

Q: Can bots or automated testing tools distort my results?
No. Cooee's assignment logic checks for bot signals (navigator.userAgent, navigator.webdriver) and excludes automated traffic from assignment and impression counting. Internal QA sessions using standard browsers with real user agents may be assigned — use a private browsing window and clear localStorage if you need to test the assignment flow without affecting results.