CRO A/B Testing – Standard Operating Procedure (SOP)
This SOP defines the standardized process for executing Image, Title, Element Position, Colour, and Placement A/B tests across all CRO initiatives. The objective is to ensure data integrity, consistent decision-making, and repeatable learnings across experiments.
1. Purpose
To establish a unified framework for planning, executing, validating, and scaling A/B experiments that directly impact conversion rate, revenue, and user experience.
2. Scope
This SOP applies to all CRO-led experiments including:
Image testing
Title / Headline testing
Element Position testing
Colour testing
Placement testing
Applicable to:
PDP, PLP, Cart, Checkout, Homepage
Mobile and Desktop
Paid and Organic traffic
3. Universal Rules (Applicable to All Tests)
Only ONE Primary KPI per test.
Numeric success criteria must be defined before launch.
Sample size and MDE must be pre-calculated.
Every test must run for a minimum of 2 full business cycles.
Device-wise and traffic-wise analysis is mandatory.
Guardrail metrics must remain stable for a test to be declared a winner.
No early stopping based on short-term performance spikes.
Post-test documentation is mandatory for every concluded experiment.
4. KPI Framework
4.1 Primary KPIs (Choose One Per Test)
Add-to-Cart Rate (ATC)
Conversion Rate (CVR)
Revenue per Visitor (RPV)
Click-through Rate (CTR)
Engagement (Scroll Depth)
4.2 Guardrail KPIs
CVR (if not Primary)
RPV
AOV
Bounce Rate
5. Goal Definition Framework
Each test must define numeric targets and safety thresholds before launch. Example:
Primary Goal: +6% uplift in ATC
Guardrails:
CVR drop must not exceed 2%
Bounce Rate must not increase beyond 3%
LCP must remain within +300ms of baseline
6. Hypothesis Framework (Mandatory)
All experiments must follow this structure: "If we change [variable] for [user segment], then [primary KPI] will improve because [behavioural reason]."
TEST-TYPE SPECIFIC SOPs
7. Image A/B Testing SOP
7.1 Primary KPI Mapping
PDP → ATC Rate
PLP → CTR to PDP
Cart/Checkout → CVR
Bundles/Cross-sell → RPV
7.2 Guardrails
CVR, RPV, AOV, Bounce, LCP, Refund Rate
7.3 Allowed Variables (Only One at a Time)
Lifestyle vs Studio
Model vs No Model
Close-up vs Wide Angle
Video vs Still
First Thumbnail Only
Mobile Crop Only
7.4 Validation Rules
Same CDN, compression, and file format
No CLS introduced
Hero image must not be lazy-loaded
Separate mobile and desktop analysis
Minimum 2 business cycles
7.5 Winner Declaration
Statistically significant Primary KPI lift
Guardrails stable
No performance degradation
8. Title / Headline A/B Testing SOP
8.1 Primary KPIs
PDP → ATC Rate
PLP / Homepage → CTR
Checkout/Trust Titles → CVR
8.2 Guardrails
Bounce, Scroll Depth, CVR, RPV
8.3 What Can Be Tested
Benefit vs Feature-led
Emotional vs Functional
Short vs Descriptive
Problem-led vs Outcome-led
Urgency vs Informational
8.4 What Cannot Change
Price
Offer
Layout
Supporting imagery
8.5 Validation Rules
Same font, size, placement
No capitalization bias
Equal hierarchy across variants
50:50 traffic split
Minimum 7–10 day runtime
9. Element Position A/B Testing SOP
(Applies to CTA, Price, Reviews, Trust Badges, Offers)
9.1 Primary KPIs
CTA / Price Movement → ATC Rate
Trust Elements → CVR
9.2 Guardrails
Bounce, Scroll Depth, RPV, Time to Interaction
9.3 Allowed Changes
Above vs Below Fold
Left vs Right Column
Sticky vs Non-Sticky
Inline vs Sectioned
9.4 Disallowed Changes
Copy
Colour
CTA Text
Page Layout Structure
9.5 Validation Rules
No CLS introduction
No sticky overlap with UI
Heatmap validation recommended
Device segmentation mandatory
10. Colour A/B Testing SOP
10.1 Primary KPIs
CTA Colour → ATC Rate
Offer Highlight Colour → CVR
UX Background Colour → Bounce Rate
10.2 Guardrails
CVR, RPV, Accessibility Contrast Score
10.3 Allowed Changes
Button colour only
Badge highlight only
Section background only
10.4 Disallowed Changes
CTA Text
Button Size
Position
Shape
10.5 Validation Rules
Accessibility AA contrast compliance
Same hover state intensity
No seasonal bias unless seasonality is being tested
Weekday + Weekend coverage mandatory
11. Placement A/B Testing SOP
(Applies to Offers, Banners, Social Proof, Cross-sells, Widgets)
11.1 Primary KPIs
Offers → CVR or RPV
Social Proof → CVR
Cross-sell → RPV or AOV
11.2 Guardrails
ATC, Bounce, Engagement, Scroll Abandonment
11.3 Allowed Placement Changes
Above Fold vs Mid-Page
Inline vs Modal
Pre-ATC vs Post-ATC
PDP vs Cart Placement
11.4 Disallowed Changes
Offer Value
Copy
Creative Assets
Pricing
11.5 Validation Rules
No trigger duplication
No overlap with core UI
Funnel stage consistency maintained
Correct attribution mapping
12. Pre-Launch QA Checklist (Mandatory for All Tests)
Correct page and SKU targeting
All variants load correctly
Mobile and Desktop validation completed
Analytics events firing for Product View, ATC, Purchase
No cache bleed between variants
Baseline metrics captured for 3–5 days
13. Test Validity & Winner Declaration Criteria
A test can be declared valid only when all conditions are met:
Statistical significance achieved
Planned sample size reached
Planned duration completed
Primary KPI shows meaningful lift
Guardrails show no critical degradation
14. Post-Test Documentation (Mandatory)
Each test must record:
What changed
Primary KPI
Result summary
Segment insights
Final rollout decision
Follow-up test recommendation
15. Rollout & Re-Validation Rules
Winners must be validated on fresh traffic or second SKU
If replicated → Full rollout
If not replicated → Segment-only rollout
16. Final Decision Matrix
Strong lift + clean guardrails → Full rollout
Lift only on mobile → Device-specific rollout
Engagement up, CVR flat → Downstream funnel test
CVR up, AOV down → Evaluate RPV
Performance degraded → Variant rejected
End of SOP