Glossary
A/B testing
What is A/B testing?
A/B testing compares two versions of a product, page, or message by randomly assigning users to each and measuring an outcome chosen in advance.
How it works
One group receives the current version, A. Another receives version B. A checkout test, for example, might compare two buttons and measure completed purchases. Random assignment helps separate the effect of the change from differences in who sees it.
Decide how much data to collect and how to analyze it before starting. With a conventional fixed-sample test, stopping the moment a p-value looks good can increase false positives. Johari, Pekelis, and Walsh describe statistical methods designed for continuous monitoring; ordinary fixed-sample tests do not become those methods just because a dashboard updates every hour.
A worked checkout example
Suppose a hypothetical test assigns 10,000 visitors to each checkout. Version A produces 300 completed purchases and version B produces 330. Their conversion rates are 3% and 3.3%: an increase of 0.3 percentage points, or 10% relative to A's rate. Those are two ways of describing the same observed difference, not interchangeable percentages.
The extra 30 purchases are an observed result, not yet a reliable estimate of what will happen after rollout. Apply the analysis chosen before the test, report uncertainty around the difference, and compare the plausible benefit with the cost of the change. The counts alone do not establish statistical significance.
Why it matters
A/B testing lets a team test a design argument against what users do. The result still needs interpretation. How large was the change? How uncertain is the estimate? Were users assigned and purchases measured correctly? A statistically significant difference can be too small to matter to the business or the customer.
Conversion rate, click-through rate, and retention answer different questions. Choose the outcome carefully, too. In the checkout example, a button that gets more clicks has not necessarily produced more completed purchases.