What Is A/B Testing?

Should the "Buy" button be green or orange? Will a shorter signup form get more users? Instead of guessing, companies show version A to some users and version B to others, then compare results. That's an A/B test.

The hard part isn't building two versions. It's knowing whether a difference is real or just luck. That's where a little statistics comes in, and where most teams make mistakes.

๐ŸŽฏ Quick Answer (30-Second Read)

  • Split users randomly: control (A) vs variant (B).
  • Pick one main metric before starting, such as conversion rate.
  • Statistical significance asks: if there were no real difference, how surprising would this result be? That's the p-value. The usual bar is p < 0.05.
  • Sample size matters most: small improvements need lots of users to detect.
  • Don't peek and stop early when it looks good. That inflates false wins.

How an A/B Test Works

flowchart LR U["๐Ÿ‘ฅ Visitors"] --> SPLIT{"Random
50 / 50 split"} SPLIT --> A["A: current page
(control)"] SPLIT --> B["B: new page
(variant)"] A --> MA["Measure conversions"] B --> MB["Measure conversions"] MA --> CMP["Compare with a
statistical test"] MB --> CMP CMP --> D{"Significant?"} D -->|Yes| SHIP["Ship the winner"] D -->|No| KEEP["Keep A, or test
a bigger change"]

Random assignment is the magic. Because users are split by chance, the only systematic difference between the groups is the change you made. Each user should consistently see the same version, usually by hashing their user ID.

The Core Question: Real Effect or Luck?

Say you run a test:

Visitors Signups Conversion
A (control) 10,000 500 5.0%
B (variant) 10,000 560 5.6%

B looks 12% better (5.6 vs 5.0). But flip a coin 100 times and you won't get exactly 50 heads. Random variation happens. So we ask:

If A and B were actually identical, how likely is a gap at least this big, just by chance?

That probability is the p-value.

Working It Out

For conversion rates, a standard method is the two-proportion z-test:

  1. Pooled rate: p = (500 + 560) / 20,000 = 0.053
  2. Standard error: โˆš[p(1 โˆ’ p)(1/10,000 + 1/10,000)] โ‰ˆ 0.00317
  3. z = (0.056 โˆ’ 0.050) / 0.00317 โ‰ˆ 1.89
  4. A z of 1.89 gives a two-sided p-value โ‰ˆ 0.058

Verdict: 0.058 is just above 0.05. This result is not quite significant. B might be better, but the data can't rule out luck yet.

from statsmodels.stats.proportion import proportions_ztest

z, p = proportions_ztest(count=[560, 500], nobs=[10_000, 10_000])
print(round(z, 2), round(p, 3))   # 1.89 0.058

What a P-Value Is (and Isn't)

โœ… It is
The chance of seeing a difference this large (or larger) if there were no real effect.
โŒ It isn't
The probability that B is better, or the probability your result is wrong. A tiny p-value can also come with a tiny, unimportant effect.

p < 0.05 means: if there were no effect, a result this extreme would happen less than 5% of the time. Run 20 tests on changes that do nothing, and on average one will look like a winner anyway.

Sample Size: Decide Before You Start

The smaller the improvement you want to detect, the more users you need. A widely used shortcut (for 80% power and 5% significance) is:

n per group โ‰ˆ 16 ร— p(1 โˆ’ p) รท ฮดยฒ

where p is the baseline conversion rate and ฮด is the absolute difference you want to detect.

Baseline Want to detect Users per group
5% 5.0% โ†’ 6.0% (+1 point) โ‰ˆ 7,600
5% 5.0% โ†’ 5.5% (+0.5 point) โ‰ˆ 30,400
5% 5.0% โ†’ 5.25% (+0.25 point) โ‰ˆ 121,600

Halving the effect size quadruples the sample needed. That's why small sites struggle to test tiny tweaks, and should test bolder changes.

Mistakes That Fake a Win

Watch out, peeking: checking results every day and stopping the moment p dips below 0.05 dramatically increases false positives, because p-values bounce around early on. Decide the sample size up front, or use methods built for continuous monitoring (sequential testing).
  • Too many metrics: test 20 metrics and one will "win" by chance. Pick one primary metric.
  • Stopping too early: run at least one full week to cover weekday/weekend behaviour.
  • Novelty effect: users click new things just because they're new. Watch whether the lift fades.
  • Unequal split by accident (sample ratio mismatch): if you planned 50/50 and got 52/48 on large numbers, something's broken. Investigate before trusting results.
  • Ignoring practical significance: a "significant" +0.1% may not be worth the engineering cost.

Frequently Asked Questions

How long should an A/B test run?

Until it reaches the sample size you calculated before starting, and for at least one full week so every day of the week is represented. Stopping as soon as results look significant inflates false positives, so decide the length in advance.

What does statistically significant mean in A/B testing?

It means the observed difference would be unlikely if the two versions truly performed the same, usually defined as a p-value below 0.05. It doesn't guarantee the variant is better or that the improvement is large enough to matter for the business.

What if my site doesn't have enough traffic for A/B tests?

Test bigger, bolder changes that could produce large effects, focus on high-traffic pages, use a metric closer to the change (like clicks rather than purchases), and complement tests with user interviews and qualitative research.

Conclusion

A/B testing turns opinions into evidence: split users randomly, measure one metric, and use statistics to separate real effects from luck. Calculate the sample size first, don't stop early when results look good, and remember that a significant result still has to be big enough to matter.

Sources

Last verified: 14 October 2026.

Related reads: How the Instagram Algorithm Works ยท How the YouTube Algorithm Works ยท Ship It or Kill It: A Solo Product Framework

I explain the tech and math behind products. Follow on LinkedIn or X, or invite me to speak.