A/B Testing Statistics: The Complete Guide for Growth Teams

Statistical significance, p-values, and what they actually mean

A p-value is the probability of observing your result (or something more extreme) if there were actually no effect. A p-value of 0.05 means there is a 5% chance of seeing this result by random chance alone if the null hypothesis is true. 'Statistical significance at 95% confidence' does not mean there is a 95% chance your variant is better — it means you'd see a false positive 5% of the time if you ran this test on no-effect data. The practical implication: if you run 20 tests with no real effect, you expect 1 false positive. In a high-velocity testing program, false positives compound quickly unless you control for them.

Statistical power and the danger of underpowered tests

Statistical power is the probability of detecting a real effect if one exists. The standard target is 80% power — meaning you accept a 20% chance of a false negative (missing a real effect). An underpowered test doesn't just risk missing effects — it also produces inflated effect size estimates when it does reach significance (the 'winner's curse'). The most common cause of underpowered tests is stopping early: seeing a positive trend on day 3 and ending the test before reaching your pre-calculated sample size. This inflates your measured effect by 2–5× and produces a false positive at roughly the same rate as running a test on completely random data.

How to calculate the right sample size

Sample size depends on four inputs: your baseline conversion rate, your minimum detectable effect (MDE), your desired statistical power (default 80%), and your significance threshold (default 5%). For a signup flow with a 3% baseline conversion rate and a target of detecting a 20% relative improvement (0.6 percentage points absolute), you need roughly 11,000 users per variant at 80% power. Smaller MDEs require exponentially larger samples. The most common mistake is setting MDE too small (trying to detect a 5% relative change with insufficient traffic) and either running underpowered tests or running tests for months. Our sample size calculator handles this math automatically.

The peeking problem and how to solve it

Peeking — looking at test results before reaching your pre-calculated sample size and potentially stopping early — inflates your false positive rate dramatically. A test run with an 'optional stopping' rule (stop when p < 0.05) has an actual false positive rate of 22–26%, not 5%. The solutions: (1) pre-commit to your sample size and don't look at results until you've reached it (hard to maintain in practice), (2) use sequential testing methods (SPRT or mSPRT) that account for multiple looks, or (3) use Bayesian methods that naturally accommodate early stopping without inflating error rates. Tools like Statsig, Eppo, and Amplitude Experiment have sequential testing built in.

A/B test statistical standards checklist

• Sample size is calculated before the test launches (not after)
• Minimum detectable effect is defined and realistic for your traffic volume
• Statistical power is set to 80% or higher
• Significance threshold is defined (95% standard; 99% for high-stakes decisions)
• Test runs for the full pre-calculated duration regardless of early trends
• No peeking: results not reviewed until sample size is reached
• Null results are documented the same way as positive results
• A minimum test duration of 7 days accounts for weekday/weekend variation