A/B Testing & Experimentation: Significance, Power & Pitfalls

Q: How long should I run an A/B test?

Until you reach the sample size you calculated before starting — based on your baseline conversion rate, the minimum effect size worth detecting, and your desired statistical power (commonly 80%) and significance (commonly 95%). Practically, also run for at least one or two full business cycles (usually 1–2 weeks) so day-of-week effects average out. The cardinal sin is stopping early because it "looks" significant — early results swing wildly, and peek-and-stop dramatically inflates false positives. Decide the duration up front and honor it.

Q: What does "statistically significant" actually mean?

It means the difference you observed is unlikely to have occurred by random chance, given a chosen threshold (e.g. p < 0.05 means roughly a 5% chance of seeing this difference if there were truly no effect). It is not proof the effect is real, large, or important — it's a risk level for one type of error. A significant result with a tiny effect may not be worth shipping, and an underpowered test can miss a real effect entirely. Pair significance with effect size and confidence intervals for a complete picture.

Q: What is p-hacking and how do I avoid it?

P-hacking is torturing the data until it confesses — slicing by segments, trying many metrics, or stopping at a convenient moment until something crosses the significance threshold. Because random noise will occasionally look significant, this manufactures false "wins." Avoid it by pre-registering your hypothesis and a single primary metric before running, computing sample size in advance, running the full planned duration, and correcting for multiple comparisons when you do look at several metrics. Discipline about deciding what you'll measure before you see the data is the core defense.

Q: How is A/B testing different from just shipping and watching the metric?

Shipping and watching is a before/after comparison, which is confounded by everything else that changes over time — seasonality, marketing, other releases, news events. You can't tell whether the metric moved because of your change or something unrelated. A/B testing runs control and variant simultaneously with random assignment, so both groups experience the same external conditions and the only systematic difference is your change. That's what makes the result causal rather than correlational. It's more rigorous precisely because randomization removes the confounds a simple before/after can'

A/B Testing & Experimentation

A/B testing is how you find out whether a change actually caused an improvement, rather than guessing. You randomly split users into a control group (current experience) and one or more variants (the change), then compare a metric. Because the split is random, any difference in outcomes can be attributed to the change — making A/B testing the gold standard for causal measurement in product and growth, where opinions and intuition are notoriously unreliable.

The catch is that doing it correctly requires statistical discipline. It's easy to run a test, see a number go up, and ship a change that was actually noise. Understanding significance, power, sample size, and the classic pitfalls (peeking, p-hacking) is what separates real learning from confident self-deception.

TL;DR

Randomly split users into control vs variant; the randomization makes differences causal.
Define one primary metric and the required sample size before starting.
Wait for statistical significance — and don't stop early ("peeking").
Beware p-hacking and multiple comparisons — they manufacture false wins.

Quick Example

The structure of a clean experiment:

How It Works

Hypothesis — a specific, testable prediction ("changing X improves metric Y").
Randomize — assign users to control or variant(s) at random, so the groups are comparable.
Measure — track the primary metric for each group.
Analyze — is the difference statistically significant, or could it be chance?
Decide — ship the winner, discard the loser, or iterate.

Randomization is the magic: it controls for everything else, so the only systematic difference between groups is the change you made.

The Statistics That Matter

Statistical significance (p-value) — the probability the observed difference happened by chance. A common threshold is p < 0.05, but it's not a guarantee — it's a risk level.
Statistical power — the test's ability to detect a real effect; depends on sample size and effect size. Underpowered tests miss real wins.
Sample size — calculate it before running, from your baseline rate, the minimum effect you care about, and desired power. Too small → inconclusive or misleading.
Confidence interval — the plausible range of the true effect, more informative than a single point estimate.

⚠️ "The variant is winning after 2 days!" means little if you haven't reached your pre-computed sample size. Early results swing wildly.

Common Pitfalls

Peeking — repeatedly checking and stopping the moment it's significant inflates false positives. Decide the duration/sample size up front and wait.
P-hacking — slicing the data many ways until something looks significant. Pre-register your primary metric.
Multiple comparisons — test 20 metrics and ~1 will look "significant" by chance. Correct for it, or focus on one primary metric.
Too small / too short — underpowered tests and ignoring weekly cycles produce noise.
Sample ratio mismatch — if your 50/50 split isn't actually 50/50, the experiment is broken; check it.

Best Practices

Define one primary metric and the sample size before you start — and a clear hypothesis.
Run for the planned duration (full business cycles) and don't peek-and-stop.
Watch guardrail metrics — make sure a "win" didn't harm something else (latency, churn, revenue).
Account for multiple comparisons — don't fish through dozens of metrics.
Validate with feature flags for safe rollout, and tie results to real value, not vanity metrics.

Common Mistakes

Stopping the test as soon as it looks significant

Testing without enough traffic

FAQ

How long should I run an A/B test?

Until you reach the sample size you calculated before starting — based on your baseline conversion rate, the minimum effect size worth detecting, and your desired statistical power (commonly 80%) and significance (commonly 95%). Practically, also run for at least one or two full business cycles (usually 1–2 weeks) so day-of-week effects average out. The cardinal sin is stopping early because it "looks" significant — early results swing wildly, and peek-and-stop dramatically inflates false positives. Decide the duration up front and honor it.

What does "statistically significant" actually mean?

It means the difference you observed is unlikely to have occurred by random chance, given a chosen threshold (e.g. p < 0.05 means roughly a 5% chance of seeing this difference if there were truly no effect). It is not proof the effect is real, large, or important — it's a risk level for one type of error. A significant result with a tiny effect may not be worth shipping, and an underpowered test can miss a real effect entirely. Pair significance with effect size and confidence intervals for a complete picture.

What is p-hacking and how do I avoid it?

P-hacking is torturing the data until it confesses — slicing by segments, trying many metrics, or stopping at a convenient moment until something crosses the significance threshold. Because random noise will occasionally look significant, this manufactures false "wins." Avoid it by pre-registering your hypothesis and a single primary metric before running, computing sample size in advance, running the full planned duration, and correcting for multiple comparisons when you do look at several metrics. Discipline about deciding what you'll measure before you see the data is the core defense.

How is A/B testing different from just shipping and watching the metric?

Shipping and watching is a before/after comparison, which is confounded by everything else that changes over time — seasonality, marketing, other releases, news events. You can't tell whether the metric moved because of your change or something unrelated. A/B testing runs control and variant simultaneously with random assignment, so both groups experience the same external conditions and the only systematic difference is your change. That's what makes the result causal rather than correlational. It's more rigorous precisely because randomization removes the confounds a simple before/after can't.