Sample size is usually decided by how long somebody is willing to wait. The test runs a fortnight, collects whatever traffic a fortnight brings, and the result is read as though the number were chosen rather than inherited.

What that sample size buys is statistical power, and it is measurable. This simulates experiments in which a real difference exists, at a range of sample sizes, and counts how often the test notices.

The instrument, checked first

The same harness was first run with no effect at all, where it should reject at roughly the significance threshold:

no effect, n=2,000:   5.4% rejected   (expected ~5%)

Close enough to five to proceed. Without that check every number below would be indistinguishable from a broken simulation.

What each sample size actually buys

Twenty thousand simulated experiments per cell, a baseline conversion of ten percent, and a genuine lift of half a point, one point, or two points. The figure is the percentage of experiments that detect the effect:

lift        n=500  1,000  2,000  5,000  10,000  20,000  50,000
+0.5pp        5.9    6.4    8.3   12.9    21.7    38.0    74.1
+1.0pp        7.8   11.5   17.8   37.4    63.7    90.5    99.9
+2.0pp       17.7   30.4   52.6   89.4    99.5   100.0   100.0

Read the middle row at two thousand per arm: 17.8%. A one-point improvement, which is a ten percent relative gain on this baseline and the kind of change a team would be delighted with, is found in fewer than one experiment in five. Four times out of five the test reports nothing and the change is filed as ineffective.

Taking the usual eighty percent threshold, a one-point lift first clears it at twenty thousand per arm. That is ten times the sample most people would have guessed, for an effect most people would consider large.

A half-point lift does not reach eighty percent anywhere in this grid. At fifty thousand per arm it is still only 74.1%, and detecting it reliably means a test most sites cannot run at all.

Small tests do not fail neutrally

An underpowered test is often described as inconclusive, which sounds harmless. Its failures are not neutral.

The first is that a null result gets read as evidence of no effect. "We tested it and it did not help" is the sentence that ends the idea, and at 17.8% power that sentence is wrong far more often than it is right.

The second failure is subtler and worse. Among the small minority of underpowered tests that do reach significance, the measured effect must have been enormous just to clear the threshold on a small sample. So the winners you see are systematically overstated. A one-point lift, when found by a test barely able to find it, will be reported as two or three, and the follow-up that fails to reproduce it looks like a mystery rather than a consequence.

Underpowering dismisses real improvements, and leaves the few that survive exaggerated.

Working out your own number before you start

The inputs are the ones the table above uses, and none of them require the test to have run:

baseline rate     what the control does today
smallest effect   the smallest lift worth shipping
alpha             usually 0.05
power             usually 0.80

The one that decides everything is the second, and it is a business question rather than a statistical one. Sample size scales with roughly the inverse square of the effect: halving the effect you want to catch multiplies the required sample by about four. That relationship is visible in the table — the half-point row needs far more than twice what the one-point row needs.

So the question worth asking at the start is "how small an improvement would still be worth shipping?", rather than "how long can we run this?" If the answer is half a point, the honest follow-up is that this experiment cannot be run on this traffic, and something other than an A/B test has to make the decision.

That is a better outcome than running it anyway. A test you know is underpowered is a decision made on other grounds with a statistic attached to it.

The two failures compound

Two thousand per arm is the same sample size used in the companion simulation on checking a test early, where repeated peeking pushed the false-positive rate to 24.8%.

Put the two together and the ordinary experiment is in a genuinely bad position. At two thousand per arm it finds a real one-point lift only 17.8% of the time, and if it is watched the way people actually watch, it will announce one that is not there in a quarter of cases. The same test, over the same fortnight, failing in both directions at once.

How this was measured, and what it does not settle

Twenty thousand simulated experiments per cell, ten percent baseline, absolute lifts of 0.5, 1.0 and 2.0 points, two-proportion z-test at alpha 0.05, seeded for reproducibility, with a no-effect control confirming the harness rejects at approximately alpha.

The grid is coarse. "First reaches eighty percent at twenty thousand" means the smallest size tested that cleared it, not the exact requirement — the true figure sits somewhere between ten and twenty thousand. A closed-form calculator gives the precise number and this is not one.

Absolute lifts, one baseline. A one-point lift on a ten percent baseline is a ten percent relative change; the same absolute lift on a fifty percent baseline is a much smaller relative one and behaves differently. Anyone applying this should recompute at their own rate rather than borrowing the cells.

Eighty percent is a convention. It is what tooling defaults to, not a property of the world, and it encodes a willingness to miss a real effect one time in five. Choosing it deliberately is fine. Inheriting it without noticing that it is a choice is how the number stops meaning anything.