Nobody starts a test and waits. The dashboard is right there, the result updates continuously, and looking costs nothing — so people look daily, and stop when the number turns green.

That habit has a price. To measure it, we simulated twenty thousand experiments in which there is no difference to find, each run twice: once by someone who waits for the end, and once by someone who looks every so often and stops as soon as the result is significant.

The setup, and why it can only produce false positives

Both arms are given the same true conversion rate — ten percent — and two thousand visitors each. Because nothing distinguishes them, any result that comes back "significant" is wrong by construction. The count of those errors is a direct measurement of the false-positive rate rather than an estimate of one.

trials             20,000
per arm             2,000
true rate, both       10%
alpha               0.05

First, check the instrument

Before trusting anything, the harness has to reproduce the answer we already know. A test looked at once, at the end, should call about five percent of these significant, because that is what a 0.05 threshold means.

looked once, at the end:   4.9% significant

Four point nine against a theoretical five. The simulation is doing arithmetic correctly, so the next number is a finding rather than a bug.

Then look twenty times instead of once

Same data, same threshold. The only change is that the experimenter checks after every hundred visitors per arm — twenty times over the life of the test — and stops the moment the result crosses 0.05.

looked once:      4.9% significant
looked 20 times: 24.8% significant

A five-fold increase. One experiment in four now produces a confident, statistically significant result about a difference that does not exist, and the experimenter has done nothing except watch.

How fast it goes wrong

Twenty looks are not required for the damage to arrive.

looks   false positives
    1              5.3%
    2              8.2%
    5             14.2%
   10             19.3%
   20             24.8%

Checking twice rather than once already takes you from five percent to eight — the cost of having "only looked a couple of times", which is the version of this habit that feels obviously harmless.

The mechanism is simple once the shape is visible. A p-value below 0.05 means the observed gap is unlikely on a single look. Random walks wander; over the life of a test the gap between two identical arms drifts up and down, and each additional look is another opportunity to catch it at a wandering extreme. Stopping there freezes the wander and calls it a result.

The false positives arrive early, which is what makes them convincing

When peeking did fire, the median stopping point was 400 visitors per arm — a fifth of the way through a test that was scheduled to run to two thousand.

That is the worst possible timing for a human. An effect that appears early and is significant reads as a strong effect: large enough to detect quickly, obviously worth shipping. Early significance is where the noise is largest. On a small sample a random gap needs to be enormous to clear the threshold on four hundred observations, so the most persuasive false positives are the ones that show up soonest.

What to do instead

Fix the horizon before you start. Decide the sample size in advance and read the result once, which is what the 0.05 threshold assumes you will do. For most teams that is the whole fix, and it costs nothing.

If you must watch, do not act. Monitoring a test for operational reasons — is it running, is one arm erroring, is traffic split evenly — is fine. The rule is about stopping, not looking, and the two are easy to conflate when the same screen shows both.

If stopping early is a requirement, use a method built for it. Sequential designs — alpha spending, group sequential boundaries, always-valid confidence intervals — exist to permit repeated looks. They keep the total error at five percent by demanding more evidence at each look. You pay for that in sample size when the effect is real, and what you get back is an answer that means what it says.

What does not work is looking repeatedly at a fixed-horizon test and applying judgement. The 24.8% above is the judgement of somebody who stops when the result looks good.

How this was measured, and what it does not show

Twenty thousand simulated A/A experiments, two thousand observations per arm, both arms at a ten percent rate, two-proportion z-test at alpha 0.05, twenty equally spaced looks, seeded for reproducibility. The script is in the repository and reproduces these figures exactly.

This is a simulation, and that is the appropriate tool. It shows what the procedure does, which is the claim being made. It is not evidence about what any particular team's experiments did, and it cannot be — the whole problem is that a false positive looks exactly like a true one from inside the test.

Only one abuse is modelled. Stopping early on success is the common one. Extending a test that has not reached significance, quietly dropping an arm, or running many metrics and reporting the one that moved are separate practices with their own inflation, and none of them are counted here.

The exact percentages belong to this grid. More looks inflate further, fewer inflate less; the baseline rate barely matters, and the sample size affects when the wandering is largest rather than whether it happens. The shape of the table is what carries over to another setting; the individual cells do not.

The other half of this problem is the opposite failure: a test too small to find an effect that is actually there. Two thousand per arm turns out to be very small indeed.