An experiment dashboard rarely shows one number. It shows conversion, revenue per visitor, bounce rate, time on page, add-to-cart, sign-ups, retention at seven days, and a dozen more — all updating together, all with a significance marker beside them.

Something on that screen will usually be green. This is a measurement of how often that happens when nothing has changed at all.

The setup

Two arms drawn from the same distribution, five thousand visitors each, and a set of metrics that are all equally null. Twenty thousand simulated experiments at each metric count, with significance at the usual 0.05.

Because no metric differs between the arms, every green marker is a false positive. The question is only how many chances the experimenter gives themselves to see one.

The control

1 metric examined:   5.1% of experiments show something significant

Five point one against a theoretical five. The harness is doing the arithmetic correctly, so the rest of the table is a finding rather than a bug.

What each extra metric costs

metrics    at least one "significant"    theory
      1                        5.1%       5.0%
      2                        9.5%       9.8%
      5                       22.8%      22.6%
     10                       40.4%      40.1%
     20                       64.2%      64.2%
     50                       92.3%      92.3%

With twenty metrics — a standard dashboard — 64.2% of experiments produce at least one significant result even though nothing has changed. Roughly two experiments in three hand you a finding to explain.

At fifty metrics it is 92.3%. An experiment examining fifty things is, for practical purposes, guaranteed to produce a result, and the only question is which one.

The simulated and theoretical columns agree to within a few tenths at every row. This is not a subtle empirical effect, it is arithmetic: each metric is an independent chance to be unlucky, and the chance of avoiding all of them is 0.95 raised to the number of metrics.

The correction, and what it costs

The blunt fix is to divide the threshold by the number of metrics — testing twenty things at 0.05 means judging each at 0.0025. It works exactly as advertised:

20 metrics, uncorrected:   64.2% false positive
20 metrics, Bonferroni:     4.8% false positive

Back to where a single test would be. But the correction is not free, and the price is the part that usually goes unmentioned. Taking a genuine one-point lift on the same sample size:

found uncorrected:          37.1% of the time
found under Bonferroni:      8.0% of the time

An uncorrected test would find this real improvement roughly once in three attempts; the corrected version finds it once in twelve. The correction did not make the experiment better, it moved the failure from one kind to the other.

That trade-off is not a flaw in the method, it is the reality of the situation. At a fixed sample size, examining more things means either more false alarms or less sensitivity, and no statistical technique creates information the sample does not contain.

What to do instead

Name the metric before you start. One primary metric, decided in advance and written down, is the only approach that costs nothing. Everything else on the dashboard is then explicitly secondary — worth looking at, not worth deciding on.

Treat the rest as questions for the next experiment, not answers from this one. A surprising move in a secondary metric is a reason to design a test focused on it. It is not a result, and the difference matters because the focused test will usually not reproduce it.

If the experiment really does have several primary metrics, apply a correction — just not this one. Bonferroni is used here because it is the easiest to verify, and it is the harshest. Holm's method gives the same guarantee with more power for free. Benjamini–Hochberg controls a different and often more appropriate thing — the share of your discoveries that are false, rather than the chance of any false discovery at all.

Or buy the sensitivity back with sample size. The correction costs power; more data returns it. That is the honest version of "we want to look at twenty metrics", and it has a price tag rather than a workaround.

How this was measured, and what it does not establish

Twenty thousand simulated experiments per metric count, five thousand observations per arm, both arms at a ten percent base rate, two-proportion z-tests at alpha 0.05, seeded and reproducible from the script in the repository.

The metrics here are independent, and yours are not. Conversion, revenue per visitor and add-to-cart move together, so the real inflation on a correlated dashboard is lower than the independent case. How much lower depends on the correlations, which means the 64.2% is an upper bound for twenty metrics rather than a prediction about any particular dashboard.

This measures one form of multiplicity. Many metrics is the visible one. Looking at many segments — by country, by device, by new versus returning — multiplies the problem the same way, and is far easier to do accidentally, because slicing a result feels like investigating it rather than testing again.

None of this means secondary metrics are meaningless. It means a secondary metric crossing a threshold is weak evidence on its own, and its strength depends on how many other metrics sat beside it — information the dashboard does not display.

Multiplicity is one way a result appears from nowhere. Selecting on a noisy measurement is another, and it produces movement without anybody running a test at all.