A/B Test Significance Calculator
Enter control + variant conversions and users. Get z-score, p-value, a confidence interval for the lift at your chosen α, and a statistical-significance verdict. Free, no signup.
A/B Test Significance Calculator
After your A/B test runs, paste the results here. The tool computes the z-score, the p-value and a confidence interval for the difference at the same level as your α, and gives a ship/don't-ship verdict. It is the two-proportion z-test — the large-sample test for comparing two conversion rates in the NIST/SEMATECH e-Handbook (§7.3.3) — two-sided by default, one-sided if you planned for it before the test started.
Control (A)
Variant (B)
How to Use the Significance Calculator
Enter unique users + conversions per variant
Users = unique visitors exposed to the variant. Conversions = number of users who completed the goal action. Both must be from the same time window and segmentation.
Pick α and the test type
0.05 (95% confidence) is the convention; use a stricter α such as 0.01 for high-stakes decisions. Two-sided asks whether B differs from A in either direction; one-sided asks only whether B is better. Choose both before the test starts — picking α or switching to one-sided after seeing the results is p-hacking.
Read the verdict bar
Green = variant won at your chosen threshold. Yellow = not significant; either need more data or there's no effect. Red = variant lost; don't ship.
Sanity-check with the confidence interval
The interval for the absolute lift, at the same confidence as your α (95% at α = 0.05), gives the range of plausible true lifts. If it crosses zero, the lift could realistically be zero or negative. A wide interval means the test was too small to pin the effect down.
Interpreting A/B Test Results — Beyond the p-Value
What the p-Value Actually Means
The p-value is the probability of observing your test result (or a more extreme one) assuming the variant has zero effect. A p-value of 0.03 means: if the variant truly does nothing, we'd see this much (or more) difference in our test data only 3% of the time. Below the chosen α (typically 0.05), we reject the null hypothesis of zero effect and call the result "statistically significant." The p-value does NOT tell you the probability the variant actually beats control — that interpretation is wrong but extremely common.
What you really care about is "is the variant better?" Bayesian methods answer that directly ("85% probability variant > control"). The frequentist p-value is the closest off-the-shelf proxy, but always interpret it alongside (1) effect size — a 0.1% relative lift with p = 0.04 is statistically significant but practically meaningless; (2) confidence interval — if it barely crosses zero, the true effect could be tiny; (3) sample size — small-N tests are noisy regardless of p-value.
The Confidence Interval Is The More Useful Number
A 95% confidence interval of [+0.1 pp, +2.1 pp] for absolute lift tells you the plausible range of true lift much more clearly than a p-value. It includes information about effect size AND uncertainty AND statistical significance in one number — if the interval at the matching confidence doesn't include zero, the result is significant at that α. (Not quite always: the test uses the pooled standard error and the interval the unpooled one, so right at the boundary they can disagree. This page tells you when they do.) Modern Bayesian-style reporting often shows credible intervals instead of p-values for this reason. Even if you don't switch to Bayesian, training yourself to look at the CI first and the p-value second produces better A/B decisions.
Wide CI = more data would help. Narrow CI not crossing zero = confident win or loss. Narrow CI crossing zero = confident no-effect result. Wide CI crossing zero = test is inconclusive and underpowered. The CI also lets you decide whether the lift is large enough to matter operationally — a CI of [+0.01 pp, +0.5 pp] is significant but the lift is too small to justify the engineering work to ship.
"A p-value of 0.04 does not mean the variant has a 96% chance of being better. It means: if the variant did nothing, you'd see this much improvement in your data only 4% of the time. The interpretations look similar — they are not the same."
Common Failure Modes
Peeking: checking results as they come in and stopping the moment p drops below 0.05. Evan Miller (2010) worked an example: at a 50% conversion rate, testing after every visitor for up to 150 visitors and stopping at the first p < 0.05 wrongly finds a significant result 26.1% of the time, not 5%. Repeated looks always inflate the false-positive rate; checking less often inflates it less but never removes it. Either commit to the planned sample size (from the A/B test sample size calculator) or use a sequential method built for continuous monitoring, such as the always-valid p-values of Johari, Pekelis and Walsh. Multiple comparisons: test 10 independent metrics at α = 0.05 and the chance that at least one comes up significant by luck alone is 1 − 0.95¹⁰, about 40%. Use a Bonferroni correction (α / k for k tests) or pre-register one primary metric. Cherry-picked segments: a test that lost overall but won in the "iOS / California / Tuesday" segment is most likely noise found by searching. Validate segment wins on fresh data.
The Discipline of Pre-Registration
Pre-registration is the practice of writing down what you'll test, which metric is primary, what sample size you'll run, and what decision rule you'll apply BEFORE the test launches. It removes most of the freedom that turns a 5% false-positive rate into a much higher one. A simple template works: "We hypothesise variant X will lift primary metric Y by Z%. We will run N users per arm. At p < α, we ship; otherwise we don't." Stick to this even when results look exciting in unexpected metrics — those are hypotheses for the next test, not conclusions from this one.
Peeking turns a 5% error rate into 26%, and most ideas never win
p-value = probability of observed result, assuming variant has zero effect. NOT "probability variant is better."
The 0.05 threshold traces to R.A. Fisher (1925); not a fundamental statistical truth, just convention.
Two-proportion z-test: NIST's large-sample test for comparing two proportions; for small samples it points to Fisher's exact test.
Confidence interval is usually more useful than p-value: shows effect size + uncertainty + significance.
Statistical significance ≠ practical significance. A tiny lift can be significant in large samples.
Peeking inflates the Type I error rate — to 26.1% in Evan Miller's (2010) example of testing after every visitor, up to 150, and stopping at the first p < 0.05. Either commit to N or use sequential testing.
Bonferroni correction: divide α by the number of tests when running multiple comparisons.
Ten metrics tested at α = 0.05 give a 40% chance that at least one is "significant" by luck alone: 1 − 0.95¹⁰.
Bayesian A/B reports a "probability variant beats control" — a different question from the p-value, and not interchangeable with it.
Most ideas lose: at Microsoft only about one third of tested ideas improved the metrics they were designed to improve (Kohavi et al., Online Experimentation at Microsoft, 2009).
Frequently Asked Questions
- It means: if the variant had zero true effect, the probability of seeing the observed difference (or larger) is less than 5%. It does NOT mean "95% chance the variant is better" — that's a Bayesian interpretation that's wrong under frequentist methods. Combine the p-value with the confidence interval and effect size for a complete picture.
- Not on this evidence at α = 0.05. Options: (1) if the test has not reached the sample size you planned, let it finish — but extending a finished test because p came close is the peeking problem in another form; (2) accept "not significant" as the verdict: the test could not tell the variant from the control; (3) look at the interval — if it is mostly above zero, plan a larger follow-up test. Don't relax α after the fact; that is p-hacking.
- Pooled SE assumes both variants have the same true rate (i.e. tests the null hypothesis of zero effect) and is used for the p-value calculation. Non-pooled SE doesn't make that assumption and is used for the confidence interval of the actual difference. The two SEs are usually close, which is why this split is common practice; right at the boundary they can disagree (the test says not significant while the interval just excludes zero, or the reverse), and the page says so when that happens.
- The CI shows the plausible range of true lift in human-readable form. "Lift is between +0.5 pp and +2.0 pp" is much more useful than "p = 0.03". The CI also tells you whether significance is fragile (CI barely excludes zero) or robust (CI clearly on one side). Modern experimentation platforms increasingly lead with CI; the p-value is becoming a secondary number.
- Two-sided, almost always. A one-sided test only asks whether the variant WINS, so it cannot report a significant loss — and a losing variant is important to detect, because you don't want to ship it. The page offers one-sided for a directional hypothesis written down before the test started; switching to it after seeing the data halves the p-value for free and is p-hacking.
- It means your test doesn't have enough power to estimate the true effect precisely. Either the sample size is too small for the baseline rate, or the true effect is too small to detect cleanly. Use the sample-size calculator to compute how many more users you'd need. If the wide CI clearly excludes zero, the variant probably wins; if it crosses zero, you can't tell.
- Because the rate alone (e.g. "3.5%") doesn't determine statistical significance — sample size matters too. 3.5% from 100 users is much noisier than 3.5% from 100,000 users. The two-proportion z-test uses both raw counts to weight the noise correctly. Always pull the raw numerator + denominator from analytics rather than the pre-computed rate.
- Only if you did not stop it early because of what you saw. This is a fixed-horizon test: its p-value is valid for one look at a sample size chosen in advance. If you monitor continuously and want to be free to stop, you need a sequential method such as the always-valid p-values and confidence intervals of Johari, Pekelis and Walsh, which some commercial experimentation platforms implement. For "did the test we already ran to its planned size win?", this calculator is the right tool.
- This tool handles binary metrics only (converted yes/no). For continuous metrics like revenue per user, time-on-site, or basket size, you need a two-sample t-test (Welch's, if the two groups' variances differ) or a Mann-Whitney U-test instead. For a quick check, scipy.stats.ttest_ind in Python or t.test in R does it.
- Ask early which framework the client's experimentation platform uses, because the three common ones answer different questions: a fixed-horizon test like this one, a sequential test with always-valid p-values, or a Bayesian "probability to beat control". A p-value from this page is not interchangeable with either of the others, so state in the write-up which method produced each number. And agree on the reporting day: a test that "ran for 14 days" in Singapore time and in US Eastern time covers different hours of traffic.
Method & sources
How it computes
Two-proportion z-test as in NIST §7.3.3: z = (p̂₂ − p̂₁) ÷ √(p̂(1 − p̂)(1/n₁ + 1/n₂)), p̂ the pooled rate; two-sided p = 2·P(Z > |z|), one-sided P(Z > z), the normal tail from erfc to double precision. Interval for the difference with the unpooled SE at 1 − α, two- or one-sided to match. Flags expected counts under 5 (NIST §7.2.4) and a test/interval disagreement (AbTestKernel).
What this tool implements
- Two-proportion z-test, NIST/SEMATECH e-Handbook of Statistical Methods §7.3.3 (Case 1, large samples): pooled p̂; compare |z| with z₁₋α/₂ two-sided, or z with z₁₋α one-sided. For small samples the same section names Fisher's exact test, which the page points to.
- Small-sample flag: NIST §7.2.4's criterion min{Np₀, N(1 − p₀)} ≥ 5, applied to each arm at the pooled rate; a flagged result is still computed and labelled rough.
- Interval for the difference p̂₂ − p̂₁ uses the unpooled standard error at the same confidence (1 − α) and sidedness as the test, so a significant result is not printed beside an interval of a different level. Because the two standard errors differ, the page says when the test and the interval disagree.
- Normal tail: erfc by its power series (all terms positive) below x = 3 and its continued fraction above, upper tail computed directly; agrees with Python's math.erfc / statistics.NormalDist to 1e-12 relative and with effect-size-kernel.js (Hart 1968) to 1e-14 on [−8, 8]. Quantile by bisection on that CDF.
- Corrected 2026-10-16: interval fixed at 95% whatever α was; Abramowitz & Stegun 7.1.26 normal CDF (±1.5e-7 absolute) replaced; blank and comma-grouped counts read as 0 / 5 by parseFloat, α = 0 silently 0.05, a 0% control shown as +0% lift; one-sided test and small-sample and disagreement notes added; peeking "after every batch … about 26%" → Miller's every-visitor, up-to-150 example (26.1%); "10 metrics → expect one significant" → 1 − 0.95¹⁰ ≈ 40%; "p = 0.06: collect more data" (optional stopping) rewritten; an unsourced 14-day minimum, "similar sample needs" for Bayesian tests, "even in published academic work", and vendor claims (Optimizely Stats Engine, VWO, Statsig, Eppo, Meta, Airbnb, Stripe, Shopify) removed; Kohavi 2009 author list corrected to "Kohavi et al.".
Sources
- NIST/SEMATECH e-Handbook of Statistical Methods, §7.3.3: How can we determine whether two processes produce the same proportion of defectives? https://www.itl.nist.gov/div898/handbook/prc/section3/prc33.htm
- NIST/SEMATECH e-Handbook of Statistical Methods, §7.2.4: Does the proportion of defectives meet requirements? (min{Np₀, N(1 − p₀)} ≥ 5) https://www.itl.nist.gov/div898/handbook/prc/section2/prc24.htm
- Miller E. How Not To Run an A/B Test. 18 April 2010 (26.1% when testing after every observation, up to 150, at a 50% rate). https://www.evanmiller.org/how-not-to-run-an-ab-test.html
- Johari R, Pekelis L, Walsh DJ. Always Valid Inference: Bringing Sequential Analysis to A/B Testing. arXiv:1512.04922. https://arxiv.org/abs/1512.04922
- Kohavi R, Frasca B, Crook T, Henne R, Longbotham R, Lavista Ferres J, Melamed T. Online Experimentation at Microsoft. Microsoft ThinkWeek, 2009 ("only about 1/3 of ideas improve the metrics they were designed to improve"). https://ai.stanford.edu/~ronnyk/ExPThinkWeek2009Public.pdf
- Fisher RA. Statistical Methods for Research Workers. 1925, chapter III ("The value for which P = .05, or 1 in 20, is 1.96 or nearly 2"). https://psychclassics.yorku.ca/Fisher/Methods/chap3.htm
Pick up where you left off
Stored only in this browser — never sent to our servers.