A two-sample t-test asks whether two groups' averages differ by more than their scatter would explain. Student's version pools the two variances; Welch's keeps them apart. Use Welch's unless you have a reason not to. We computed each test's false-positive rate exactly, for normal data with no real difference and a nominal 5%. When a group of 10 had four times the standard deviation of a group of 40, the equal-variance test claimed a difference 29.4% of the time. Swap the group sizes and it did so 0.08% of the time. Welch's stayed within 0.05 of a point of 5% in both. Neither test is safe everywhere.
Small samples posed Student's question in 1908
"Student" published "The Probable Error of a Mean" in Biometrika in 1908. In the University of York's transcription, the aim is "to determine the point at which we may use the tables of the probability integral" in judging a mean. His statistic was z, with n rather than n − 1 in the standard deviation; today's t is z multiplied by √(n − 1). He checked it by shuffling measurements written on "3000 pieces of cardboard".
Every illustration in the paper is a single series or a column of differences. Nothing in it pools two independent groups. That form is in Fisher's Statistical Methods for Research Workers of 1925, where the standard deviation "is estimated by pooling the sums of squares from the two samples" (p. 110).
Pairing puts the question to each patient in turn
Student's first illustration is the extra sleep of ten patients under two forms of one drug, as he labelled them, taken from a 1904 table by Cushny and Peebles. He settled it by "subtracting 1 from 2", a column of differences averaging 1.58 hours, and put the odds at "about 666 to one". In Example 20 of his 1925 book, Fisher re-read the figures as if from "different patients" and found a result that "cannot be regarded as significant".
Both readings can be re-run on our t-test calculator. Enter 1.9, 0.8, 1.1, 0.1, -0.1, 4.4, 5.5, 1.6, 4.6, 3.4 as group A and 0.7, -1.6, -0.2, -1.2, -0.1, 3.4, 3.7, 0.8, 0, 2.0 as group B. Paired mode returns t 4.06 on 9 degrees of freedom, p 0.0028 two-tailed. Welch's two-sample mode gives t 1.86 on 17.78 degrees of freedom, p 0.0794. The columns correlate at 0.795, meaning patients who slept longer on one drug also slept longer on the other. Pairing removes that shared variation. The table is a 1908 record, silent on either drug today.
Before a revision dated 3 November 2026, the calculator graded p against fixed bands whatever α was chosen, so at α 0.01 one screen could say "Significant" beside "fail to reject", and its FAQ credited the 1908 paper with an equal-variance two-group test.
Equal-variance testing answers yes too readily or hardly at all
Answering yes means claiming a difference. The pooled test weights each group's variance by its degrees of freedom, one fewer than its size, so the larger group's spread dominates. If the larger group is the quieter one, the pooled figure understates the small group's noise; reversed, the test turns timid. Our rates below assume normal data, no true difference and a nominal 5%, computed exactly by numerical integration.
| Group sizes | SD, first group to second | Student's, % | Welch's, % |
|---|---|---|---|
| 10 and 10 | 4 to 1 | 6.1 | 5.1 |
| 30 and 30 | 4 to 1 | 5.4 | 5.0 |
| 10 and 40 | 4 to 1 | 29.4 | 5.0 |
| 40 and 10 | 4 to 1 | 0.08 | 5.0 |
| 20 and 80 | 4 to 1 | 28.5 | 5.0 |
Doubling both groups barely moves the rate: the ratio of group sizes drives the error, and more data does not cure it. The direction is not our finding. Delacre, Lakens and Leys wrote in 2017 that "When the larger variance is associated with the smaller sample size, the Type 1 error rate is inflated" and argued for Welch's as the default.
Welch's test replies with less power when small groups are lopsided
With truly equal spreads Welch's gives up some power. We computed power exactly, by the same integration, at a true difference the pooled test catches about 80% of the time. At 10 per group the cost was under half a point. At 10 and 40, Welch's caught it 74.7% of the time against 79.1%. At 5 and 20 the gap was 11 points, and Welch's false-positive rate there is 5.6% on our exact calculation.
Skew changes the ranking. With both groups, of 10 and 40, drawn from one lognormal distribution (σ = 1, deliberately harsh), Welch's gave 8.3% false positives in our simulation and the pooled test 4.7%. The authors' 2022 correction said they could not generalise their conclusions to samples of different sizes drawn from highly skewed distributions, a region that takes in this case. They did not find Welch's safe there, and neither did we.
Thirty observations leave the skewed-data question open
A familiar rule says that past about 30 observations the normal curve can stand in for t. Judging t against the normal curve's 1.96 gives 8.2% false positives at 9 degrees of freedom, by exact calculation, and 6.0% at 29. The rate does not fall below 5.1% until 278 degrees of freedom. Thirty is one point on a smooth curve.
Skew matters more for a one-sample test. In our simulations, a one-sample t-test of the true mean on lognormal data with σ = 1 rejected about 11.8% of the time at n = 30 against a nominal 5%, and still 7.0% at 200. At n = 30, 98% of the lognormal false positives fell in the lower tail, so the two-sided test behaved almost like a one-sided test pointed the wrong way. Two equal groups of one shape fared better: lognormal groups of 30 each gave 4.1% pooled and 3.9% Welch.
Ranks answer a different question
The usual fallback, the Mann-Whitney rank test, does not test means, and unequal spread still disturbs it. On normal data with equal means, in 200,000 simulated runs per case using its normal approximation, it rejected about 15% of the time with groups of 10 and 40 when the small group was four times as spread out, and under 1% when the large one was. A false positive here means rejecting equal means: the reader's question, though the rank test asks another.
P-values leave most readers' questions unanswered
A p of 0.0028 for the sleep table says ten differences this consistent would be rare if the two treatments did not differ on average. It says nothing about how likely a difference is, or whether 1.58 hours mattered to anyone.
In March 2016 the American Statistical Association released six principles on p-values. Its news release quotes its president calling this the first statement on the issues from "the community of statisticians, as represented by the ASA Board of Directors". P-values "do not measure the probability that the studied hypothesis is true". Conclusions and business or policy decisions "should not be based only on whether a p-value passes a specific threshold". And a p-value "does not measure the size of an effect or the importance of a result".
View author profile → · Editorial policy
