ABC Tools

The t-Test, Explained

After reading this you will know which of the four t-tests to run, how to read the t statistic and p-value, and how to check the assumptions before you trust the answer.

What a t-test does

A t-test compares means and asks whether the difference you see is larger than you would expect from sampling noise alone. Suppose you measure the length of parts from two machines. Machine A gives an average of 5.117 mm, machine B gives 5.917 mm. The gap is 0.8 mm. Is that a real difference, or could two random samples from identical machines look this different by chance?

The t-test answers by scaling the difference by its own uncertainty. It divides the observed effect by the standard error, which measures how much the estimate would wobble if you drew the samples again. The result is the t statistic. A t of 0.3 means the difference is a fraction of a standard error and easily explained by noise. A t of 5 means the difference is five standard errors out, which is hard to explain by luck.

This calculator runs four versions from raw numbers: one-sample, two-sample pooled, two-sample Welch, and paired. They share the same logic and differ only in how the standard error is built.

Choosing the right mode

Pick the mode from the design of your study, not from the data.

One-sample
You have one group and a fixed target. You test whether the group mean differs from a hypothesised value \mu_0. Example: a bag of chips is labelled 150 g. You weigh 20 bags and test against 150.
Two-sample, pooled (equal variances)
Two independent groups whose spreads you believe are similar. The two variances are combined into one pooled estimate.
Two-sample, Welch (unequal variances)
Two independent groups with no assumption about equal spread. This is the safer default and costs you almost nothing when the variances really are equal.
Paired
Two measurements on the same units, such as before and after a treatment. The test works on the differences A - B, so it removes the variation between subjects.

If two groups come from the same people, machines or plots measured twice, use the paired test. Pairing often cuts the standard error sharply because it cancels the differences between individuals.

When not to use a t-test

The t-test assumes the data (or the paired differences) are roughly normal and, for the pooled version, that the two groups share a variance. It also assumes independent observations. With small samples these assumptions matter more, because you cannot lean on the central limit theorem to rescue you.

If your data are heavily skewed or full of ties, a rank-based test is safer: the Mann-Whitney U test for two independent groups, or the Wilcoxon signed-rank test for paired data. For three or more groups, do not run many pairwise t-tests; use one-way ANOVA or the Kruskal-Wallis test. To check the equal-variance assumption before choosing pooled or Welch, use Levene's test, and to check normality try the normality test battery. For proportions rather than means, see the A/B test calculator.

The formula and the intuition

Every t statistic has the same shape: an estimate minus its target, divided by a standard error.

t = \frac{\bar{x} - \mu_0}{s / \sqrt{n}}

That is the one-sample form. Here \bar{x} is the sample mean, \mu_0 the hypothesised mean, s the sample standard deviation and n the sample size. The denominator s/\sqrt{n} is the standard error of the mean. It shrinks as n grows, so larger samples turn the same effect into a larger t.

For Welch's two-sample test the standard error combines both groups without pooling:

t = \frac{\bar{x}_A - \bar{x}_B}{\sqrt{\frac{s_A^2}{n_A} + \frac{s_B^2}{n_B}}}

The pooled test replaces both s^2 terms with a single weighted average variance and uses n_A + n_B - 2 degrees of freedom. Welch uses a fractional degrees of freedom given by the Welch-Satterthwaite formula, which is usually not a whole number.

Degrees of freedom set the shape of the reference distribution. With few degrees of freedom the t distribution has heavy tails, so you need a larger t to reach significance. At 5 degrees of freedom a two-sided test at \alpha = 0.05 needs |t| \ge 2.571; at 30 it needs only 2.042.

Worked example with the demo data

Two machines, Welch's test

Use the demo data. Sample A is 5.1, 4.9, 5.6, 4.8, 5.3, 5.0 and Sample B is 5.8, 6.0, 5.5, 6.2, 5.9, 6.1, with n_A = n_B = 6.

  1. Means: \bar{x}_A = 5.1167, \bar{x}_B = 5.9167. The difference is -0.8.
  2. Sample variances: s_A^2 = 0.08967, s_B^2 = 0.06967.
  3. Standard error: \sqrt{0.08967/6 + 0.06967/6} = \sqrt{0.02656} = 0.1630.
  4. t statistic: -0.8 / 0.1630 = -4.909.
  5. Welch degrees of freedom come out to about 9.8. The two-sided p-value is roughly 0.0007.

The p-value is far below 0.05, so you reject the hypothesis that the two machines have the same mean. The 95% confidence interval for the difference is about -0.8 \pm 2.26 \times 0.1630, or -1.168 to -0.432 mm. The interval excludes zero, which agrees with the small p-value.

Sample B sits above Sample A at almost every position, which is why the t statistic is large in magnitude.

Reading the results

The calculator reports the t statistic, degrees of freedom, one and two-sided p-values, Cohen's d and a 95% confidence interval. Read them together.

The p-value is the probability of a t at least this extreme if the null hypothesis (no difference) were true. Small p means the data are unlikely under "no difference". Compare it to a threshold you set in advance, commonly 0.05.

The confidence interval is often more useful than the p-value. It gives the range of plausible values for the true difference. In the example, plausible differences run from -1.168 to -0.432 mm, all clearly nonzero.

Cohen's d reports the effect size in standard deviations, independent of sample size. Roughly, 0.2 is small, 0.5 medium and 0.8 large. The demo data give a d near 2.8, a very large gap relative to the spread. A tiny p-value with a tiny d usually means a large sample detected a difference too small to matter in practice.

With a fixed mean difference of 0.5 and a standard deviation of 1, the t statistic grows with sample size. At n = 5 per group t is about 0.79 (not significant); at n = 20 it is about 1.58; at n = 64 it is about 2.83 and clears the 0.05 threshold. Larger samples turn the same real effect into a smaller p-value.

Common mistakes

Do not run a two-sample test on paired data. If the same subjects are measured twice, the two-sample test ignores the pairing, inflates the standard error and hides real effects. Use the paired mode, which works on the differences A - B.

Other frequent errors:

  • Choosing pooled without checking variances. If the spreads differ, pooling gives the wrong degrees of freedom. Welch is the safe default.
  • Picking the test after seeing the data. Deciding on one-sided versus two-sided, or which mode, based on the result inflates your false-positive rate. Decide before you look.
  • Reading a large p-value as proof of no effect. A p of 0.4 means you failed to detect a difference, not that the difference is zero. Check the confidence interval width.
  • Ignoring outliers. One extreme value can move the mean and inflate the variance. Plot your data first.

Related tools

To turn a t statistic from elsewhere into a p-value, use the p-value calculator. For categorical outcomes rather than means, see the chi-square test of independence, the goodness-of-fit test, Fisher's exact test for small 2×2 tables, McNemar's test for paired binary data, or the one-proportion z-test for a single proportion.

Frequently asked questions

Should I use one-sided or two-sided?

Use two-sided unless you have a reason set before collecting data to test only one direction. Two-sided is the default and the more conservative choice. In the demo data the two-sided p is about 0.0007; the one-sided p is half that, about 0.00035.

What is the difference between pooled and Welch?

Both compare two independent means. Pooled assumes the groups share a variance and uses n_A + n_B - 2 degrees of freedom. Welch makes no such assumption and adjusts the degrees of freedom downward. When variances really are equal the two agree closely, so Welch is the safer default.

My p-value is 0.06. Is the effect real?

The test found no significant difference at the 0.05 threshold, but 0.06 is not evidence of no effect. Look at the confidence interval. If it stretches from a large negative to a large positive value, your sample is simply too small to decide.

What sample size do I need?

It depends on the effect size you care about. To detect a Cohen's d of 0.5 at 80% power and \alpha = 0.05, you need roughly 64 per group. For d of 0.8, about 26 per group. Smaller effects need much larger samples.

Can I use a t-test on counts or percentages?

Not directly. For proportions use the two-proportion z-test. For counts across categories use a chi-square test. The t-test is built for continuous, roughly normal measurements.

Try it yourself in the tool

Related tools

Chi-Square Test of Independence

Test association between two categorical variables from a contingency table.

One-Way ANOVA Calculator

Compare the means of three or more groups with an F-test.

P-value Calculator

Convert z, t, χ² or F statistics into p-values.

A/B Test Significance Calculator

Two-proportion z-test with p-values, uplift and CI of the difference.

Chi-Square Goodness-of-Fit Test

Test whether observed counts match an expected distribution.

Fisher's Exact Test

Exact test for association in a 2×2 table — reliable for small counts.