Statistical Significance Test Calculator
Runs three commonly needed hypothesis tests:
- One-sample t-test: does a sample's mean differ significantly from a hypothesized value?
- Two-sample t-test (Welch's): do two independent samples have significantly different means? Uses Welch's formulation, which doesn't assume equal variances between groups — a safer default than the pooled-variance Student's t-test.
- Chi-square test: do observed category frequencies differ significantly from expected frequencies?
All p-values are computed from first principles (regularized incomplete beta and gamma functions), verified against standard statistical tables.
What a p-value is, and the three things it is not
A p-value is the probability of seeing a result at least as extreme as yours if the null hypothesis were true. That conditional is the whole meaning, and almost every misreading comes from dropping it.
It is not the probability that the null hypothesis is true. It is not the probability your result happened by chance. And it is not a measure of effect size — with a large enough sample, a difference far too small to matter will produce a tiny p-value, because significance measures how confident you are that a difference exists, never how big it is.
So always report the effect alongside it. "Conversion rose from 4.0% to 4.1%, p = 0.01" is an honest sentence; "the change was statistically significant" alone is not.
Choosing the test
One-sample t-test compares one group's mean against a fixed value you specify — is this batch's weight different from the 500 g it is meant to be?
Two-sample (Welch's) t-test compares the means of two independent groups. Welch's version does not assume the two groups share a variance, which the classic Student's t-test does. Since that assumption is usually untested and often false, and Welch's loses almost nothing when it does hold, it is the better default.
Chi-square works on counts in categories rather than means — whether the split of outcomes across groups differs from what independence would predict.
The 0.05 threshold
0.05 is a convention from early twentieth-century agricultural statistics, not a law of nature, and p = 0.049 and p = 0.051 are essentially the same evidence. Treat the number as a continuous measure of surprise rather than a verdict.
Two practices protect you from fooling yourself: decide the test and the sample size before looking at the data, and remember that testing twenty hypotheses at p < 0.05 gives you roughly a one-in-two chance of at least one false positive.