The precise definition
A p-value is the probability of observing a result at least as extreme as what you actually measured, assuming the null hypothesis is true (typically, that there's no real effect or difference). That's a deliberately narrow, conditional statement — and almost every common misinterpretation comes from dropping the "assuming the null hypothesis is true" condition.
The misinterpretation almost everyone makes
A p-value is not the probability that the null hypothesis is true, and it is not the probability that your result happened "by chance" in some general sense. It only tells you: if there really were no effect, how surprising would data like yours be? A small p-value means your data would be unusual under that no-effect assumption — it does not directly tell you the probability that there's a real effect, nor the size or practical importance of that effect.
p = 0.03 means: "If there were truly no difference, data this extreme (or more)
would occur about 3% of the time."
p = 0.03 does NOT mean: "There's a 97% chance the effect is real."
Why 0.05 is a convention, not a law
The common threshold of α = 0.05 — the level below which a result is called "statistically significant" — is a widely adopted convention (popularized by early 20th-century statistical practice), not a mathematically derived boundary between "real" and "not real." A p-value of 0.049 and 0.051 represent barely different levels of evidence, despite one crossing the conventional line and the other not. Some fields use stricter thresholds (particle physics famously requires much stronger evidence before declaring a discovery) precisely because the cost of a false positive is so high in that context — the "right" threshold is a judgment call about consequences, not a universal constant.
Statistical significance vs. practical significance
A large enough sample size can make even a tiny, practically meaningless difference statistically significant — technically real, but too small to matter for any actual decision. Conversely, a genuinely important effect can fail to reach significance with too small a sample, simply from insufficient statistical power to detect it (see the Sample Size Calculator for planning around this). A p-value alone says nothing about whether an effect is large enough to matter practically — that's a separate question requiring the actual effect size, not just its statistical significance.
What a p-value doesn't tell you
- Not the size of the effect. A tiny p-value can accompany a tiny, practically irrelevant effect if the sample is large enough.
- Not the probability the null hypothesis is false. This is a genuinely different quantity requiring Bayesian reasoning and a prior probability, which classical (frequentist) p-values don't provide.
- Not proof of anything. A single p-value from a single study is evidence, not proof — replication and effect size both matter for how much weight a result should carry.
Common mistakes
- Reading a p-value as "the probability the result is due to chance." It's conditional on the null hypothesis being true — it doesn't tell you the probability of that assumption itself being correct.
- Treating p = 0.05 as a hard, meaningful cliff. It's a widely used convention; a result just above or just below this line represents nearly identical evidence, not a qualitative difference.
- Reporting statistical significance without effect size. A significant result with a negligible effect size may not be practically meaningful, regardless of how small the p-value is.
FAQ
Does a p-value of 0.01 mean there's a 99% chance the effect is real?
No — it means that if there were truly no effect, data this extreme would occur only about 1% of the time. It says nothing directly about the probability the effect itself is real.
Why isn't 0.05 always the right threshold?
It's a widely used convention, not a mathematical law — some fields require much stricter thresholds when the cost of a false positive is especially high, and the "correct" threshold is a judgment call about consequences, not a fixed constant.
Can a result be statistically significant but not practically important?
Yes — with a large enough sample, even a tiny, practically meaningless difference can become statistically significant; significance and practical importance are separate questions.
Run a one-sample t-test, two-sample t-test, or chi-square test and get the exact p-value with the Statistical Significance Test Calculator — entirely client-side.