FreeLesson 1 of 36 min read
What a p-value actually means
Learn the precise definition of a p-value, how to calculate a simple one, and the common misreadings to avoid.
P-values appear everywhere: in research papers, A/B test dashboards and machine learning reports. They are also one of the most misunderstood numbers in statistics. If you can say exactly what a p-value is, and what it is not, you will read data claims far more critically than most people.
Start with a null hypothesis
Every p-value is calculated relative to a null hypothesis. This is a default claim, usually that there is no effect or no difference. Examples: 'this coin is fair', 'the new button does not change click rates', 'the drug works no better than a placebo'.
We then ask a narrow question. If the null hypothesis were true, how surprising would our data be?
The formal definition
A p-value is the probability of getting a result at least as extreme as the one you observed, assuming the null hypothesis (and the model's other assumptions) is true.
Three parts of that sentence matter. 'At least as extreme' means we count our result plus every more extreme one. 'Assuming the null is true' means the calculation lives entirely inside an imagined world where there is no effect. 'The model's assumptions' means things like independent observations; if those fail, the p-value can mislead.
Example: testing a coin
You flip a coin 20 times and get 16 heads. Null hypothesis: the coin is fair, so P(heads) = 0.5.
Step 1: Define 'at least as extreme' (one direction): 16, 17, 18, 19 or 20 heads.
Step 2: Use the binomial formula. P(X = k) = C(20, k) / 2^20.
C(20,16) = 4845, C(20,17) = 1140, C(20,18) = 190, C(20,19) = 20, C(20,20) = 1. Total = 6196.
Step 3: 2^20 = 1,048,576, so p = 6196 / 1,048,576 ≈ 0.006.
If you also count equally extreme results in the other direction (4 or fewer heads), the two-sided p-value is about 0.012.
Interpretation: if the coin were fair, a result this lopsided would happen only around 1 time in 100. That is surprising, so it counts as evidence against fairness.
What a p-value is not
These readings are tempting but wrong:
- It is not the probability that the null hypothesis is true.
- It is not the probability that your result happened 'by chance alone'.
- It is not a measure of how big or important the effect is.
- A large p-value does not prove the null is true; it only means the data are not very surprising under it.
Why can't p = 0.006 mean 'a 0.6% chance the coin is fair'? Because the p-value assumes fairness from the start. Turning it around to get the probability of the hypothesis needs extra information, such as how likely biased coins were before you looked. That is a different (Bayesian) calculation.
About the 0.05 threshold
Many fields call a result 'statistically significant' when p < 0.05. This cutoff is a convention, not a law of nature. A p-value of 0.049 and one of 0.051 carry almost the same evidence. Also, if you run many tests, some will fall below 0.05 even when no real effect exists. This is why testing dozens of features or metrics and reporting only the 'significant' ones is risky.
Next time you see a p-value, translate it into a full sentence: 'If there were no effect, data this extreme would appear about X% of the time.' If you can't write that sentence, you don't yet know what the number is claiming.
Key takeaways
- A p-value is the probability of data at least as extreme as yours, assuming the null hypothesis is true.
- It is not the probability that the null is true, and it says nothing about effect size.
- The 0.05 cutoff is a convention, and running many tests inflates false positives.
Check your understanding
Written with AI help. Spotted a mistake? Tell us.