How to Read a P-Value Without a Statistics Degree

If you've ever stared at an A/B testing dashboard and seen “p = 0.03” with no idea what to do with that number, you're not alone. P-values get explained badly more often than almost any other statistical concept, even though the core idea is genuinely simple once it's framed the right way.

What a p-value is actually answering

A p-value answers one specific question: if there were truly no difference between A and B, how likely would it be to see a result this big (or bigger) just by random chance?

That's it. It's not “the probability B is better than A.” It's not “how confident you should be.” It's specifically a measure of how surprising your result would be if nothing real were going on at all.

A p-value of 0.03 means: if A and B were actually identical, you'd see a difference this large (or larger) only 3% of the time by random luck. A p-value of 0.40 means you'd see a difference this size 40% of the time even with no real effect — which means your result could very easily just be noise.

Why 0.05 became the line in the sand

You'll often see 0.05 (5%) used as the threshold for “statistically significant.” This isn't a law of nature — it's a convention. It means: we're willing to accept a 5% chance of being fooled by randomness. There's nothing magic about it; some fields use stricter thresholds (medical research often uses much smaller thresholds because the cost of a wrong conclusion is higher), and for low-stakes tests a looser threshold can be a reasonable trade-off for speed.

The important thing isn't memorizing 0.05 — it's understanding that a p-value is a measure of how surprising your result is under the assumption that nothing is really happening, not a measure of how big or important the effect is.

A common mistake: confusing significance with magnitude

A test can be statistically significant (low p-value) while the actual difference is tiny and not worth acting on — this often happens with very large sample sizes, where even a trivial 0.1% difference can become “significant” just because you have enough data to detect it precisely.

Conversely, a test can show a meaningful-looking difference that isn't statistically significant simply because the sample size is too small to be confident it's not just noise.

This is why a p-value should never be read alone. Always look at it alongside the actual size of the effect (how big is the difference, in real terms?) and the sample size (how much data is this based on?).

A simple mental model

Think of a p-value like a smoke detector's sensitivity. A very sensitive detector (low p-value threshold) catches real fires reliably but also goes off sometimes for burnt toast. A p-value tells you how “burnt toast”-like your result could be — how plausible it is that what you're seeing is just noise dressed up as a pattern.

When the p-value is low, the smoke detector is telling you this probably isn't toast. When it's high, there's a real chance you're reacting to nothing.

What to actually do with a p-value in practice

Check it's below your threshold (commonly 0.05, but use whatever bar makes sense for the stakes of your decision).
Check the effect size makes sense to act on — a statistically significant 0.2% lift might not be worth the engineering effort to ship.
Check your sample size is reasonable— very small samples can produce misleadingly low p-values by chance, especially if you peek at results early and stop as soon as you see a good number (a common and easy mistake called “peeking”).
If you're stratifying results, check the p-value within each segment, not just overall — a result that's significant in aggregate can be driven entirely by one segment, while another segment shows no real effect at all.

The takeaway

A p-value isn't a verdict, it's one input into a decision. Once you know it's measuring “how surprising would this be if nothing were really going on,” you can use it for what it's actually good for: a sanity check against mistaking noise for a real pattern, not a magic number that tells you what to do.

Stratafy calculates p-values automatically — overall and per segment — so you don't need a statistics background to know when a result is trustworthy. See how it works.