What Is Stratified A/B Testing?

What Is Stratified A/B Testing? A Plain-English Guide

If you've run an A/B test before, you already know the basic idea: show half your visitors version A, half version B, and see which one performs better. It's simple, and that simplicity is exactly why it's so popular.

But it's also why standard A/B testing can quietly mislead you. Stratified A/B testing fixes a specific blind spot in the basic version — and once you see the blind spot, it's hard to unsee.

The problem with a single average

Imagine you run a test on your checkout page. Version B wins overall, with a conversion rate of 19.8% versus 14.2% for version A. You roll out B to everyone. Done, right?

Here's the catch: that 19.8% might be an average of two very different stories. Maybe desktop users loved B and pushed the average up — but mobile users, who make up a third of your traffic, actually preferred A. The overall number told you B won. It didn't tell you B won for some people and lost for others.

This is the core limitation of standard A/B testing: it gives you one number, and one number can hide a lot.

What stratification actually means

Stratification is a sampling and analysis technique borrowed from statistics more broadly, not invented for A/B testing specifically. It means dividing your population into meaningful subgroups — strata — before you analyze results, rather than treating everyone as one undifferentiated pool.

In A/B testing, that usually means segmenting by things like:

Once you have those segments, you don't just look at the overall winner — you check whether that winner holds up withineach segment. Sometimes it does, and you can be more confident in your result. Sometimes it doesn't, and you've just caught something the average would have hidden from you entirely.

A concrete example

Say an e-commerce site tests a new checkout button color. Overall, the new color wins by 3 percentage points. Looks like a clear result.

Break it down by traffic source, though, and a different picture emerges: visitors from paid ads responded well to the new color, but visitors arriving from email campaigns actually converted worse with it. The overall lift was real, but it was being driven entirely by one segment while quietly costing the business in another.

Without stratification, the team ships the new color everywhere and unknowingly hurts their email-driven conversions. With stratification, they catch it before rollout and can make a segment-specific decision — maybe even running two different experiences for two different audiences.

Why this matters more as you grow

Stratification matters more the more diverse your user base becomes. A five-person startup with a single traffic source has less to gain from segment-level analysis, simply because there's less variation to find. But once you have meaningful traffic from multiple channels, device types, or customer segments, the odds that a single average is masking a real split go up substantially.

This is also why stratified testing tends to produce more actionableinsights, not just more accurate ones. “B won” is a fact. “B won with returning desktop users but lost with new mobile visitors” is a decision you can actually act on — maybe with a segment-specific rollout, maybe with a follow-up test focused on the segment that didn't respond well.

Does this mean you need a statistics background?

No. The math behind stratified analysis (running the same significance test within each subgroup, checking sample sizes are large enough to trust the result) is something a good testing platform should handle for you. Your job is just to know what segments matter for your business and to look at the breakdown before you act on a result, not just the topline number.

The takeaway

A standard A/B test answers “which version performed better.” A stratified A/B test answers “which version performed better, and did that hold true across the groups I actually care about.” The second question is the one that actually protects you from rolling out a change that helps on average while quietly hurting a segment you didn't think to check.

Want to see this in action? Try Stratafy and run your next test with segment-level breakdowns built in from the start.