Stratified Sampling vs. Random Sampling in Experimentation

Most A/B testing tools assign users to variants through simple random sampling: each visitor gets a coin flip (or equivalent) deciding whether they see A or B, with no regard for who they are. It's the default for a reason — it's simple, unbiased, and works well in a lot of cases. But it's not the only option, and understanding the alternative, stratified sampling, clarifies what you're actually trading off.

Random sampling: the baseline

In pure random assignment, every visitor has an equal, independent chance of landing in either group, regardless of their characteristics. Over a large enough sample, this tends to balance out — if your traffic is 60% mobile, both your A group and B group will end up close to 60% mobile purely by chance, given enough volume.

The catch is “given enough volume.” With smaller sample sizes, random assignment can produce groups that are accidentally imbalanced on some characteristic — not because of a flaw in the method, but because randomness doesn't guarantee even splits in finite samples. You might end up with a B group that happens to have more first-time visitors than your A group, purely by chance, and that imbalance can bias your result without you knowing it happened.

Stratified sampling: assigning within groups

Stratified sampling addresses this by first dividing the population into subgroups (strata) based on a characteristic you care about — device type, traffic source, new vs. returning, etc. — and then randomly assigning users to A or B within each subgroup separately, in proportion.

The result: if 60% of your traffic is mobile, you guarantee that roughly 60% of both your A group and B group is mobile too, rather than hoping random chance gets you there. This removes one specific source of sampling noise: imbalance on the stratifying variable.

Two different things people mean by “stratification” in testing

It's worth being precise here, because the term gets used for two related but distinct practices:

Stratified assignment— what's described above: ensuring balanced representation of a characteristic across your A and B groups at the point of assignment. This reduces the chance that an imbalanced split causes a misleading result.

Stratified analysis— analyzing results separately within each segment after the test has run, regardless of how assignment happened. This is what surfaces segment-level divergence (the “B wins overall but A wins for mobile” pattern). This is the practice most relevant to catching segment-masked results, and it's possible even without stratified assignment, as long as you tracked the segment data alongside the test.

Most modern stratified testing platforms, including the kind of analysis described throughout this blog, lean primarily on stratified analysis, since it doesn't require any special assignment logic and works retroactively on data you're already collecting.

When the distinction matters

For most product and growth teams, stratified analysis is the more practical and valuable of the two. It requires no change to how you run your test — only that you tag users with the relevant segment information so you can break results down after the fact. Stratified assignment matters more in smaller-sample, high-stakes experiments (clinical trials are the classic example) where even small imbalances in a small sample could meaningfully skew results.

The bottom line

Random sampling is a fine default and works well at scale. Stratified sampling — whether at assignment or analysis — is the upgrade you reach for when you want to either guarantee balanced groups (assignment) or understand whether your result holds up across the subgroups that matter to your business (analysis). For most teams, the second is where the real value is, and it's the layer most testing tools skip.

Stratafy applies stratified analysis automatically to every test, breaking results down by the segments you define without requiring any changes to how you assign traffic. See how it works.