How Long Should You Run an A/B Test?

There's no single universal number of days that's right for every A/B test — but there are clear, practical guidelines that will save you from the two most common mistakes: stopping too early and chasing noise, or running forever and wasting time on a test that already has a clear answer.

Why “until it's significant” is the wrong rule

The most common mistake is treating statistical significance as the stopping signal, checking the dashboard daily, and ending the test the moment the p-value crosses the threshold. This inflates your false positive rate substantially, because random fluctuation means a test can cross the significance line temporarily before settling back down — and if you stop right when it crosses, you've captured a lucky moment, not a real result.

The right approach is to decide your stopping point — either a sample size or a time duration — before the test starts, and let it run to that point regardless of what the dashboard shows along the way.

Factor 1: Sample size needed for your baseline conversion rate

The lower your baseline conversion rate, the more visitors you need to detect a given size of improvement. A page converting at 20% needs far less traffic to reliably detect a 2-point lift than a page converting at 2%, because rare events need more observations to distinguish a real signal from noise.

Most A/B testing calculators (freely available online) will tell you the sample size you need given your baseline rate, the minimum lift you care about detecting, and your desired confidence level. Running that calculation before starting the test, rather than guessing, is the single best habit you can build.

Factor 2: Full business cycles, not just sample size

Even if you hit your target sample size in three days, that's not necessarily long enough. User behavior often varies meaningfully by day of week (weekday vs. weekend behavior can differ substantially for many products) and sometimes by time of month (payday effects, billing cycles). A test that only spans a Tuesday through Thursday might miss how weekend users behave entirely.

Rule of thumb: Run for at least one full week, even if you hit your sample size target sooner, and ideally two full weeks if your business has any monthly cyclicality (e.g., subscription renewals, payroll-driven purchases).

Factor 3: Time-to-convert for your specific action

If the action you're measuring doesn't happen instantly — a free trial that converts to paid 10 days later, an enterprise sale with a multi-week cycle — your test needs to run at least as long as a typical conversion window, plus the time you spend collecting traffic into the test. Otherwise you're only capturing the visitors who convert fast, which can be a biased subset of your full population.

Rule of thumb:Test duration should be at least your traffic collection period plus your typical time-to-convert for the metric you're measuring.

Factor 4: Segment-level sample sizes, if you're stratifying

If you plan to look at results by segment, remember that each segment needs its own adequate sample size — not just the test overall. A test that's well-powered in aggregate can be badly underpowered for a smaller segment that only makes up 10% of your traffic. If a key segment is small, you may need to run the test longer than the aggregate sample size calculation alone would suggest.

A simple decision framework

  1. Calculate the minimum sample size needed using your baseline rate and the smallest lift worth detecting.
  2. Estimate how many days of traffic that requires, given your typical volume.
  3. Round up to at least one full week (two if there's monthly cyclicality).
  4. Add your typical time-to-convert if the action isn't instant.
  5. If stratifying, check whether your smaller segments will have enough volume by that point — extend if not.
  6. Set that as your fixed end date, and don't act on the result until you reach it.

The takeaway

The right test duration is the answer to a calculation, not a feeling. It's tempting to stop as soon as a result looks good, but the small amount of extra patience required to run a properly powered, full-cycle test is far cheaper than rolling out a change based on a result that doesn't hold up.

Stratafy tracks confidence over time and segment-level sample sizes automatically, so you can see exactly when a test — and each of its segments — has actually reached a trustworthy conclusion. See how it works.