How Much Traffic Do You Need Before Testing Means Anything?

A split test is readable when each version collects enough conversions for the difference you care about to stand out from ordinary luck. For most small stores that is a harder condition than expected: detecting a realistic improvement - say a 20 percent relative lift - usually needs hundreds of conversions per version. If your whole store makes 60 orders a month, a product-page test splitting that traffic will take many months to read, by which time seasonality has eaten the result. The honest options below that line are bigger changes, longer reads, and judgement-based fixes kept and watched - covered in the split bench.

Visitors are plentiful; conversions are the events whose rate you are comparing, so their count drives the noise. Illustration: a page converting at 2 percent sends 1,000 visitors to each version. Expect about 20 orders per version, but ordinary randomness alone can easily swing that by plus or minus 8 or 9. A real difference of 4 orders - a genuine 20 percent lift - drowns in that swing. Double the traffic and the swing grows slower than the signal, which is the only reason more traffic helps. There is no wording trick or tool setting that repeals this arithmetic.

Bench rules, stated as rules of thumb rather than laws. Below roughly 30 orders a month on the page type, do not split test at all: fix what evidence and judgement indict, and watch the step rates. Between roughly 30 and 150, test only large, structural changes (a new offer framing, a rebuilt page) where a big effect is plausible, and run for whole weeks across at least one full trading cycle. Above that, ordinary testing discipline starts to pay. Whatever the band, pre-commit to the read period and the decision rule before starting; moving goalposts convert noise into false wins.

Low traffic does not mean no evidence. Five alternatives produce real signal at small scale: session recordings and heatmaps showing where people hesitate; a short on-site or post-purchase survey asking what nearly stopped the buyer; usability sessions with five people from outside the business; a before-and-after read on the affected step rate with the change logged; and fixing known defects - errors, dead ends, missing information - which never needed a test to justify. The mistake is not skipping the split test. The mistake is running an unreadable one and calling the coin flip a result.

Online sample-size calculators are correct arithmetic applied to your inputs - and your inputs include a guessed effect size, usually the optimistic one. If you tell the calculator you expect a 50 percent lift, it will cheerfully approve a test your traffic can finish, but effects that large are rare outside of fixing something broken. Use calculators to discover what your traffic can detect, not to license the test you wanted to run. When the detectable effect is larger than any plausible effect, the calculator has given you an answer: do something else.

For each candidate test, write three lines in the log: orders per month on this page type, the smallest effect worth having, and the read period that follows from them. If the period runs past a seasonal change or a planned promotion, the test is not viable - schedule the fix instead. This small paperwork habit separates a store that learns from a store that merely experiments, and it costs four minutes.

When conversions are scarce, the only honest way to make a test readable is to test a bigger difference. A button colour change might move conversion by a few percent relative - invisible at your counts. A rebuilt product page with a new offer structure, clearer delivery terms and a different lead image might move it by thirty - visible. This is why the split bench advises low-traffic stores to test concepts, not cosmetics: bundle the change into one coherent variant whose plausible effect matches your detectable effect. The trade-off is attribution - you learn that the new page beats the old, not which element did it. At small scale that is a fair trade: knowing what works, in total, this quarter, beats knowing nothing precisely.