Bench 12: A/B testing on low traffic
Small stores cannot run textbook tests, and pretending otherwise wastes months. Methods that respect small numbers: bigger swings, longer reads, honest stopping rules.
The bench guide
Working guides on this bench
A/B testing on low traffic Before-and-After Reads: The Small Store's Honest Alternative When a split cannot be read, a careful before-and-after comparison can still support a decision: same segment, whole weeks, shocks logged, and the limits stated in the same breath as the result. A/B testing on low traffic Choosing the First Test: Big, Plausible, Reversible Your first test sets the habits of the whole programme. Choose a change that is big enough to see, plausible enough to believe, and reversible enough that a loss costs a lesson rather than a quarter. A/B testing on low traffic Multivariate and Other Overkill: What to Skip at Your Scale Multivariate tests, personalisation engines and elaborate sequential designs all divide traffic you do not have. This guide names the overkill, the rare exceptions, and the simpler designs that serve a small store better. A/B testing on low traffic Reading Results Without Self-Deception: Peeking and Other Sins The commonest way a small test lies is not the maths; it is the reader. Peeking early, moving the finish line, switching measures and rescuing losses with stories all turn noise into false certainty. A/B testing on low traffic Test Offers, Not Cosmetics: Swing Size at Low Traffic At low traffic only large effects can be read, and large effects live in the offer: price framing, bundles, thresholds, guarantees and payment terms. Cosmetic changes move too little to judge and should ship on judgement or wait. A/B testing on low traffic Testing Tools on a Budget: What You Actually Need A small store needs a way to split traffic fairly, record both versions in analytics, and read step rates. Everything beyond that is optional, and some paid sophistication actively harms low traffic work. A/B testing on low traffic The Test Archive: Recording Losses at Full Size An archive that only holds wins is a trophy cabinet, not a learning system. Record every test and read at full size: hypothesis, conditions, numbers, losses and inconclusive results included. A/B testing on low traffic Time-Based and Sequential Tests: Splitting by Calendar Splitting by time, one version this week and the other next, is easy to run and easy to misread. This guide covers when calendar designs are acceptable, how to alternate fairly, and the traps that make them lie. A/B testing on low traffic Usability Sessions and Recordings: Evidence Stronger Than a Weak Test Five people thinking aloud on your key pages, and recordings of real visitors hesitating, often teach more than a split test your traffic cannot read. This guide covers running both without leading the witness. A/B testing on low traffic When Not to Run a Test: The Readability Gate Some tests should never start: the page is broken, the traffic is too thin to read any result, the change would be kept anyway, or trading conditions are about to shift. This gate saves the months those tests would have wasted. A/B testing on low traffic Writing a Hypothesis You Can Lose A useful hypothesis names the observation, the change and the expected effect, in a form that can turn out to be wrong. Vague hopes cannot lose, which is exactly why they cannot teach anything either.