Time-Based and Sequential Tests: Splitting by Calendar

A time-based test shows one version for a period, then the other, and compares the periods. It is the easiest design to implement, since nothing needs splitting, and the easiest to misread, because the two periods differ in more than the version: weekday mix, weather, payday timing, campaigns and news all sit inside the comparison. Used with discipline it can still inform large decisions: alternate versions across several whole weeks, keep every other condition as steady as trading allows, and read only effects large enough to survive the calendar's noise. Used casually, a single week against a single week, it is a before-and-after read with extra confidence it has not earned.

The weakest form is one switch: version A for a fortnight, version B for a fortnight, done. The stronger form alternates: A, B, A, B across whole weeks, so a one-off event lands in one block without deciding the whole read, and a slow trend shows up as drift across both. Stronger still is alternating by day of week matched blocks where the platform allows, though small stores rarely need that complexity. Every design shares the same requirement: the switch itself must be clean, with caches cleared, the change logged to the hour, and no other page change riding along. A calendar design already carries enough confounders without adding sloppy switches of your own.

Weekday composition is the classic trap: a period weighted toward weekends will out-convert a weekday-heavy period for many stores regardless of the version shown. Payday cycles, school holidays, weather that keeps people indoors or sends them out, and your own email sends all bend results the same way. Returning visitors add another: someone who saw version A last week and version B this week is not two independent visitors, and on tiny counts a few such people can move the read. None of these traps is exotic. They are ordinary trading, which is exactly why before-and-after reads and calendar tests must be graded with modest language.

Choose it when the platform cannot split the element at all, when the change is site-wide by nature, such as a delivery policy presentation, or when a simultaneous split would show different prices or terms to visitors who might compare notes, which creates a fairness problem a calendar avoids. It also suits sequential rollouts where each stage is a prerequisite for the next. In each case the decision the read supports should be proportionate: keeping or reverting a large change. Do not use a calendar design to arbitrate a small wording question. The instrument cannot see that finely, and asking it to only manufactures an argument.

Plan the blocks before starting: at least two blocks per version, whole weeks, with promotions and planned sends marked and, where possible, balanced across versions. Change nothing else on the page or the offer during the run. Log every switch with its date and time, and exclude switch days from the read if the change went live mid-day. At the end, compare each version's blocks together and also look at the blocks in sequence: if both versions improve steadily across the run, a trend, not the version, is doing the work, and the honest entry says so. That sequence check is cheap and catches the commonest self-deception in calendar testing.

Grade the conclusion in the language of the test archive: strong indication only when the effect is large, consistent across alternating blocks, and no logged event explains it. Anything less is a weak indication or no readable difference, with the decision labelled as judgement. Record the block map in the entry, because without it a later reader cannot tell an alternated design from a single switch, and the two deserve very different trust. Where the result will drive a costly commitment, treat a calendar read as a reason to run a simultaneous split later if one becomes possible, not as a substitute that settles the question forever.

Set against the bench's other tools, the calendar design sits below a readable simultaneous split and beside a careful before-and-after read, which it resembles with alternation added. It sits above an unlogged impression, because its windows, switches and events are at least written down. Its best role in a small business is as the design of last resort for large, unsplittable changes, run patiently and read modestly. The pillar guide places all these instruments on one scale deliberately: the skill is matching the strength of the claim to the strength of the design, every time, including when the result is the one you wanted.