How Long to Run a Change Before You Judge It

Run a change for at least one full trading cycle - usually two to four whole weeks for a small store - and until the affected step has collected a meaningful count of conversions on the new version, then compare against your baseline for the same segment. Stop early only if the change is plainly broken. Judging sooner reads noise as signal: at small counts, a good change can look bad for a week and a useless one can look brilliant. The discipline is to decide the window and the decision rule before the change ships, write them in the log, and abide by them when the early numbers argue.

Small-store trading has a weekly heartbeat: weekend browsers, payday spikes, a quiet Tuesday pattern. A read of ten days contains one weekend; a read of five days might contain none, and both mislead in opposite directions. Whole weeks are the smallest honest unit; two to four of them average weekday mix, a newsletter send or two, and ordinary weather. Seasonal businesses need more care: a change read across a season boundary (into a sale period, into the holidays) cannot be separated from the season, and the honest entry in the log is 'unreadable - kept on judgement, recheck in the new season'.

Time alone is not the criterion - events are. A change to a checkout reassurance line should be read on checkout completions: if the store completes 40 checkouts a week, two weeks gives a readable count for a large effect. A change to a rarely visited page might collect twelve relevant conversions in a month, and no window fixes that; the read stays provisional however long you wait. State both conditions in the log entry: 'read after four whole weeks and at least 60 checkout completions on the new version, whichever is later'. That sentence, written in advance, is the difference between measurement and vibes.

Some events poison a comparison: a promotion running in one period and not the other, a stock-out of a hero product, a traffic source starting or stopping, a price change, an outage. When one lands mid-read, you have three honest options: extend the read until a clean window exists, abandon the read and judge on other evidence, or keep the change on judgement with the read marked invalid. The dishonest option - quoting the numbers anyway because they favour the change - is the one that slowly destroys a change log's value. Future you reads that log; write it for them.

Not every change earns a verdict. For fixes of known defects (a broken link, a missing delivery cost, a form error that ate submissions), no read is needed: the fix is correct on its face, and the step rate is watched afterwards to confirm recovery. For judgement changes at traffic too low to read, 'kept and watched' is a legitimate log outcome: the change stays, the relevant step rate is reviewed monthly, and a sustained move against it reopens the question. What is not legitimate is upgrading kept-and-watched to 'tested and proven' in your own head. The log's honesty is the asset.

Break the window rule in exactly one case: evidence the change broke something - orders or step conversions falling off a cliff in a pattern that cannot be noise, errors in recordings, customer complaints naming the page. Then revert first and analyse second; revenue does not wait for a full cycle. Distinguish breakage from a merely disappointing early read by magnitude and mechanism: a cliff with a visible cause is breakage; a soft week inside normal variation is weather. Your baseline's normal variation, recorded in advance, is what makes this judgement possible.