Reading Results Without Self-Deception: Peeking and Other Sins

Read a test only when the period and conversion count you committed to have arrived, on the measure you named in advance, against the segment you planned to judge. Peeking early and stopping on a lucky spike, extending a losing test until it turns, switching to whichever measure happened to move, and slicing segments until one looks good are the standard ways honest people deceive themselves with real dashboards. Each feels like diligence in the moment. Together they guarantee that noise gets promoted to knowledge and filed in the archive as fact. The defence is procedural, not moral: write the rules before the test, keep them visible during it, and let the reading be almost boring.

Early in a test, each new conversion moves the rates dramatically, so the lead changes hands repeatedly and the graph looks dramatic daily. Checking it is human; acting on it is the error. Stopping early on a spike banks a random high point, which later trading will usually take back, leaving you with a kept change that never truly won. Set a check rhythm that serves operations only: confirm the test is running, both versions record correctly, and nothing is broken. Judge nothing until the planned read date. If the temptation to peek is constant, restrict the dashboard view or have the read date in the calendar as the first moment the result tab gets opened.

The mirror sin is extending a test that is losing, in the hope that more time will rescue the preferred version. Sometimes an extension is legitimate: conversions arrived slower than planned, and the pre-written rule said whichever is later. That is following the rule, not moving it. The illegitimate version is deciding after seeing the result that the window was wrong, the season was odd, or one more week feels fairer. Those judgements may even be right, but made after the fact they cannot be distinguished from wishful thinking, including by you. If conditions genuinely invalidated the read, declare the test invalid in the archive, with the reason, and start clean if the question still matters.

A test with one primary measure can disappoint; a test examined across ten measures will always find one that moved, by chance alone. The same applies to segments: slice by device, source, new and returning, day of week, and somewhere a flattering subgroup will appear. Subgroup patterns can be worth a follow-up hypothesis, tested fresh. They are not the result of the test you ran. The reading discipline is simple to state: the primary measure decides, guard measures can veto, everything else is a note for future work. Write that order into the log entry before launch, following the hypothesis template, so the reading meeting, even a meeting of one, has an agenda it cannot quietly rewrite.

At low traffic, many honest tests end with an effect too small to separate from noise inside the planned window. Recording that plainly protects the business twice: it stops a coin flip from becoming a permanent change with a false pedigree, and it tells you the swing was too small for your counts, which points to a larger change next time. The options after an inconclusive read are keep on judgement, revert on judgement, or retest with a bigger swing, and each is respectable if labelled honestly. What is not respectable is rounding an inconclusive read up to a win in conversation. The pillar guide treats this outcome as a normal cost of learning at small scale.

Stock-outs, outages, a mention that doubles traffic, a competitor closing: real trading does not pause for your experiment. When a shock hits one part of the run, note the dates in the log and judge whether both versions lived through the same conditions. A simultaneous split usually shares shocks fairly, which is one of its strengths over before-and-after reads, where a shock can impersonate the change entirely. If the shock plainly swamps the effect you were hunting, the clean act is to void the test rather than interpret through the noise. A voided test costs weeks. A misread test costs the changes built on its false lesson, which is the more expensive bill.

On the read date, work in this order. Restate the hypothesis and decision rule aloud or in writing. Report the primary measure for the full planned period only. Check the guards. Note any logged shocks or defects. Make the decision the rule dictates, or explicitly declare the read invalid and say why. Write the entry, including the numbers at full size, before discussing explanations. Explanations come last because they are the most enjoyable part and the least reliable. Kept in that order, reading a test takes twenty minutes and produces an archive entry your future self can trust, which is the entire point of running the test at all.