Audit Mistakes: False Findings and Vanity Work

The five audit mistakes that produce false findings: auditing on unrepresentative data (a sale period, a traffic anomaly, a migration week) without flagging it in every finding it touches; writing preferences as findings ('the hero feels dated' with no rate, recording or quote behind it); inflating severity so the list gets attention, which destroys the list's credibility within two cycles; recommending repairs that cannot be verified (no stage rate, no read rule), so the audit can never be wrong and therefore can never be right; and auditing the pages instead of the funnel, producing forty page notes and no stage diagnosis. Each has a procedural antidote, and the antidotes are cheap because they are habits, not tools.

Traffic has weather. An audit run across a promotion, a press spike or a stock crisis will find leaks that are really events, and its repairs will be verified against a different season's data, producing verdicts nobody can interpret. Antidote: the audit's first page states its window and the window's known anomalies, and every finding inherits a window note where the anomaly plausibly touches it. Where the anomaly is total (a relaunch month), postpone the deep audit and run only the mechanical hour - defects are season-proof; rates are not.

Every auditor has taste; findings require evidence. The test is grammatical: a finding names a visitor behaviour or a rate ('visitors cannot state the delivery date after reading the product page - 6 of 8 cold readers') while a preference names a feeling about the page ('the product page feels cluttered'). Preferences are not banned; they are labelled as leads and routed to evidence (recordings, a cold read) before they may cost money. Audits that skip the labelling teach the business to stop believing audits, because the business can smell the difference even when it cannot name it.

When everything is critical, nothing is. Inflated severity is usually a bid for resources, and it works once. The cousin failure is the hundred-finding audit: exhaustive, unprioritised, and operationally identical to no audit, because a list that cannot be ordered cannot be followed. Antidotes: severity defined by task damage (blocks, slows, decorates) with examples, list length capped by the quarter's real capacity, and the deferred list carrying the remainder at full dignity. An audit's credibility is its scarcest asset; spend it on accuracy.

'Improve the visual hierarchy' cannot be verified, and what cannot be verified cannot be managed: it will be declared done whenever convenient and blamed whenever useful. Every recommendation must name the stage rate it expects to move and the read that will judge it, per the fix list format. Where no rate exists, the recommendation's first form is an instrumentation task. This rule feels bureaucratic for exactly one audit cycle; after the first verified win and the first honourable loss, it becomes the reason the business trusts the process.

Page-by-page auditing produces local optima and misses the chain: the product page polished while its traffic arrives mismatched, the checkout tuned while the cart leaks the intent the checkout needed. Antidote is structural: the audit opens with the funnel and its stages, findings are filed by stage, and page notes that do not attach to a stage are demoted to leads. The funnel-first order is the audit's spine; everything in this bench hangs from it, including the mistakes in this guide.

If verdicts keep embarrassing findings - repairs verified as noise, causes re-diagnosed twice - audit the audit: sample its findings and re-try each against its cited evidence, checking the window notes, the finding grammar and the read rules' arithmetic. The usual discoveries are procedural (evidence paraphrased instead of verbatim, severities set in the meeting rather than at the screen) and therefore fixable in the template rather than in the people. Record the meta-audit's corrections in the method document itself, so the method carries its own scar tissue. A programme that can correct its instrument is worth trusting with bigger questions.