Multiple comparisons

Multiple comparisons are what a team creates whenever one experiment is judged against more than one measure, and they turn the five per cent threshold into a much weaker guarantee than its name suggests.

Every page so far has treated a result as one number examined once. A real experiment report carries dozens. A modern platform computes every metric in the catalogue for every test, breaks each one down by country, device, plan and tenure, and presents the whole grid with the significant cells coloured.

The colouring is where the damage starts. A five per cent threshold means one comparison in twenty comes back significant when nothing is happening, so a grid of two hundred cells will light up about ten of them on a change that does nothing at all. Somebody then reads the coloured cells and builds an explanation for each.

The sections below give the arithmetic for a modest number of measures and for a realistic segment grid, then show what the same fault looks like when one person performs it one analysis at a time. The page covers the two standard corrections and closes on the practice that makes both of them unnecessary for the decision itself.

The arithmetic of testing many measures

The calculation is short. A single comparison at the five per cent threshold has a 95 per cent chance of behaving, so twenty independent comparisons have a 95 per cent chance each, and the chance that all twenty behave is 0.95 multiplied by itself twenty times.

The middle column is the one to hold onto, because it needs no probability theory to explain to a stakeholder. Twenty measures on a change that does nothing produce one false winner on average. Fifty produce two or three. A team reporting fifty measures and highlighting the ones that moved has built a machine for generating findings from noise.

Two qualifications make the real picture slightly different without changing the conclusion. Product measures correlate with each other, since signups, activation and revenue move together, so the true chance sits somewhat below the table. And a change that genuinely does something will show real movement in several places, which is a different situation from the one the table describes. Neither qualification helps a team that reports everything and explains whatever lit up.

Slicing by segment as the same arithmetic

The segment grid multiplies the problem quietly, because each cut looks like a separate question rather than another draw from the same lottery.

A payments product runs one test and reports twelve measures. The analysis breaks each measure down by eight segments, which are three regions, three device types and two pricing plans. That is 96 comparisons. On a change with no effect at all, the expected number of significant cells is 4.8, and the chance of seeing at least one is above 99 per cent.

What happens next is the part worth recognising. The team finds that the change helped mobile users in one region on the higher plan, at a p value of 0.03, and an explanation arrives within minutes, because a plausible story exists for every cell in a grid that size. The finding is then written down, and the next roadmap includes work built on it.

The defence is not to stop looking at segments. Segment analysis is how a team finds the population a change actually helps. The defence is to treat every segment finding as a hypothesis for a new test and never as a result, and to say so in the write up in those words.

P hacking inside a product team

P hacking is the same arithmetic performed by one person making choices, usually without any intention to deceive. Each choice looks reasonable on its own, and the set of them searches a space of analyses until one returns a number below the threshold.

The choices are familiar to anybody who has run an analysis.

1. Removing outliers after seeing what removing them does to the result.
2. Switching the measure from total revenue to revenue per active user because the first one came back flat.
3. Dropping the first three days as a warm up period once the first three days turn out to be the bad ones.
4. Adding a covariate, a filter or an exclusion that nobody planned.
5. Running the test for another week because the number is close.

Joseph Simmons, Leif Nelson and Uri Simonsohn measured what a handful of these choices does, in a paper published in Psychological Science in 2011. Combining a few of them raised the false positive rate from five per cent to over sixty. Their subject was academic psychology and the mechanism transfers without modification to a product analyst with a notebook open.

The signature of the fault is an analysis plan that grew during the analysis. A team that can produce the document it wrote before the test, and show that the analysis matches it, has ruled the whole family out in one move.

The two corrections and what each costs

Statistics offers two established ways to hold the error rate down across many comparisons, and they answer different questions.

The Bonferroni correction divides the threshold by the number of comparisons. For the 96 cell grid above, the threshold becomes 0.00052, and a result has to be far stronger to count. The correction is simple, it is safe and it is severe, because it protects against any false positive anywhere and pays for that with an enormous loss of power. Real effects in small segments disappear entirely.

The false discovery rate answers a softer question, which is what share of the findings declared significant are false. Yoav Benjamini and Yosef Hochberg published the procedure in 1995, in the Journal of the Royal Statistical Society, and it ranks the p values and applies a sliding threshold. It accepts that a stated proportion of the declarations will be wrong in exchange for finding far more of the real effects. Genomics adopted it for exactly this reason, and experimentation platforms now use it to control the noise in a metric catalogue.

Neither correction rescues a decision that was made by scanning the grid. Both control how often a declared finding is false, and neither one restores the honesty of a process that chose its measure after seeing the numbers.

The primary measure decided in advance

What makes a programme trustworthy is the same discipline that made the hypothesis worth writing down. One measure is named before the test runs, the decision rests on that measure alone, and the practical threshold it has to clear is recorded next to it.

Everything else on the report keeps a job and loses its vote. Guardrails are watched for damage, because looking for harm across many measures is the direction where a false alarm costs a conversation and a missed signal costs a customer. Secondary measures explain the primary one and describe what happened. Segment cuts generate candidates for the next test.

Writing that hierarchy down converts a grid of two hundred cells from a threat into a resource. The decision has one input that nobody can argue about after the fact, and the other 199 cells become material for the next round of questions, which is what they were always good for.

A programme that reports one honest number per test has done the analysis. It has not yet made a decision, and the step from a trustworthy result to a choice between shipping, iterating and abandoning is where most of the value either appears or is lost.