The peeking problem

The peeking problem is the inflation of the false positive rate that follows from checking a running experiment repeatedly and stopping it when the result turns significant.

Both the p value and the confidence interval assume a fixed procedure. The sample size is chosen before the test starts, the data is examined once at the end, and the five per cent threshold describes how often that one examination misleads. A dashboard that updates every hour breaks the assumption in the most natural way available, which is by letting a curious person look.

The pressure to look is not a character flaw. A test running for three weeks occupies traffic, holds up a release and costs money every day it continues, so somebody senior asks how it is going, and the answer is one click away. The cost of that click is arithmetic and it has been measured for more than fifty years.

The sections below explain what each additional look does, give the published figures for the number of checks against the false positive rate, and show why the habit leaves no trace a team can notice from inside. The page then covers the two designs that make early stopping legitimate, and closes on the rule that has to exist before a test starts.

What each additional look does to the arithmetic

A running test wanders. Early in its life the measured difference between the two groups swings widely, because a small number of users carries a large share of the total, and the swing narrows as the sample grows. That wandering is ordinary behaviour of a test in which nothing is happening.

Checking once at the end asks a single question of a single number. Checking ten times asks the same question of ten numbers along a path, and the test is declared a winner if any one of them crosses the line. Every look is another opportunity for the wandering to touch the threshold, so the chance of touching it at least once rises with the number of looks.

Armitage, McPherson and Rowe calculated the size of the rise in 1969, in a paper on repeated significance testing published in the Journal of the Royal Statistical Society. Their work came out of clinical trials, where a committee watching an accumulating series had every reason to stop as soon as a treatment looked better. The arithmetic transferred to software without a change.

The false positive rate against the number of checks

The figures below are for a test in which the change does nothing at all, with the checks evenly spaced through the run and the usual five per cent threshold applied at each one.

Two readings of the table matter to a product team. The first is how quickly the damage arrives, since a single extra look at the halfway point already takes five per cent past eight. The second is how ordinary the high rows are. A team that opens the dashboard on each working morning of a fortnight's test has looked ten times, and one in five of its null results will come back a winner.

Continuous monitoring is worse than the last row suggests, because a modern platform recomputes the result on every visitor. Ramesh Johari, Leo Pekelis and David Walsh reported that at a sample size of ten thousand, which is small for an online test, continuous monitoring can raise the error rate roughly fivefold. Their paper circulated from 2015 and appeared in Operations Research in 2022, and the applied version they wrote with Pete Koomen was presented at the KDD conference in 2017.

Why the habit leaves no trace inside a team

The inflation would be easier to notice if teams stopped tests in both directions. They do not. A test that looks bad on day three carries on, because somebody argues the sample is too small to judge, and a test that looks good on day three gets shipped, because the same argument reads as caution about good news.

That asymmetry puts the whole of the inflated error rate onto the winning side. The false positives ship and enter the record as successes. The false negatives were never acted on, so nobody counts them.

A programme run this way reports a higher success rate than it has, and no internal signal exists to say so. The corrective evidence arrives later and indirectly, through a shipped feature that fails to show up in the quarterly numbers, or through a holdout group that comes back showing half the gains the separate tests claimed.

Sequential testing as the legitimate route to stopping early

Stopping early is a reasonable thing to want, and the statistical literature has supported it since the 1970s. What the methods change is the threshold, which stops being a fixed five per cent at every look.

Group sequential designs fix the number of looks in advance and set a stricter threshold at each one, so that the total error across all the looks comes to five per cent. Stuart Pocock published one such scheme in Biometrika in 1977, using the same tighter boundary at every look. O'Brien and Fleming published another in Biometrics in 1979, with a very demanding boundary early in the test that relaxes towards the end, which suits a team willing to stop early only for a large effect.

Always valid p values remove the need to fix the looks in advance. The quantity reported is constructed so that it holds whatever stopping rule a user applies, which means a person may watch the test continuously and act the moment it drops below the threshold. Johari, Pekelis and Walsh developed the method for exactly this situation, and several commercial platforms now report it by default.

Both approaches cost sample size. A test that keeps the right to stop early needs more users to reach the same power as one that commits to a single look, and the extra is the price of the option. A team choosing between them is choosing whether the ability to stop early is worth a longer test in the cases where it does not stop.

The stopping rule that exists before the test starts

Everything above turns into one item of practice. The rule for ending a test belongs in the experiment brief, alongside the primary measure and the sample size, and it names either a fixed end point or a sequential method with the looks scheduled.

Writing it down changes what happens in a review meeting. A senior person asking how the test is going receives the planned end date and the reason for it, and the question of whether today's number is good enough never arises, because the team agreed in advance that today's number would not be examined. That is a much easier conversation to have before a test has produced an encouraging figure than after.

A rule that keeps the error rate honest still leaves one thing unchecked. Every figure above assumes the two groups were built correctly in the first place, and the cheapest way to find out whether they were is to count the people in each one.