The duration of an experiment is the number of days it stays running before anybody acts on the result, and the exposure is the set of people whose behaviour ends up in the comparison.
A criterion, a sample size and a set of guardrails describe what a test measures. Neither of those decides when to stop, and the day the numbers first look convincing is almost never the right answer.
The convention that a test runs for whole weeks came out of the first large online experimentation programmes in the 2000s. Teams there found that a result computed on a Wednesday disagreed with the same result computed on a Sunday, for reasons that had nothing to do with the change. Kohavi and colleagues recommended in their published guidance that a test should cover at least one full week and then extend in whole weeks.
For a product manager, the practical shape of this is a promise made to somebody else. A launch date, a board meeting or a marketing campaign depends on a decision, and the honest answer to when the decision will be ready is a calculation and never an estimate. Getting it wrong in the optimistic direction produces the worst outcome available, which is a decision taken on half a test.
The sections below set out the three things that fix a duration. Whole weeks and the weekly cycle follow, then the exposure point that decides who is counted at all. The last sections cover ramping a change before a full test and what happens on the day a number turns significant.
The three things that fix a duration
The sample size the calculation asked for. A test needing 38,400 people per group at a product receiving twelve thousand new signups a week collects six thousand per group per week, so it needs 6.4 weeks. Nothing about watching the chart changes that figure.
The whole week rule. 6.4 weeks becomes seven weeks, because a test that stops partway through a week has a final slice weighted towards whichever days it happened to include.
The time the effect takes to appear. A change to an onboarding step shows up in signup conversion within a day and in retention at day thirty after thirty days. If the criterion is measured over a period, the test has to run long enough for the last person assigned to have lived through that period.
The third of these is the one teams forget. A test on day seven retention cannot report anything useful until seven days after the final assignment, so a four week test on that criterion actually occupies five weeks of calendar.
Whole weeks and the weekly cycle
A product's population changes shape through the week, and the change is larger than most teams expect. Business software sees professional users from Monday to Friday and a thinner, different audience at the weekend. Consumer products often run the other way. Either pattern means that the mix of people in a test depends on which days it covered.
Covering whole weeks removes the problem by construction. Every weekday appears the same number of times in the treatment group and the control group, so the comparison is between two groups with the same day mix. A test running nine days has two Mondays and one of everything else, which weights Mondays at double.
A second reason for whole weeks concerns how people actually use a product. A change that affects a weekly workflow, such as a report somebody runs on a Friday, cannot be measured at all in three days. The test has to span at least one complete cycle of the behaviour it is meant to change.
Both reasons point at the same minimum. A test runs for at least seven days, and then for fourteen, twenty one or twenty eight days, because each of those lengths covers a whole number of weekly cycles and leaves the two groups with an identical mix of days behind them. Anything shorter is a preview.
The exposure point and who gets counted
The exposure point is the moment a person could first have experienced the difference between the two versions. That choice decides the population entering the comparison, and making it badly is the most common way to waste a correctly sized test.
Consider a change to the invoice screen at a product where a quarter of customers ever open that screen. If every person who logs in is assigned, the comparison holds four times as many people as the change can possibly affect. If the change lifts conversion on that screen by 8 per cent, the diluted figure across everybody is about 2 per cent, and a test powered to see 8 per cent sees nothing at all.
Restricting the count to people who reached the invoice screen recovers both the effect and the power. The one condition is that the trigger has to be something the change itself cannot influence. Reaching the screen is a safe trigger, because assignment does not affect who arrives there. Clicking the new button is not a safe trigger, since the button exists in only one group and the comparison then runs between two different populations.
A practical consequence follows for instrumentation. The exposure event has to fire in both groups at the same point in the flow, which means the control group records an exposure it never sees any difference from. Teams that instrument exposure only in the treatment group discover the problem after the test has finished.
Ramping a change before the full test
Most teams release a change to a small share of traffic first, watch the guardrails and then increase it. This is a sensible safety practice and a poor measurement, and separating the two purposes is what makes the sequence work.
The cost of an uneven split is arithmetic. Relative to an even split, the duration needed scales with one divided by four times the treatment share times the control share.
| Share on treatment | The multiplier | Duration against an even split |
|---|---|---|
| 1 per cent | 25.3 | about 25 times as long |
| 5 per cent | 5.3 | over 5 times as long |
| 10 per cent | 2.8 | nearly 3 times as long |
| 25 per cent | 1.3 | a third longer |
| 50 per cent | 1.0 | the baseline |
Each multiplier is one division. For the 10 per cent row, four times 0.10 times 0.90 is 0.36, and one divided by 0.36 is 2.78. For the 1 per cent row, four times 0.01 times 0.99 is 0.0396, and one divided by 0.0396 is 25.25.
The table converts directly into a plan. A test that needs two weeks at an even split needs about fifty weeks at one per cent, which is nobody's idea of a fortnight. So the ramp is short and its job is to catch breakage. The split then moves to even, and the clock for the measurement starts at that point and never at the first release.
The day a number turns significant
With the clock started, the remaining pressure is on when to stop it. A running test crosses and recrosses the significance threshold as data arrives, and the temptation to stop on the day it crosses upwards is the strongest force acting on any experimentation programme. Somebody is waiting for the answer, the answer is currently favourable, and stopping saves a week.
The reason this is wrong belongs to a later page in this course, and the short version is that the false positive rate quoted for a test assumes the team looked once. Repeated looks with the option to stop change that rate substantially, and the change is in the unhelpful direction.
What fixes it here is the plan. The stopping date is written down with the hypothesis and the criterion, the test runs to that date, and the result is read once. A team that genuinely needs the ability to stop early has a legitimate route through sequential methods, and taking that route means deciding to do so before the test starts.
Everything in this module assumes the change can be split between two groups of people at the same time. Some changes cannot be, and a price, a brand or a market wide launch reaches everybody at once by its nature.