Alternatives to a controlled experiment are the designs a team uses to judge a change that cannot be randomised into a control group and a treatment group.
A run plan settles how long a test lasts and how much of the traffic sees the new version, and every part of that plan assumes the traffic can be divided at all. Several of the decisions a product manager cares most about break the assumption before the plan begins. A price rise reaches every buyer or none of them. A new brand name appears across the whole site on one morning. A launch into Germany has no second Germany to hold back for comparison.
Economics and public health met the same obstacle long before software did, and they built designs that estimate a causal effect without random assignment. Kohavi, Tang and Xu gathered those designs for online products in 2020 under the name observational causal studies. Each one buys an answer with an assumption that somebody has to defend out loud.
The sections below name the changes that resist a split, then take four designs in order of how much each one assumes. A table compares them on what they need and what they leave open. The page closes on the size of the claim a team is entitled to make once the randomisation has gone.
The changes that resist a random split
Four kinds of change defeat random assignment, and each defeats it for its own reason.
A price change. Two customers on the same plan who pay different amounts will find out, through a forum, a renewal conversation or a sales call, and the company then has a fairness problem bigger than the pricing question it was trying to settle. Some firms do randomise price for new customers only, which narrows the question to people who never saw the old price.
A brand or navigation change. The value of a consistent name is that everybody sees the same name. Splitting it damages the thing being measured, and the supporting material carries one version anyway, since help articles, sales decks and advertising cannot be forked for a fortnight.
A market wide launch. A product entering a new country, a new regulator or a new sales channel is one event with one population behind it. No half of that population was held back, so no comparison exists to make.
A change with strong interference between users. In a marketplace, a delivery network or a messaging product, treating one person changes what an untreated person experiences. A courier given a better dispatch rule takes a job that a control courier would otherwise have taken, so the control group has been affected by the treatment and the measured difference is wrong in a direction nobody can predict.
The holdout group
A holdout is the smallest departure from a controlled experiment, because it keeps the randomisation and changes only what is being compared. The team keeps one or two per cent of the population out of every change a programme ships, for six months or a year, and everything the team builds goes to the rest. The difference between the two groups at the end is the cumulative effect of the whole programme.
That is a question no single test can answer. Individual results do not add up, because effects overlap, some decay after launch and some were false positives that shipped anyway. A holdout measures the total once, with the same arithmetic a single experiment uses.
The cost is real and falls on two groups. A small number of customers go without every improvement for a year, which matters more if the product is one they pay for. And the comparison needs a large population before one per cent of it carries enough people to detect anything, which puts the design out of reach for most products.
Switchback tests for a shared resource
A switchback test removes the interference problem by randomising time. The whole market runs the treatment for one interval, then the control for the next, on a schedule the team fixes before the test starts.
A courier network testing a new dispatch rule in one city shows the shape. The team switches every half hour over a fortnight, which gives 672 intervals, and assigns each one at random to the new rule or the old one. The unit of analysis is the interval. Both rules meet the same demand pattern across weekdays, evenings and weekends, because the schedule spreads them evenly across all of it.
Two costs come with the design. Carryover is the first, since an interval inherits couriers already dispatched under the previous rule, and an effect that lasts longer than the interval contaminates the next one. The effective sample is the second. Those 672 intervals are the observations, and the hundreds of thousands of deliveries inside them do not each count as one. Iavor Bojinov, David Simchi-Levi and Jinglong Zhao published the design, and the arithmetic for choosing an interval length, in a 2020 paper that appeared in Management Science in 2023.
Difference in differences
Difference in differences compares the movement over time in a group that received the change with the movement over the same period in a group that did not. Subtracting the second movement from the first removes anything that hit both groups, such as a season, an economic shift or a platform update.
The design's best known use is the study David Card and Alan Krueger published in the American Economic Review in 1994. New Jersey raised its minimum wage from $4.25 to $5.05 an hour in April 1992 and eastern Pennsylvania did not, so Card and Krueger surveyed fast food restaurants on both sides of the border before the rise and again several months later.
A product team can run the same arithmetic on a staged rollout. A company releases a new onboarding flow in Canada in March and leaves Australia unchanged. Activation in Canada moves from 31 per cent to 36 per cent, a gain of five points. Activation in Australia moves from 29 per cent to 31 per cent, a gain of two points. The difference in differences is three points, and that is the estimate of what the new flow did.
Everything rests on one assumption, which is that the two markets would have moved together without the change. Nothing in the data after the change can test it. The only support available is the record before the change, so a team plots both series for several months beforehand and shows that they moved in step.
Before and after comparison
The weakest design is the one used most often and named least. A team ships a change on a Tuesday and compares the fortnight that follows with the fortnight before it.
Four things move a measure across a fortnight and the change is one of them. The season moves it. A marketing campaign that launched in the same week moves it. A release that went out on the Thursday moves it. The change moves it too, and a single before and after comparison has no way to say which part of the movement belongs to which cause.
The disciplined version is an interrupted time series, which fits a trend through many periods before the change and asks whether the periods afterwards depart from that trend by more than the usual week to week variation. It rules out a steady trend and a regular seasonal pattern, which is worth having. It still cannot rule out anything else that happened on the same day.
What each design needs and what it leaves open
| Design | What it needs | What it cannot rule out |
|---|---|---|
| Holdout group | A population large enough to spare one per cent for months | Little, since assignment is random. The cost falls on the held back customers |
| Switchback test | A market that can be switched safely, and a carryover shorter than the interval | Effects that persist past an interval boundary |
| Difference in differences | A comparison group that moved in step before the change | Anything that hit one group and not the other in the same period |
| Before and after | A stable period on either side of the change | The season, campaigns, other releases and competitors |
The table is ordered by how much each design assumes, and the order is also the order of how much the result is worth in a disagreement. A holdout survives a sceptical finance director. A before and after comparison usually does not.
The size of the claim each design supports
What a weaker design costs is the sentence a team is allowed to write at the end. A randomised test supports the claim that the change caused the difference. Difference in differences supports the claim that the change is consistent with the difference, given that the two markets would otherwise have moved together. A before and after comparison supports the claim that the measure moved after the change went out.
Writing the smaller sentence is what keeps the larger one believable later. A team that reports every result in the language of cause spends its credibility on the cases where cause was never established, and the randomised test it runs next quarter arrives to an audience that has stopped separating the two.
Randomisation is the instrument that earns the strongest sentence, and it earns it only when the result is read for what it actually says. A correctly run test can still mislead the team that ran it, and the first thing to settle is what a significant result claims.