A controlled experiment assigns comparable units at random to two or more groups, applies exactly one deliberate difference to one of them and compares an agreed measure between the groups.
Segmentation exhausted what observation can offer. Cutting a population finer and finer explains where a number came from, and it never settles what would happen if the product changed, because every group being compared differed for its own reasons before anybody looked. Random assignment removes those reasons by construction.
The method is older than software by a long way. Ronald Fisher set out randomisation, replication and the null hypothesis in The Design of Experiments in 1935, working on agricultural field trials where the soil itself differed from one plot to the next. His answer to unmeasurable variation was to let chance allocate the plots, so that differences nobody could see were spread evenly across the treatments. Medicine adopted the same logic for clinical trials, and software inherited it once products were delivered over a network and the allocation could happen on every request.
A product manager reaches for this whenever a decision costs more than the test does. Rebuilding onboarding takes a quarter, and the argument about whether to do it can run for months on opinion alone. An experiment converts that argument into a number with a known error, which is usually cheaper than the argument.
The sections below explain what random assignment buys and what the two groups are for. Choosing the unit of assignment follows, with the failures that come from choosing it badly. Finally they set out the question an experiment answers that observation cannot, and what the instrument costs.
What random assignment buys
The two groups are built by chance from the same arriving population, so on average they hold the same mix of new and returning customers, the same devices and the same intentions. One deliberate difference is then applied to one of them, and it is the only thing that can explain a gap in the measure.
Random assignment does one job and it is worth stating precisely. If people are allocated by a coin toss, the two groups hold the same mix of every attribute in expectation, including attributes nobody recorded, nobody named and nobody knew existed. Intent, mood, company size, browser, time zone and whatever else drives behaviour are all spread evenly, because nothing about the allocation depended on any of them.
That property is what makes the comparison causal. Any difference in the measure afterwards has two possible sources, which are the change the team applied and chance. Statistics quantifies the second source, and the first source is the answer the experiment was run to get. Observation never reduces the list to two.
The word random is doing real work here and it is often abused. Assigning by the last digit of a customer identifier is fine only if that digit carries no information. Assigning by alternate days is not random at all, and the usual objection to it is the wrong one. Across whole weeks the weekdays do balance out, so the mix of days is not the problem. The problem is that anything carrying a date lands entirely in one group. A campaign, a payday, an outage or a storm arrives on odd days or on even days and never on both, so the two groups are shaped by different conditions. The same customer also meets both versions on consecutive visits, which means the design never allocated people at all. Anything predictable from a person's attributes, or from the moment they happened to arrive, reintroduces exactly the problem randomisation removes.
The control group and the treatment group
The control group receives the product as it already is. Its purpose is to measure what would have happened anyway, over the same days, the same campaigns and the same outages. A team that compares this month against last month has no control group and is comparing across everything that changed in the world during the interval.
The treatment group receives the product with one deliberate difference. The word one matters. A release that changes the onboarding copy, the button colour and the default plan produces a result nobody can act on. A positive number does not say which of the three earned it. A flat result leaves open whether two of them cancelled each other out.
Between them, the two groups supply something observation can never produce. The control group is an estimate of what the treatment group would have done without the change, measured on real people at the same moment. That estimate is the entire point of the exercise.
Choosing the assignment unit
That comparison depends on something being allocated in the first place. The unit is what gets allocated, and the sensible default is a person. Somebody assigned to the treatment sees the changed product on every visit, on every device where they are recognised and for the whole test. Consistency matters because a product that changes appearance between Monday and Tuesday teaches a customer that the product is broken.
Three other units come up and each carries a cost.
- A session. Finer grained, so it detects smaller effects with fewer people. It also shows the same person two different products, which makes it unsuitable for anything a customer would notice across visits.
- An account or company. Necessary where colleagues work in a shared workspace, because assigning colleagues to different versions produces confusion and no usable measurement. The cost is severe, since a hundred thousand people may be only two thousand accounts.
- A region or a market. The last resort for changes that reach everybody, such as a price or a marketing campaign. Regions differ from each other in ways randomisation cannot fix at small numbers.
The failure these choices guard against is interference, meaning one person's assignment affecting another person's outcome. A marketplace where the treatment group bids more aggressively leaves less inventory for the control group, so the measured gap includes damage the experiment caused to its own comparison. Social products, marketplaces and anything with shared capacity all have this problem, and the usual answer is to move the unit up to something that contains the interaction.
What an experiment answers that observation cannot
Observation answers what happened to people who differed. An experiment answers what would happen if a change were applied to people who did not differ, which is the question every roadmap decision actually asks.
The activation finding from earlier in this course is the clearest case. Retention among the customers who connected a bank feed in their first week was 55.5 per cent against 4.8 per cent for everybody else, and three explanations fitted that gap equally well. Forcing the step into onboarding for a randomly chosen half of new signups separates them in a fortnight. If retention rises in the treatment group, the step has causal force. If the connection rate rises and retention stays flat, the step was a symptom of intent all along.
No quantity of extra segmentation settles that question, because every extra cut still compares people who chose differently. The experiment settles it by removing the choice.
What the instrument costs
A controlled experiment is expensive in three currencies and each one catches teams out. It costs engineering, since the product has to serve two versions at once and record which version each person saw. It costs time, because the number of people needed to see a small effect is large and a product only receives so many people a week. And it costs optionality, because a change under test cannot also be adjusted while the test runs.
There is a further limit that applies before any of those. An experiment compares two groups on a measure, so the measure has to exist and has to be agreed before the test starts. A team that runs a test and then goes looking for a number that moved has not run an experiment, and the next page is about the discipline that prevents it.