Reading an experiment result is the step that turns a measured difference into one of three decisions, which are to ship the change, to iterate on it or to abandon it.
A team that named its primary measure in advance and held to it has a number it can defend. Nothing inside that number says what to do next. Two teams looking at the same gain of 0.2 percentage points will reach opposite conclusions, depending on what they agreed the gain was worth beforehand and on what else moved while it happened.
The decision usually lands on the product manager, because that is the only role in the room holding the statistics, the engineering cost of keeping the change and the commitment the roadmap made elsewhere. A data scientist can say what the interval is. Nobody else can say whether a gain of that size pays for a second code path that somebody maintains for four years.
The sections below set out the three decisions and the evidence each needs, then give the flat result the treatment its frequency deserves. A table of result shapes against decisions follows. The page closes on the cost of shipping a change that turned out to do nothing.
The three decisions and what each one requires
Ship. The evidence supports shipping when the primary measure improved by an amount the team agreed in advance was worth having, the confidence interval keeps that amount inside its range, and no guardrail moved against the product. All three conditions matter. A gain that clears the threshold with an interval stretching down to nothing has established that something happened, while leaving the size of it open.
Iterate. The evidence supports another attempt when the mechanism looks right and the execution does not. A checkout change that moved the intended step and lost the gain two steps later is a case for iteration, because the hypothesis survived and the implementation did not. So is a change that helped one segment strongly and hurt another, which is an argument for targeting rather than for abandonment.
Abandon. The evidence supports abandonment when the primary measure did not move, when it moved by less than the change costs to keep, or when a guardrail was damaged enough to outweigh the gain. Abandonment is the most common of the three and the least practised, because the work is already built and somebody has spent a month on it.
Kohavi and Thomke published the proportions in Harvard Business Review in September 2017, drawing on Microsoft's experimentation programme. About a third of experiments prove effective, about a third come back neutral and about a third come back negative. An earlier Microsoft paper from 2009 put the same finding more bluntly, reporting that of well designed experiments built to improve a key measure, only about one in three succeeded in improving it.
The flat result and what it actually says
A flat result is the most common outcome an experiment produces and the one almost no article describes. It arrives with a small difference, a p value well above the threshold and an interval that contains zero, and a team reads it as a failed test.
The reading is wrong in an important way. A flat result with a narrow interval is a finding. It says the change did not move the measure by anything the product would notice, and it says so with precision. A belief the roadmap was built on has been removed for the price of one test, which is the cheapest piece of information any experiment produces.
A flat result with a wide interval is a different object entirely and deserves a different word. The test failed to answer the question, because the sample was too small, the exposure too narrow or the measure too noisy. Nothing has been learned about the change, and the honest report says so instead of saying the change did nothing.
Telling the two apart takes one look at the interval. A signup test that reports no significant difference with an interval from a 0.4 per cent loss to a 0.5 per cent gain has settled the question. The same test reporting an interval from a 9 per cent loss to a 10 per cent gain has settled nothing, and the words no significant difference describe both.
The result shapes and the decision each supports
The table below assumes the primary measure was named in advance and that a practical threshold was set beside it.
| Result shape | What it means | Decision it supports |
|---|---|---|
| Significant, interval entirely above the practical threshold, guardrails clean | The change helps by an amount worth keeping | Ship |
| Significant, interval entirely below the practical threshold | The change helps by less than it costs to maintain | Abandon, and record the size for future reference |
| Significant, interval straddling the practical threshold | The change helps, and whether it helps enough is open | Extend the test, or ship if the maintenance cost is near zero |
| Flat, interval narrow and close to zero | The change moves nothing the product would notice | Abandon, and delete the code |
| Flat, interval wide | The test could not answer the question | No decision. Resize and rerun, or accept that the question is out of reach |
| Significant gain with a guardrail breached | The gain was bought with damage elsewhere | Treat as a loss. Iterate on the mechanism or abandon |
| Significant harm | The change makes the product worse | Abandon, and keep the finding in the record |
| Any shape with a failed validity check | Nothing has been measured at all | Discard, fix the cause and rerun |
Two rows deserve attention because teams collapse them. The fourth row and the fifth row both read as no significant difference on a dashboard and they support opposite actions, since one ends the question and the other reopens it. Any write up that uses the phrase without the interval beside it has hidden which of the two happened.
The cost of shipping a change that did nothing
Shipping a flat result feels harmless. The work is finished, the change is inoffensive and removing it means telling somebody their month produced nothing. Four costs follow anyway and none of them appears on the test report.
- Every code path is maintained forever. A second variant of a screen carries its own defects, its own tests and its own migration work every time the framework beneath it changes.
- Every option shown to a user costs attention. A feature nobody needed still occupies a menu, a settings page and a paragraph of the help documentation.
- Every shipped change becomes precedent. A team that ships neutral results has told itself that evidence is advisory, and the next decision is easier to make without any.
- Every shipped change pollutes the record. A programme measuring its own value later cannot separate the changes that worked from the changes that were merely shipped.
The alternative is cheap and unpopular. A change that did not clear its threshold gets removed, the finding gets written down with its interval, and the team keeps the knowledge without keeping the code. Doing that once is easy. Doing it as standing practice needs a culture where a negative result counts as work delivered, which is a question about incentives.
That question turns out to be the difficult one. A measure used to decide whether work was worthwhile becomes a measure people are judged on, and a measure people are judged on stops behaving the way it did when nobody was watching it.