An experimentation practice is the set of habits, records and infrastructure that lets an organisation run controlled tests as routine work.
One finding, written so that it travels, changes one decision. An organisation that wants evidence behind its decisions every quarter needs something different from a good analyst, because the analyst is a bottleneck and the supply of questions is not.
The gap between the two shows up in the numbers large programmes publish. Kohavi and Thomke reported in Harvard Business Review in September 2017 that Microsoft, Amazon, Booking.com, Facebook and Google were each running more than ten thousand controlled experiments a year. Thomke recorded in 2020 that Booking.com alone had more than a thousand tests running at the same moment. Those figures describe an operating model, and no amount of analytical skill produces one.
The sections below set out what the practice needs beyond a working platform, covering the backlog of hypotheses, the infrastructure that lets many tests share one population and the record of what has already been settled. The page then works through experiment velocity and closes on what a one in three success rate means for the way the whole thing is built.
The backlog of hypotheses
The constraint most teams meet first is a shortage of things worth testing. A platform can run fifty experiments. A roadmap usually offers three, and two of those are commitments somebody has already promised to a customer, which leaves one genuine question for an instrument built to answer dozens.
A hypothesis backlog is a list of predictions, each one written so that it can fail. Every entry names the change, the measure it is expected to move, the direction and the size, and the reason somebody believes it. The reason is the part that earns its place, because a prediction with a mechanism behind it teaches something whichever way the result goes, and a prediction with no mechanism teaches only what happened.
The sources of good entries are the same ones that fill a discovery process. Support tickets name the places people get stuck, funnel analysis names the steps that lose people, and retention curves name the cohorts that leave. Segment findings from earlier tests arrive as candidates rather than conclusions, which is what makes the discipline around multiple comparisons pay off.
What raises the quality of the backlog over time is scoring each entry on how much a result would change what the team does. An entry nobody would act on either way is worth deleting, however interesting it is, because the traffic it would occupy is the scarcest thing in the whole practice.
Running many tests on one population
Traffic is finite and tests compete for it. A product with forty thousand weekly visitors, needing 120,000 users in each group to detect the effect it cares about, occupies its entire audience for six weeks on a single question. Running experiments one at a time gives that product eight or nine tests a year. That is too few to learn anything.
Diane Tang, Ashish Agarwal, Deirdre O'Brien and Mike Meyer published Google's answer to this in 2010, at the KDD conference, under the name overlapping experiment infrastructure. The idea is that traffic is divided into layers, each user sits in one variant per layer, and experiments in different layers are assigned independently. One person can be in a search ranking test, a pricing test and a notification test at the same time without any of the three contaminating the others.
The condition that makes this safe is independence. Two experiments that touch the same screen, the same measure or the same workflow belong in the same layer, so that no user meets both. Two experiments on unrelated parts of the product can share the population freely, and the randomisation keeps the interaction spread evenly across both comparisons.
Setting this up is where most of the engineering cost of a practice sits. Getting it wrong produces a fault the validity checks already cover, since one experiment interfering with another is a recognised cause of a sample ratio mismatch.
The record of what has already been settled
The third component is institutional memory. It is the one most often missing, because a programme running two hundred tests a year generates two hundred findings, and an organisation with ordinary turnover loses the people who hold those findings in their heads within about three years.
A usable record has one entry per test, holding the date, the hypothesis, the design, the primary measure, the effect with its interval, the guardrails and the decision taken. The entries stay after the feature ships or is removed, they are searchable by measure and by area of the product, and they are open to anybody in the organisation.
The return on the record compounds in three ways. A team proposing a change can find out whether something similar has been tried, which prevents the same test running a third time. A team sizing a new test can look up how much the measure varied in earlier tests, which is the input the sample size arithmetic needs. And a leadership team can ask what the programme has actually delivered, which is a question no programme can answer from memory.
Experiment velocity and the arithmetic behind it
Experiment velocity is the number of tests a team completes in a period, and it is the closest thing a practice has to a rate of learning.
The arithmetic is worth seeing because it explains why velocity receives so much attention. At a success rate of one in three, a team running one test a month completes twelve tests a year and produces four winners. A team running one test a week completes fifty two and produces about seventeen. The second team is not four times better at product work. It has simply taken four times as many attempts.
Velocity has two honest limits and one dishonest route around them. The first limit is traffic, which sets how many tests can run at once and how long each takes. The second is the backlog, since a team cannot run more tests than it has predictions worth making.
The dishonest route is to raise the count by testing smaller and smaller changes. Fifty tests on button colours produce fifty results and no understanding, and a programme measured on its own experiment count will drift towards exactly that. Goodhart's law reaches the practice that was built to apply it.
What a one in three success rate means for the design of a practice
The figure that shapes everything is the failure rate. Microsoft's experimentation team reported in 2009 that among well designed experiments built to improve a key measure, only about one in three succeeded. Kohavi and Thomke gave the fuller breakdown in 2017, which is that roughly a third of experiments prove effective, a third come back neutral and a third come back negative.
Three consequences follow, and a practice that ignores any of them fails quietly.
- The cost of running a test has to be low. A process that takes six weeks of meetings per experiment cannot survive a two thirds failure rate, because the organisation will stop paying for it long before the winners arrive.
- A negative result has to count as delivered work. A team rewarded only for shipping will stop running tests that might say no, and the tests that might say no are the ones worth running.
- Portfolio thinking replaces project thinking. No individual test is expected to succeed. The programme succeeds, and it succeeds because a small number of large wins pay for everything else.
The Bing case Kohavi and Thomke describe shows why the portfolio view is the right one. An idea about how advertisement headlines were displayed needed a few days of engineering, was judged a low priority among hundreds of proposals and sat untouched for more than six months. When an engineer finally ran it, revenue rose by 12 per cent, worth more than one hundred million dollars a year in the United States alone, with no damage to the user experience measures. Nobody in the organisation had ranked it highly, which is the argument for testing many cheap ideas and the argument against trusting a prioritisation exercise to find the winner.
What the practice is finally for
This course opened on a single claim, which is that analytics earns its cost only when a number changes a decision. Everything since has been an account of what that takes. It runs from choosing a measure worth moving, through instrumenting the product so the measure exists and reading behaviour without mistaking correlation for cause, to designing a test that can answer the question and reading the answer without flattering it.
A practice is what makes the claim hold every quarter. The platform, the backlog, the layered traffic and the record all exist so that the next decision has evidence behind it without anybody heroically arranging for that to happen.
The test of the whole thing stays the one the course started with. An organisation with a working practice can point at the decisions its analytics changed, including the features it removed after a flat result and the plans it abandoned when a test disagreed with them. An organisation that cannot point at any of that has built a reporting function and called it analytics. The roadmap pays for that mistake for years, and the tooling budget never shows it.