Hypotheses and the overall evaluation criterion

A hypothesis is a prediction about what a specific change will do to a specific measure, stated precisely enough that a real result can contradict it.

A controlled experiment supplies an unbiased comparison between two groups and supplies nothing at all about what the comparison means. The judgement is a separate piece of work, and doing it before the test runs is what keeps it honest.

The requirement comes from the philosophy of science and arrived in product work through two routes. Karl Popper argued in 1934 that a scientific claim is one that forbids something, so a theory compatible with every observation explains nothing. Online experimentation added the operational half, and the single measure a test is judged on acquired a name, which Ron Kohavi, Diane Tang and Ya Xu set out at length as the overall evaluation criterion in their 2020 book on trustworthy experiments.

A product manager runs into this at the point where a result arrives. Twenty measures moved, four of them by a wide margin, and two of those four point in opposite directions. At that moment, the choice of which measure counts falls to whoever argues best, and the team will reliably pick the measure agreeing with what they already wanted.

The sections below set out what makes a prediction testable and the four parts a usable hypothesis carries. The overall evaluation criterion follows, with the properties a candidate has to hold. One composite criterion is then worked through, showing how much the weights decide.

A prediction that can fail

Most product hypotheses fail the basic test, which is that nothing could contradict them. Improving the onboarding flow will help users get value faster is compatible with every possible outcome of every possible experiment, because help, value and faster are all unmeasured.

Popper's point in 1934 was that the strength of a claim lies in what it rules out. A claim that forbids nothing survives every test and teaches nobody anything, and a claim that forbids a great deal is worth testing precisely because it is exposed.

Translating that into product work produces a simple check. Before running the test, the team writes down the result that would make them abandon the idea. If no such result exists, the experiment is a formality and the change was always going to ship.

The four parts of a usable hypothesis

1. The change. What differs between the two groups, in enough detail that somebody else could build it. Moving the plan chooser above the fold is a change. Simplifying signup is a project.
2. The population. The people included in the test. New signups on mobile is a population, and everybody is usually a mistake, since a change aimed at new users is diluted by existing ones who never see it.
3. The measure and the direction. Which single number is expected to move and which way. Trial to paid conversion, upwards.
4. The size worth having. How large a movement would justify the work. This number is the one teams skip, and it is the one that decides how long the test has to run.

Written out, the four parts produce a sentence somebody can disagree with. Moving the plan chooser above the fold will raise trial to paid conversion among new signups on mobile from 6.0 per cent to at least 6.6 per cent. That sentence names the change, the population, the measure and the threshold, and a result of 6.1 per cent refutes it.

The overall evaluation criterion

The overall evaluation criterion is the single measure, agreed before the test begins, that the decision will rest on. Everything else the experiment reports is context, diagnosis or a check on validity, and none of it overturns the criterion without a separate argument.

Three properties decide whether a candidate can do the job.

It has to be sensitive enough to move. Annual revenue per customer is the measure a business cares about, and no two week test on a fraction of traffic will move it detectably. A criterion that cannot move inside the test produces a flat result whatever the change did.

It has to point the same way as the thing that matters. Sensitivity alone is easy to buy and worthless on its own. Page views per session moves quickly and rises when navigation confuses people, so optimising it makes the product worse while the number improves.

It has to be hard to satisfy by damage. A criterion of sessions per user is met by a product that logs people out, and a criterion of clicks is met by a heading that misleads. The test worth applying is whether an obviously bad change could move the measure in the good direction.

Kohavi, Tang and Xu describe the search for a criterion with these properties as the hardest part of running an experimentation programme, and the reason is the tension between the first two. Measures that move quickly are usually short term behaviours, and measures tied closely to value are usually slow.

A composite criterion worked through

Resolving that tension often means combining two measures into one number with agreed weights. The worked example is a news product testing a redesigned article page. The team cares about reading and about subscriptions, and neither measure alone is trustworthy.

The criterion agreed before the test is 70 per cent weight on articles finished per reader each week and 30 per cent weight on subscription starts per thousand readers. Each measure is expressed as a ratio against the control group, so the control scores exactly 1.0 by construction.

The criterion for the treatment group is 0.7 times 1.10 plus 0.3 times 0.95, which is 0.77 plus 0.285, or 1.055. The redesigned page scores 5.5 per cent above the control and the team ships it.

Now reverse the weights and run the same arithmetic on the same result. At 30 per cent on reading and 70 per cent on subscriptions, the criterion is 0.3 times 1.10 plus 0.7 times 0.95, which is 0.33 plus 0.665, or 0.995. The redesigned page scores half a per cent below the control and the team reverts it.

Identical data produced opposite decisions, and the only difference was a pair of numbers somebody chose. That is the whole argument for agreeing the weights in advance. A team setting them after seeing the result is choosing the verdict and calling it analysis.

Agreeing the criterion before the test starts

The discipline is administrative and it takes about ten minutes. Before the test starts, one document records the hypothesis in the four parts above, the criterion with any weights, the size worth having and the date the test will stop. Anybody likely to argue about the result later signs the document while nothing is at stake.

Three habits break the discipline and each one has a name a team will recognise.

1. Adding a measure once the result is in, because the agreed one came back flat.
2. Reinterpreting a flat criterion as a win on the grounds that one segment improved.
3. Declaring the test invalid only when the answer is unwelcome.

None of those is dishonesty in the usual sense, which is what makes the written criterion worth having. The people involved genuinely believe the argument they are making, and the document from two weeks earlier is the only thing in the room that has not changed its mind.

A criterion decides how the result will be read. Whether the test can produce a readable result at all is a separate question about numbers, and the team settles that one before anybody starts.