Discovery
Experiment brief
The belief being tested, a hypothesis with a threshold and a date, how long the test runs and on how much traffic, and a table saying in advance what you will do with each possible result.
What it is
An experiment brief states what you believe, what you will do to find out, what result would change your mind, and what you will do in either case. It is one page written before the experiment runs, and the whole discipline lies in that order.
The reason to write the decision down first is that people are very good at explaining a result once they have seen it. A brief that names the threshold in advance turns an argument about interpretation into a matter of reading a number.
It also catches the experiments not worth running. Write both branches, find you would do the same thing either way, and the honest move is to skip the test and act, which saves the week the experiment would have taken.
When to use it
- A team is about to build something significant on an assumption nobody has tested.
- Two people disagree about what customers will do and the argument has now run twice.
- You are running a split test and want the success threshold agreed before the data arrives.
- A stakeholder has asked for a feature and you want to test the need behind it cheaply first.
What is in it
- The belief we are testing
- Why it matters
- Hypothesis
- What we will build
- Measure, threshold and sample
- What we will do with each result
- Risks and what we are prepared to accept
- Result
The template
Owner [name] / Written on [date] / Runs from [date] to [date] / Status Planned, running or read
The belief we are testing
One sentence on what the team currently believes, and one on where that belief came from. If the honest answer is that somebody suggested it in a meeting, write that, because it changes how much evidence you need.
For example, we believe admins abandon setup at the API key step because they do not have the credentials to hand, and that comes from three support transcripts rather than from research.
Why it matters
What decision is waiting on this. Name the thing you will build, stop or change once you know. An experiment with no decision behind it is a survey.
Hypothesis
We believe that [change] for [audience] will cause [effect]. We will know we are right when [metric] reaches [threshold] by [date].
For example, we believe that one click authorisation for the three most used tools will cause more new workspaces to finish setup, and we will know we are right when day seven integration rate reaches 30% in the treatment group by 26 September.
What we will build
The smallest thing that produces a trustworthy answer. Name what is real and what is faked, because a result from a prototype and a result from shipped code carry different weight.
Measure, threshold and sample
| Item | Value |
|---|---|
| Primary measure | Share of new workspaces with a live integration by day 7 |
| Current baseline | 22% |
| Threshold to act on | 30% |
| Guardrail that must not move | Support contacts about setup, currently 40 a week |
| Audience and split | New workspaces only, 50/50, English language accounts |
| Sample needed | 1,100 per arm for this difference at 80% power |
| Expected run time | 18 days at current signup rate |
Work out the sample size before you start rather than after. Stopping an experiment on the day the number first looks good is the most common way a team ships something that does nothing.
What we will do with each result
Fill this in before the experiment runs, and get the person who owns the decision to agree to it in writing.
| Result | What we do |
|---|---|
| At or above 30%, guardrail flat | Ship to all new workspaces, then instrument the day 30 effect |
| Between 24% and 30% | Keep the flag on, run again on the Spanish and German cohorts |
| Below 24%, or guardrail moves | Turn it off, write up what we learned, return to the tree |
Risks and what we are prepared to accept
Who could have a worse experience while this runs, for how long, and what would make you stop early. Name the stop condition as a number.
Result
Filled in on the read date. The number, the confidence interval, what you decided, and one line on what surprised you. Anything the team believed before the test and does not believe now belongs here, because that is the part people forget within a month.
Questions people ask
- Does this only apply to split tests?
- No. It works for an interview round, a fake door, a prototype or a pricing conversation. Anything where you are trying to learn something and could talk yourself into the answer.
- What if the result is unclear?
- That is a result. Record it, say what would be needed to make it clear, and decide whether that is worth another week. Rerunning until you like the number is the failure the brief exists to prevent.
- How long should an experiment run?
- Long enough to reach the sample you named in the brief. Stopping when the number looks good is the most common way a test produces a wrong answer with confidence.
- Who needs to see it before it starts?
- Anyone who would argue with the result afterwards. Agreement on the threshold is worth far more before the test than after it.