Running tests that produce a decision, from the statistics of an online experiment to building a company that runs thousands.
An experiment is only worth running when the result would change what you do next, and most experimentation trouble comes from skipping that check. These books cover the design and statistics of online controlled experiments, the cheaper tests that answer a question before any code is written, and the organisational work of making experimentation ordinary rather than exceptional. The order runs from the technical reference out to the culture and measurement that surround it.
Kohavi, Tang and Xu wrote the reference on running controlled experiments properly, drawing on Microsoft, Amazon, Google and LinkedIn. The chapters on Twyman's law and on the traps that make a result look real are the ones worth rereading before you ship on a win.
Thomke on what an organisation needs in order to run experiments at volume, with Booking.com and Microsoft as the worked examples. The book to hand a leader who supports experimentation in principle and overrules it in practice.
A catalogue of forty four experiments with the cost, the strength of evidence and the setup time for each, arranged from cheap discovery tests to expensive validation. The reference you open when you know what you need to learn and not how to learn it.
An experiment needs a stated behaviour change to test, and this is the clearest treatment of writing one. The title is the template, and it settles most of the arguments about what a success metric should be before they start.
Where build measure learn and the minimum viable product came from, together with innovation accounting for showing progress before there is revenue to show. Read it for the framing that an idea is a hypothesis with a business attached.
Instrumentation, event design and the analysis that turns product usage into an answer. Without this the experiment platform has nothing trustworthy to measure, which is where most programmes quietly fail.
The growth team operating rhythm, which is a weekly cycle of idea generation, prioritisation, testing and analysis. Useful for the cadence and the meeting structure rather than for any individual tactic in it.
Spiegelhalter on what a p value does and does not tell you, in plain language and with real examples throughout. The corrective for reading a result more confidently than the data supports.