Experimentation Works
About this book
Thomke studied experimentation programmes at companies including Booking.com, Microsoft and Amazon, and wrote about what turns testing into a capability rather than an occasional activity. The book covers what makes a good business experiment, the discipline required for a valid one, and the low success rate of ideas that everybody was certain about. A large part of it is about culture and management, including how to run a company where a junior employee's test result can overrule a senior person's opinion. There is material on scaling from a handful of tests to thousands, and on the infrastructure and governance that requires. Read it for the organisational argument rather than for the statistics.
Description via Product Digest.
Topics
Related concepts
- Controlled experimentsRandom assignment as the only practical route from correlation to cause, what the control and treatment groups are for, how the assignment unit is chosen and where it goes wrong, and the question an experiment answers that no amount of observation can.
- Building an experimentation practiceWhat an organisation needs to run tests as routine work, how many tests can share the same traffic, the record that stops a question being asked twice, the arithmetic of experiment velocity and what a one in three success rate means for the way a programme is built.
- Guardrail metricsMeasures watched for damage rather than for improvement, the two kinds Kohavi, Tang and Xu separate, a table of common guardrails with what each one protects, and the arithmetic showing how a two per cent win can be a loss once latency is priced.
- Deceptive alignment and situational awarenessWhat it would mean for a system to behave well only while it believes it is being watched, what the evidence for that actually shows, and why the possibility changes how evaluations are run.
- Capability evaluationsWhat an evaluation can establish about a model, why the method used to test a model decides the answer as much as the model does, and what a dangerous capability evaluation is for.
- Reading an experiment resultHow a measured difference becomes a decision to ship, iterate or abandon, what the flat result actually says, the eight result shapes and the decision each one supports, and why shipping a change that did nothing has a cost.