Trustworthy Online Controlled Experiments
About this book
Written by experimentation leaders from Microsoft, Google and Airbnb, this is the reference on A/B testing done properly. Covers metrics and the overall evaluation criterion, statistical power and sample sizing, the twyman's law effects that show a result is too good to be true, and the institutional memory that comes from running thousands of experiments. Deep on the traps of ratio metrics, sample ratio mismatch and interference between users.
Description via Product Digest.
Topics
Related concepts
- Hypotheses and the overall evaluation criterionThe hypothesis as a prediction specific enough to fail, the four parts it needs, the overall evaluation criterion set out by Kohavi, Tang and Xu in 2020, and a worked composite where the weights agreed in advance decide whether the same result reads as a win or a loss.
- Data quality in analyticsThe six failures that quietly stop collected data matching what happened, from an event that never fires to a field renamed in a release, the checks that catch each one, reconciliation against a system of record, and the honest way to report a loss that cannot be recovered.
- Choosing a product metricDeriving a measure from the claim a product strategy makes, the goal, driver and guardrail roles a measure can play, the five tests a candidate has to pass before a team commits a quarter to it, and the two failure modes that leave a team steering by a number it cannot use.
- Building an experimentation practiceWhat an organisation needs to run tests as routine work, how many tests can share the same traffic, the record that stops a question being asked twice, the arithmetic of experiment velocity and what a one in three success rate means for the way a programme is built.
- Measuring an AI featureAcceptance rate and the unit it is counted in, edit and override rate, the error rate a team has to build because nothing emits an event for a wrong answer, the arithmetic of the trade between latency and quality, and why ordinary engagement measures point the wrong way for a generative feature.
- Sample size and statistical powerThe minimum detectable effect, statistical power and the relationship that ties them to the number of people a test needs, a table running from a one per cent improvement to a fifty per cent one with the arithmetic shown, and what follows honestly for a product with modest traffic.
- Statistical significanceWhat a p value claims and what it cannot claim, what a confidence interval adds that a verdict of significant throws away, and why a difference can be statistically significant and still too small to fund.
- The peeking problemWhat repeated checking does to the false positive rate of a running test, the published figures for one check against ten, why the habit is invisible from inside a team and the two legitimate ways to stop a test early.
- Controlled experimentsRandom assignment as the only practical route from correlation to cause, what the control and treatment groups are for, how the assignment unit is chosen and where it goes wrong, and the question an experiment answers that no amount of observation can.
- Experiment duration and exposureThe three things that set how long a test runs, why the answer comes in whole weeks, the exposure point that decides who is counted, and the arithmetic showing that a one per cent ramp needs about twenty five times the duration of an even split.
- Multiple comparisonsWhy examining many measures at the five per cent threshold produces winners out of nothing, the arithmetic for twenty measures and for ninety six segment cuts, the two corrections available and the practice that avoids needing either.
- Alternatives to a controlled experimentThe changes that cannot be randomised into two groups, the four designs available when a split is impossible, what each one assumes and how much smaller the claim becomes once the randomisation has gone.
- Guardrail metricsMeasures watched for damage rather than for improvement, the two kinds Kohavi, Tang and Xu separate, a table of common guardrails with what each one protects, and the arithmetic showing how a two per cent win can be a loss once latency is priced.
- Novelty and primacy effectsThe effect that fades once a change stops being new, the effect that grows once people have learned their way around it, the two readings that separate them from noise and the published evidence on how often either one decides anything.
- Reading an experiment resultHow a measured difference becomes a decision to ship, iterate or abandon, what the flat result actually says, the eight result shapes and the decision each one supports, and why shipping a change that did nothing has a cost.
- Sample ratio mismatchThe validity check that runs before any result is interpreted, the chi squared arithmetic behind it, the five stages of an experiment where a mismatch is created and why a mismatched test is discarded.