A guardrail metric is a measure watched during an experiment for signs of damage, with a threshold agreed in advance and a rule that crossing it overrides a win on the primary criterion.
An experiment sized correctly answers one question well. That arithmetic concerns the criterion alone, so a test can be perfectly powered to detect a 10 per cent lift in conversion while carrying no information about what the change did to page speed, error rates or the support queue.
The vocabulary comes from large experimentation programmes, and Kohavi, Tang and Xu set out the distinction most clearly in 2020 by separating metrics into three roles. Goal metrics describe what the business wants in the end. Driver metrics move faster and lead the goal. Guardrail metrics are watched for harm and are expected not to move at all.
A product manager needs guardrails because the damage a change does is almost never in the same place as the benefit. A checkout redesign that raises orders may have done so by loading a heavier page, by removing a reassurance that stops refunds or by making a subscription harder to cancel. Each of those arrives as a cost somewhere else in the company, weeks later, with nobody connecting it to the experiment that caused it.
The sections below separate the two kinds of guardrail and set out the common ones with what each protects. The arithmetic of pricing a guardrail breach follows, using published latency figures. The last sections cover how a threshold is agreed and how much a flat guardrail is worth as evidence.
The two kinds of guardrail
Organisational guardrails protect things the business cares about that the change was never meant to touch. Page load time, crash free sessions, error rates, refund rates, support contacts and unsubscribes all sit here. Each one has a level the organisation has decided to hold, and an experiment is one of the things that can break it.
Trust guardrails protect the experiment itself. The clearest example is the ratio in which people were actually split, since a test designed as fifty fifty that arrives as 50.8 against 49.2 across a million people has a fault in its plumbing and the result cannot be read. Checks of this kind run before anybody looks at the criterion, because a broken test produces numbers that look exactly like findings.
The difference matters in what a breach means. An organisational guardrail crossing its threshold makes the change a bad idea. A trust guardrail crossing its threshold makes the whole result unusable, including the guardrails themselves.
Common guardrails and what each one protects
| Guardrail | What it protects | A workable rule |
|---|---|---|
| Page load time at the 95th percentile | The experience of the slowest one session in twenty, which is where abandonment lives | No more than 3 per cent slower than control |
| Crash free sessions | Whether the product works at all on older devices | No measurable fall |
| Server error rate | The engineering cost of the change after release | No measurable rise |
| Support contacts per thousand sessions | The cost of confusion, which lands on another team's budget | No more than 5 per cent higher |
| Unsubscribes per thousand messages | The long term size of the audience a product can reach | No measurable rise |
| Revenue per visitor | The business, when the criterion is an engagement measure | No fall outside the confidence interval |
| Assignment ratio between groups | The validity of the experiment itself | Within the expected range for the sample size |
The last row belongs to a different category from the six above it, and it sits on the same dashboard anyway, because a team reading results in a hurry will skip a check that lives somewhere else.
One row is worth arguing about on every product. Revenue appears here as a guardrail, which is the right arrangement when the test is about engagement and the wrong one when the test is about pricing itself. The general rule is that whatever the change is aimed at becomes the criterion, and whatever the change might damage becomes a guardrail.
Pricing a guardrail breach
Every rule in that table names a threshold, and a threshold is easier to defend when somebody has converted it into money. Latency is the guardrail with the best published evidence, because the large search engines ran deliberate slowdown experiments and reported the results.
Kohavi, Deng, Longbotham and Xu described one of these at a conference in the summer of 2014. Bing slowed ten per cent of its users by 100 milliseconds and another ten per cent by 250 milliseconds for two weeks, and reported that every 100 milliseconds of added delay cost roughly 0.6 per cent of revenue.
Applying that figure to a worked case shows what a guardrail is for. Suppose a redesigned checkout lifts conversion by 2.0 per cent relative and adds 300 milliseconds to the page. Three hundred milliseconds at 0.6 per cent per hundred is 1.8 per cent of revenue, so the change buys 2.0 per cent with 1.8 per cent and nets 0.2 per cent. A decision that looked obvious becomes a decision about measurement error.
Two cautions belong with that arithmetic. The 0.6 per cent figure is Bing's, measured on a search product where speed is most of the experience, and no product should import it as its own constant. What transfers is the method, which is running a deliberate slowdown once, measuring what it costs and using the result to price every latency guardrail afterwards.
Agreeing a threshold before the test
A guardrail with no threshold is a chart somebody looks at afterwards and argues about. The threshold turns it into a rule, and it is written down in the same document as the hypothesis and the criterion, before anybody knows which way the result will go.
Three things go into that line.
- The direction that counts as damage. Most guardrails are one sided, since a faster page is never a problem.
- The size that counts as damage, expressed relative to the control group.
- The consequence, which is usually that the change waits until the damage is understood or removed.
The consequence is the part teams leave out, and leaving it out is what makes guardrails decorative. A breach with no agreed consequence becomes a conversation in which the person who built the change explains why this particular breach does not count.
The power a guardrail needs of its own
A guardrail reporting no significant change is weaker evidence than it looks, and the reason is the sample size arithmetic behind the test. The test was sized to detect a stated effect on the criterion, and nothing about that calculation guarantees the same sensitivity for a measure with a different baseline and a different variance.
Support contacts show the problem clearly. If contacts run at three per thousand sessions, the baseline is small, the variance is awkward and a test powered to see a 10 per cent conversion lift may be unable to detect a 30 per cent rise in contacts. The guardrail comes back flat and the support team finds out in the following quarter.
The practical repair is to report a confidence interval for every guardrail, which states what the result has excluded. An interval running from 2 per cent better to 4 per cent worse has genuinely excluded serious harm. An interval running from 20 per cent better to 25 per cent worse has excluded nothing, and saying so is more useful than the word flat.
A criterion, a sample size and a set of guardrails together describe what the test measures. How long it runs is a separate decision, and the day the numbers first cross a threshold is rarely the right answer.