Statistical significance is a judgement about how surprising a measured difference would be if the change under test had made no difference at all.
Any comparison of two groups produces two numbers that differ, because the people in each group are different people who arrived on different days in different moods. A test therefore needs a way to separate a difference produced by the change from a difference produced by that ordinary variation.
The machinery for doing it came from agricultural research. Ronald Fisher set it out in Statistical Methods for Research Workers in 1925, together with the convention of treating a result that would appear once in twenty trials as the point at which a difference is worth taking seriously. Every experimentation platform a product team touches reports a p value built on Fisher's definition, and almost none of them explain what the definition covers.
The gap between the definition and the reading is expensive, because a product manager turns that number into a decision worth engineering months. The sections below state what a p value says, name the two claims it cannot support, and introduce the confidence interval as the figure that carries the size of the effect. One result is then worked through from end to end, and the page closes on the difference between a result that is significant and a result that is worth shipping.
What a p value states
A p value is the probability of seeing a difference at least as large as the one measured, on the assumption that the change had no effect whatever. It answers one question, which is how ordinary this data would be in a world where nothing happened.
An invoicing product tests a shorter signup form and gives the arithmetic a shape. Each group receives 120,000 visitors. The original form produces 5,400 completed signups, a rate of 4.50 per cent, and the shorter form produces 5,640, a rate of 4.70 per cent. The gap is 0.20 percentage points, and the p value for that gap is 0.019.
The reading is narrow and exact. If the shorter form changed nothing at all, a gap this wide or wider would turn up in about two tests in every hundred. That is all the number says.
The two claims a p value cannot support
The American Statistical Association published six principles on p values in March 2016, after several years in which research in psychology and medicine had failed to replicate. Two of the six matter directly to a product decision.
A p value is not the probability that the hypothesis is true. The calculation assumes the change did nothing and measures how odd the data looks under that assumption. The question a team actually wants answered runs the other way, starting from the data and asking how likely it is that the shorter form helps. Turning the first quantity into the second needs one more input, which is how often ideas of this kind work at all. A p value carries no information about that.
A p value says nothing about the size of the effect. Significance depends on the sample as much as on the effect, so a large enough test makes a trivial difference significant. With five million visitors in each group, a gain of 0.03 percentage points on a base of 4.50 per cent clears the five per cent threshold with a p value of 0.022. That is a relative gain of less than one per cent, and no team would fund a quarter of engineering work to get it.
One more consequence follows from the threshold itself. A five per cent threshold means that among all the tests run on changes that do nothing, one in twenty comes back significant. A programme running two hundred such tests a year should expect ten false winners from arithmetic alone.
The confidence interval
A confidence interval reports the effect in the units of the measure, with a range around it. For the signup test, the 95 per cent interval runs from 0.03 to 0.37 percentage points. Expressed against the 4.50 per cent baseline, the shorter form is worth somewhere between a 0.7 per cent and an 8.2 per cent relative improvement.
The interval is built so that, across many repeated tests, 95 per cent of the intervals produced this way contain the true effect. Its relationship with the p value is mechanical. An interval that excludes zero corresponds to a p value below 0.05, so the two figures always agree on the verdict.
They disagree on how much they tell a reader. The verdict compresses the whole result into one bit of information, which is whether zero sits inside the range. The interval keeps the size and the uncertainty, and those are the two things a decision needs. A team that reads only the verdict has thrown away the part of the output that was worth having.
Statistical significance and practical significance
Practical significance is the size of effect that would justify the work, and the team settles it before the test runs. An invoicing team might agree that anything below a 2 per cent relative gain in signups cannot pay for the support load and the maintenance a second form brings.
Setting that line changes how the signup result reads. The measured gain is 4.4 per cent in relative terms, which clears the line. The interval runs from 0.7 per cent to 8.2 per cent, which straddles it. The test has established that the shorter form helps and has left open whether it helps enough, and a team that reports only the word significant has hidden exactly that.
Three readings follow from putting the two thresholds side by side.
- The whole interval sits above the practical line, so the change is worth shipping and the evidence supports the decision.
- The whole interval sits below the practical line, so the change is real and too small to fund.
- The interval straddles the line, so a longer test is the only way to separate the two, and the team decides whether that extra time is worth buying.
The third reading is the most common and the one teams handle worst, because the dashboard shows green and the interval is in a column nobody opened.
What a significance verdict leaves out
A verdict carries no information about whether the test itself was sound. The arithmetic behind both the p value and the interval assumes a specific procedure, which is that the sample size was fixed before the test began and the result was examined once, at the end.
Almost no team works that way. A running test sits on a dashboard that anybody can open, the numbers move every day, and the temptation to stop as soon as the gap looks convincing is strong enough that most organisations give in to it without noticing. That habit does not slightly weaken the five per cent guarantee. It replaces the five per cent with a much larger number, and the size of the replacement has been calculated since the 1960s.