Novelty and primacy effects are changes in the size of a treatment effect over the life of an experiment, caused by people reacting to a change because it is new or by people needing time to learn it.
An experiment that has passed its validity checks reports the average effect across the days it ran. A decision made on that number assumes the effect will keep behaving the way it behaved during those days, and several ordinary product changes break the assumption in opposite directions.
The two effects were named in psychology long before they reached software. The novelty case shares its shape with the Hawthorne effect, in which people behave differently because something about their situation has visibly changed. Kohavi and five colleagues brought both into the vocabulary of online experiments in a 2012 paper on puzzling experiment outcomes, and the paper also contains the warning that most suspected cases turn out to be neither.
The sections below define each effect, give a worked pair of results that show what each does to a weekly reading, and then set out the two analyses that separate a real trend from early noise. The page closes on what the published evidence says about how often either effect changes a decision.
The novelty effect
A novelty effect is a temporary rise in a measure caused by people investigating something because it is new. A changed button, a new panel or an animation draws clicks from users who want to know what it does, and those clicks look identical in the data to clicks from users who wanted the thing itself.
Kohavi and his colleagues describe an experiment on a news homepage in which a link to a mail service was changed to open in a separate window. The change produced a strong novelty effect, because users were surprised by the new behaviour and repeated it. In that case the long term effect stayed positive and the change remained in the product, which is worth noting, since a novelty effect at the start says nothing about what is left at the end.
The damage a novelty effect does is to the size of the estimate. A two week test that captures most of the investigating and little of the settled behaviour returns a number the product will never see again, and a team that builds a forecast on it will miss by the whole of the difference.
The primacy effect
A primacy effect runs the other way. Existing users have habits built on the old version, so a change to navigation, layout or a familiar workflow makes them slower at first, and the measure recovers as the habits adjust.
The population matters here in a way it does not for novelty. A primacy effect falls entirely on people who already knew the product, and somebody arriving for the first time during the test has no old habit to unlearn. That gives the control group an advantage during the early part of a test that has nothing to do with which version is better.
The practical consequence is a change that reads as harmful in week one and useful by week four. A team that stops the test on the week one reading has declined a good change and, worse, has written down a conclusion about its users that the evidence did not support.
What each effect does to a weekly reading
Two worked results show the shape. The first is a collaboration product adding an animated prompt to a template gallery, measured by template usage. The second is the same product moving its main navigation, measured by task completion for users who existed before the test. Traffic was even across the four weeks in both cases, so the pooled figure is close to the average of the weeks.
| Week | Novelty case, template usage | Primacy case, task completion |
|---|---|---|
| 1 | up 9.2 per cent | down 3.1 per cent |
| 2 | up 5.4 per cent | down 0.8 per cent |
| 3 | up 2.1 per cent | up 1.6 per cent |
| 4 | up 0.4 per cent | up 2.9 per cent |
| Pooled over four weeks | up 4.3 per cent | up 0.2 per cent |
Both pooled figures are misleading, and they mislead in opposite directions. The novelty case reports a healthy gain of 4.3 per cent for a change that was worth almost nothing by week four. The primacy case reports a flat result of 0.2 per cent for a change that was worth 2.9 per cent once people had learned it. Reading only the bottom row of the table gets both decisions wrong.
Reading the effect week by week
The first analysis is to plot the effect for each period on its own, so that week two shows what happened in week two and carries nothing forward from week one. That is the table above, and it is the reading a novelty or primacy effect shows up in.
One trap sits directly beside it. Most experimentation platforms plot the cumulative effect by default, which is the effect measured from the start of the test up to each day. A cumulative graph converges, because each new day adds users to a total that is already there, and the early points swing wide because they rest on very few people.
Kohavi and his colleagues measured how wide. The first day of a test has a 67 per cent chance of landing outside the confidence interval the test eventually settles on, and the second day has a 55 per cent chance. A cumulative line that starts at minus 0.6 per cent and climbs towards zero over four days is regression towards the truth, and the feature owner watching it reads a trend that will carry on and cross into positive territory. It almost never does.
The reading taken from new users only
The second analysis computes the result for users who joined during the test and ignores everybody else. New users have no habits attached to the old version, so a primacy effect cannot touch them, and the comparison between their result and the result for existing users is the clearest evidence available that learning is what the test measured.
The same cut helps with novelty in a different way. A novelty effect fades with exposure, so if the gain among users in their first week of the test stays steady while the gain among users in their fourth week disappears, newness is what the test measured.
Both readings cost sample size, because each one throws away part of the population. A test large enough for the primary measure is often too small to support a confident answer on a quarter of its users, and a team that wants this analysis has to plan for it when the test is sized.
How often either effect changes a decision
The honest answer from the published evidence is that it rarely does. Kohavi and his colleagues reported in 2012 that across the experiments they had run at Microsoft they could not find a single case where a clearly significant result in one direction became a clearly significant result in the other because of novelty or primacy. Most effects were stable, and the early wandering was variance.
What they did find is the common pattern that a significant negative result gets more negative over time and a significant positive result gets more positive. Their conclusion was that extending an experiment that is clearly negative after a fortnight buys very little, and moving on to the next idea is the better use of the traffic.
Both findings point the same way for practice. The weekly reading is worth producing on every test, because it costs nothing and it catches the cases that are real. The belief that a bad result is about to turn around is worth treating with suspicion, because it is the single most common way a team argues itself into shipping a change the test declined.
A test read this way gives an honest account of one measure. A product team watches dozens of measures on every experiment, and the arithmetic of watching many things at once creates winners out of nothing at all.