Robustness is the property of a model that keeps working when its input drifts away from what it was trained on, and distribution shift is the name for that drift. A trained model is a summary of the data it was fitted to. Every number anyone has about how well it works was measured on data from that same pool, and the guarantee it offers stops at the edge of the pool. The world does not stay inside it, and this page is about what happens when it leaves.
The sections below say where the guarantee ends, describe the four ways the world moves, explain why a held out test set says less than it seems to, name the quiet failure that shift produces and end with what follows in practice.
Where the guarantee ends
Machine learning finds patterns that held in past data and applies them to new cases, which is sound only while the new cases resemble the old ones. The pool a model learned from is called its training distribution, and every reported metric assumes that future inputs come from the same distribution.
That assumption is never exactly true. It is frequently badly wrong. A fraud model learns from the fraud being attempted while its data was collected, and a hiring model from the people who applied under last year's job market. Both describe a moment. Both are used in a world that has moved past it.
The four ways the world moves
The field separates four kinds of movement, and each has a technical name worth knowing because it says what to monitor.
The inputs change. The kind of input arriving is different while the underlying relationship holds, which the literature calls covariate shift. A speech model trained on studio recordings starts receiving phone calls. What the words mean has not changed, and the model is being asked about a region of the input space it saw little of.
The population changes. The mix of outcomes moves, which is called label shift. A support classifier tuned when one ticket in fifty was a billing complaint behaves badly once a pricing change makes it one in five, because the base rate it implicitly learned no longer holds.
The relationship changes. The same input now implies a different outcome, which is concept drift. Spending patterns that indicated fraud last year indicate an ordinary customer this year, once the people committing fraud adapt. This is the hardest to catch, because the incoming data looks normal while every prediction quietly means something else.
The system changes the world. Outputs feed back into inputs. A lending model only ever observes repayment for the loans it approved, so what it later learns from is partly the product of its own earlier decisions and it can confirm itself indefinitely.
Why a held out test set flatters the model
Standard practice holds back part of the data, fits the model on the rest and reports performance on the part held back, and the figure that results is the one quoted in every paper, every vendor comparison and every launch review. That guards against one specific error, which is a model memorising its examples without learning anything general.
It cannot do more, because the held back part came from the same collection at the same time and carries the same period, population and arrangement of the world. A model can score highly on it and behave poorly on genuinely new input, because the test contained none. High test accuracy rules out memorisation and says nothing about data differing in a way the test set had no examples of.
Quiet degradation
Shift is the mechanism behind the commonest way a deployed model fails, which is that it does not fail. Nothing raises an error, latency is unchanged, and every dashboard measuring whether the system is up reports that it is up. The answers are simply worse than they were, by an amount nobody has measured, over a period nobody can name. An outage announces itself. A drifted model keeps producing confident output that people act on.
What follows in practice
Three habits follow from all of this.
Monitor the inputs as well as the outputs. Whether a prediction was right often becomes known late or never, while the distribution of arriving input is available immediately, so it is tracked and an alert fires when it moves. There is no settled threshold for how much movement matters, since the tests available say that a distribution has shifted and say nothing about whether that harms the use.
Keep a sample from a different period. A test set drawn from the same week as the training data measures memorisation. A set drawn months later, or from a different region or customer segment, is the cheapest approximation of the question that matters.
Attach a time and a population to every accuracy figure. Ninety four per cent is no property of the model. It is what that model scored on that data, gathered then and from those people, and a figure quoted without those qualifiers has lost the thing that made it meaningful.