A trained model is a summary of the data it was fitted to. Every number you have about how well it works was measured on data from that same pool, and the guarantee it offers stops at the edge of the pool. The world does not stay inside it.
Where the guarantee ends
Machine learning finds patterns that held in past data and applies them to new cases, which is sound only while the new cases resemble the old ones. The pool a model learned from is called its training distribution, and every reported metric assumes that future inputs come from the same distribution.
That assumption is never exactly true and is frequently badly wrong. A fraud model learns from the fraud being attempted while its data was collected, and a hiring model from the people who applied under last year's job market. Both describe a moment, and both are used in a world that has moved past it.
The four ways the world moves
The inputs change. The kind of input arriving is different while the underlying relationship holds. A speech model trained on studio recordings starts receiving phone calls. What the words mean has not changed, but the model is being asked about a region of the input space it saw little of.
The population changes. The mix of outcomes moves. A support classifier tuned when one ticket in fifty was a billing complaint behaves badly once a pricing change makes it one in five, because the base rate it implicitly learned no longer holds.
The relationship changes. The same input now implies a different outcome. Spending patterns that indicated fraud last year indicate an ordinary customer this year, once the people committing fraud adapt. This is the hardest to catch, because the incoming data looks normal while every prediction quietly means something else.
The system changes the world. Outputs feed back into inputs. A lending model only ever observes repayment for the loans it approved, so what it later learns from is partly the product of its own earlier decisions and it can confirm itself indefinitely.
Why a held out test set flatters you
Standard practice holds back part of the data, fits the model on the rest and reports performance on the part held back. That guards against one specific error, a model memorising its examples rather than learning anything general.
It cannot do more, because the held back part came from the same collection at the same time and carries the same period, population and arrangement of the world. A model can score highly on it and behave poorly on genuinely new input, because the test contained none. High test accuracy rules out memorisation and says nothing about data differing in a way the test set had no examples of.
Quiet degradation
Shift is the mechanism behind the commonest way a deployed model fails, which is that it does not fail. Nothing raises an error, latency is unchanged, and every dashboard measuring whether the system is up reports that it is up. The answers are simply worse than they were, by an amount nobody has measured, over a period nobody can name. An outage announces itself, whereas a drifted model keeps producing confident output that people act on.
What follows in practice
Monitor the inputs, not only the outputs. Whether a prediction was right often becomes known late or never, while the distribution of arriving input is available immediately, so track it and alert when it moves. There is no settled threshold for how much movement matters, since the tests available tell you a distribution has shifted rather than whether that harms your use.
Keep a sample from a different period. A test set drawn from the same week as the training data measures memorisation. A set drawn months later, or from a different region or customer segment, is the cheapest approximation of the question you care about.
Attach a time and a population to every accuracy figure. Ninety four per cent is not a property of the model. It is what that model scored on that data, gathered then and from those people, and a figure quoted without those qualifiers has lost the thing that made it meaningful.