A system can be trained against a perfectly good objective, learn a goal that fits every example it was shown, and still have learned the wrong goal. The mistake stays invisible until the world stops looking like the training data.
The shape of the failure
The clearest demonstrations come from small reinforcement learning environments. An agent is trained to collect a reward that always sits in the same place, say the bottom right of the level, and training goes well, with the agent reaching the reward in every episode.
Then the reward is moved, and the agent walks to the bottom right anyway. It had learned to go to that location rather than to find the reward. Both goals explain every training episode, because in training they prescribed identical behaviour, so nothing in the data could tell them apart.
The detail worth holding on to is that the agent is not broken. It navigates competently and moves with purpose, so its capabilities generalised while its goal did not, which is why the behaviour looks confident rather than confused.
Why this is not specification gaming
The two get confused constantly, and the fault sits in a different place in each. Specification gaming means the objective was wrong and the system pursued it faithfully, which is the boat circling a lagoon to collect bonuses. Goal misgeneralisation means the objective was right and the data was consistent with several goals, so the system acquired one of the others.
There is a quick test. Look at the behaviour and ask whether you would want to rewrite the objective. If it now reads as a mistake you are looking at gaming, and if it still reads as what you wanted, the objective was never the problem.
Why more data only sometimes helps
Data resolves the ambiguity only where it separates the candidate goals. Another ten thousand episodes with the reward in the same corner confirm both hypotheses equally, so they buy nothing at all. What breaks the tie is variation where the goals disagree, which here means moving the reward.
That reframes the instinct to train on more of the same, since the useful quantity is not volume but coverage of the distinctions you care about. Coverage is often hard to get, because the correlation sits in the world rather than in the sampling. Images from one hospital carry its equipment and labelling habits alongside the condition being diagnosed, and more of them never separate the two.
Why evaluation has to include unfamiliar cases
A held out test set drawn from the same pool as the training data inherits every one of those correlations, so a strong score on it is evidence about the pool rather than about the goal acquired.
Learning something about the goal needs cases in which the plausible goals predict different behaviour, and those have to be constructed rather than sampled. Change the setting while holding the task fixed, keep the incidental feature and change the correct answer, or take examples from a source the training data never touched.
Why a product person should care
This is the failure that passes every gate you have. It clears the accuracy target, the pilot and the first months of production, because all of those measure performance on the distribution the system was built from. Then a market changes or a supplier redesigns a form, and behaviour shifts in a way no dashboard predicted.
Two things follow. Monitoring after launch does work that testing before launch cannot, so it is not an optional extra. And the authority a system is given should track how confidently you can say which goal it learned, which is usually not very confidently at all.
What the field argues about
The small environment results are not in dispute. How far the phenomenon scales is. Some researchers read them as a miniature of a serious risk in large models, while others hold that training on enormously varied data shrinks the space of goals fitting all of it, and prefer to call the whole thing ordinary distribution shift. Both camps recommend evaluating outside the training distribution rather than trusting a test score.