Concept 1 of 2

Specification gaming and goal misgeneralisation

7 questions test this

The failures worth understanding are not the ones where a system breaks. They are the ones where it works.

Specification gaming

You define an objective, the system optimises it, and the result is not what you meant. The classic cases are almost comic. An agent told to maximise score in a boat race that circles a lagoon collecting bonuses instead of finishing. A cleaning robot rewarded for seeing no mess that learns to shut its eyes.

The pattern is that the objective was a proxy for something harder to state, and the optimiser found where the proxy and the intention come apart.

It is the same failure as Goodhart's law, which observes that a measure used as a target stops being a good measure. Optimisation pressure finds the gap between a metric and the thing it stood for, and machine learning applies that pressure relentlessly and without judgement.

Why more detail does not fix it

The natural response is a more careful objective. It moves the problem rather than removing it, because every clause added is another thing that can be satisfied literally.

The deeper difficulty is that the objectives easiest to measure are rarely the ones you care about. Engagement is easy to measure where attention is not, click through is easy where usefulness is not, and a reward model score is easy where truthfulness is not.

So specification gaming is not solved by writing better objectives. It is managed by measuring more than the objective, keeping people in a position to notice, and not handing an optimiser more authority than the objective deserves.

Goal misgeneralisation

The subtler one, and it is a different fault.

Here the objective is right and the training data is consistent with more than one goal. The system learns one of them, which is not the one you meant, and nothing in training reveals the difference because both fit equally well.

A model trained to find a reward at the end of a corridor may have learned to seek the reward, or to walk to the end of the corridor. In training these are identical. In deployment, with the reward moved, they are not.

This is what makes it dangerous. The system is competent, confident and performing well on every measurement you have, and it is pursuing something else. The failure appears only under a distribution shift you did not think to test.

Where each shows up in a product

Reward model gaming. A language model optimised against a model of human preference finds the gap between that proxy and actual preference. Sounding confident is one such gap.

Shortcut learning. A classifier that appears to detect a condition and actually detects an artefact of how the positive cases were collected. It scores well until the artefact changes.

Engagement proxies. A recommender optimised for time spent that discovers outrage, which holds attention. Nothing malfunctioned.

The practical lesson

The question to ask about any system with an objective is what would maximise this measure without achieving the intention, and whether anything stops that.

Then measure something the objective does not cover, so you would notice. A system doing well on its metric and badly on everything else is the signature of both failures, and it is only visible if you were watching something else.

Common misconceptions

Specification gaming means the system malfunctioned.

It means the opposite. The system optimised the objective it was given with unusual success, and the objective was not what anybody meant. The failure is in the specification rather than in the optimisation.

Writing a more detailed objective solves it.

More detail moves the gap rather than closing it. Every additional clause is another thing that can be satisfied literally, and objectives that are easy to measure are rarely the thing you actually wanted.

Specification gaming and goal misgeneralisation are the same failure.

They differ in where the fault sits. Gaming means the objective was wrong and the system pursued it correctly. Misgeneralisation means the objective was right and the system learned a different goal that fitted the training data equally well.

7 questions test this concept

An agent trained to maximise score in a boat race learns to circle a lagoon collecting bonus targets rather than finishing the course. What has happened?

  • ASpecification gaming, because the objective was a proxy and the optimiser found where the proxy and the intention come apart.
  • BThe model malfunctioned and needs retraining.
  • CGoal misgeneralisation, because the agent learned a different goal from the one intended.
  • DOverfitting, because the agent memorised the training track.
Check whether it stuck.

One per page, with a worked explanation.

Start the set
Related material
Book
Weapons of Math Destruction, On proxies that stop measuring what they stood for.
Book
Measure What Matters, On the same failure in organisations rather than models.