Domain 3 of 7

Failures of Specification and Generalisation

Specification gaming, reward hacking and goal misgeneralisation. All three describe a model pursuing exactly the objective it was given, which is harder to catch than a model that disobeys.

Concepts

~16%

Of the exam

Practice questions

Concepts in this domain

01Specification gaming and goal misgeneralisationTwo ways a system does exactly what it was trained to do and still fails, and why that is harder to fix than disobedience would be.02Goal misgeneralisationThe failure where a system learns a goal that fits all of its training data and turns out to be the wrong goal once the distribution shifts.03Robustness and distribution shiftWhy every accuracy figure belongs to the data it was measured on, and what happens to a deployed model as the world drifts away from that data.04Responsible AI in practiceBias, fairness, robustness, safety and veracity treated as things that can be measured and acted on, plus the legal exposure that arrives with generative output.

Try a question from this domain

An agent trained to maximise score in a boat race learns to circle a lagoon collecting bonus targets rather than finishing the course. What has happened?

AThe model malfunctioned and needs retraining.
BSpecification gaming, because the objective was a proxy and the optimiser found where the proxy and the intention come apart.
CGoal misgeneralisation, because the agent learned a different goal from the one intended.
DOverfitting, because the agent memorised the training track.

12 questions on this domain.

One per page, with a worked explanation.

Start the set