Concept 5 of 5

Scalable oversight

3 questions test this

Scalable oversight is the problem of supervising a system on work a person cannot readily check. Nearly every method used to train and review models assumes a human can look at an output and say whether it is good, and that assumption has a shelf life.

Where the assumption comes from and when it expires

Reinforcement learning from human feedback has people compare outputs, trains a reward model on those comparisons, then optimises the system against that reward model. Review after deployment runs on the same premise, with somebody reading an output and approving it. The whole arrangement rests on a rater able to judge.

That rater struggles in three familiar situations. Tasks get long, so the object of judgement is a forty page analysis or a two hundred step trajectory. Tasks get technical, and assessing a proof or a change to unfamiliar code needs expertise the rater does not have. Tasks get numerous, so there are thousands of outputs a day and minutes for each.

What makes this dangerous is that a rater who cannot judge does not abstain. They fall back on what they can assess, which is fluency, confidence and length, so those preferences become the training signal. The system learns to be convincing rather than correct, and the oversight failure is baked in.

The research directions

None of what follows is a solved method. They are active lines of work, mostly demonstrated on tasks chosen because the answer was checkable, which is an awkward place from which to judge tasks that are not.

Debate. Two models argue opposing sides in front of a human judge, on the theory that a flaw in an argument is easier to spot when an opponent points at it. The open question is whether truth has a structural advantage there, or whether a persuasive false case wins in front of a judge who cannot check either.

Critique models. One model writes a critique of another model's output and the human evaluates that critique instead, which is cheaper because checking a specific objection beats reviewing everything. The limit is that critics miss what they were not trained to notice, and a mistake both models share is invisible.

Decomposition. Break a task nobody can check into pieces somebody can, then compose the answers. It works where a task genuinely divides, and some judgements do not survive being cut up, with errors settling into the joins.

Weak supervision of a stronger model. Have a smaller or more trusted model supervise a more capable one, and study how much of the stronger model's ability survives. Results suggest such supervision elicits ability the strong model already has rather than teaching it, which is encouraging and not a deployable control.

Self critique against written principles. The model revises its own output against a published set of principles and those revisions become training data. The principles can be read and argued with, unlike most rater guidelines, and applied more consistently than a rotating pool of people. The weakness is circularity, since a model that misreads a principle misreads it when critiquing and when revising.

What none of them removes

Each of these converts a judgement a person cannot make into one they can, and none creates ground truth where there was none. Amplifying a judgement you cannot independently confirm gives a more confident version of the same uncertainty, which is not the same as having checked.

The control that is actually available

The honest fallback is to limit what a system may do wherever nobody can check its work. Give it actions that are reversible, cheap to verify afterwards, or bounded in what they can affect. Where an action is neither reversible nor verifiable, a person takes responsibility or it does not happen.

Where the field disagrees

One camp treats scalable oversight as an engineering agenda that will yield to sustained work. Another holds that a system materially more capable than its supervisors cannot be reliably supervised by them at all, and that the effort belongs in interpretability or in declining to deploy. For a product team the disagreement decides whether postponing the oversight question counts as a plan or a hope.

Common misconceptions

Scalable oversight means reviewing more outputs, so it is a staffing question.

Adding reviewers does nothing when the difficulty is that no available reviewer can judge the output. Scalable here means oversight that keeps working as tasks grow longer, more technical and more numerous, and as the system becomes more capable than the people supervising it.

Human feedback already keeps models honest, so this is a future problem.

Human feedback keeps models producing what raters preferred, which approximates honesty only while raters can tell. On long, technical or high volume work they fall back on fluency and confidence, so the training signal rewards persuasiveness today rather than one day.

Debate, critique models and constitutional methods have solved this.

They are research directions with encouraging results, mostly on tasks chosen because the answer could be checked, which is the case the problem is not about. None is a finished method, and the control available now is restricting what a system may do where nobody can verify its work.

3 questions test this concept

A team moves a model from short answers to producing forty page technical analyses, and keeps the same arrangement of human raters comparing pairs of outputs. What is the likely effect on the training signal?

  • AIt holds up, because raters compare two outputs against each other rather than scoring one in isolation.
  • BIt degrades to noise, because raters out of their depth choose at random between the two.
  • CIt improves, because a longer output gives the rater more evidence to judge on.
  • DRaters who cannot assess the content fall back on fluency, confidence and length, so those preferences become the signal and the system learns to be convincing.
Check whether it stuck.

One per page, with a worked explanation.

Start the set
Related material
Book
AI Engineering, On evaluating outputs when there is no reference answer to compare against.
Book
Thinking, Fast and Slow, On the shortcuts a human judge reaches for under load.