Concept 1 of 3

How models are trained to behave

7 questions test this

A model that answers helpfully and declines harmful requests was made that way in stages. Knowing the stages explains most of what goes wrong later.

Pre-training

The model learns to predict the next token across an enormous corpus. This is where capability comes from, and it costs millions of dollars and weeks of compute.

What comes out is knowledgeable and not useful. A base model continues text. Ask it a question and it may produce more questions, because that is what often follows a question in its training data. It has no notion of being asked anything.

Everything else is added to that.

Supervised fine tuning

Training on examples of instructions paired with good responses, usually written or curated by people. This is what turns a text continuer into something that answers.

It is straightforward and its limit is coverage. The model imitates the demonstrations, so behaviour on anything unlike them is extrapolation.

Reinforcement learning from human feedback

Producing several responses, having people rank them, training a reward model to predict those rankings, then optimising the language model against that reward model.

The reason for the indirection is scale. People cannot rank every output, and a reward model can score millions.

Two consequences follow, and both matter.

The model is optimised against a model of human preference, not against human preference. Any gap between the two is a gap the optimiser will find, which is specification gaming arriving by the front door.

And raters prefer what reads well. Confident, complete, fluent answers score higher than hedged ones, so the process quietly rewards sounding right. This is one mechanism behind confident wrong answers, and it is a training artefact rather than a mystery.

Constitutional and AI feedback methods

Rather than people ranking everything, the model critiques and revises its own output against a written set of principles, and those revisions become the training signal.

The advantages are cost and consistency. A written constitution can be published and argued with, where a rater guideline usually cannot, and the model applies it more consistently than a rotating pool of people.

The limit is circularity. A model judging itself against principles it also interprets can be wrong in a correlated way, so human evaluation stays the reference.

Refusal training and why it leaks

Refusals are trained. Examples of harmful requests paired with declines, and the model generalises from them.

Generalisation is the whole problem. It extends unevenly to phrasings, languages and framings that were thin in the training data, which is why jailbreaks work at all. A jailbreak is not a bypass of a rule, because there is no rule. It is a region where the training generalised weakly.

That is also why refusal behaviour is fragile against further training. Fine tuning an aligned model on unrelated data can degrade its refusals, without anybody intending it.

What this means downstream

Every safety property of a deployed model is a property of a training process somebody ran, using data you cannot see, optimising a proxy for a goal.

So it holds on average, over the distribution it was trained against, and it weakens at the edges. That is why controls outside the model, filtering, grounding, permissions and oversight, do work that better training cannot.

Common misconceptions

A model refuses harmful requests because it understands they are harmful.

It refuses because refusing was rewarded during training on examples resembling the request. That is why refusals generalise unevenly, and why rephrasing, translating or framing a request as fiction can get past them.

Reinforcement learning from human feedback makes a model truthful.

It makes a model produce what raters preferred. Raters prefer answers that sound confident and complete, so the process can reward plausible over accurate, which is one route by which a model learns to bluff rather than to say it does not know.

The helpful assistant is what the base model is like.

A base model completes text and does not answer questions, refuse anything or hold a conversation. Every one of those behaviours is added afterwards, deliberately, and can be removed by further training.

7 questions test this concept

What does a base model do before any further training is applied?

  • AIt answers questions, but refuses nothing and holds no conversation.
  • BIt behaves as a helpful assistant, and less reliably than a tuned model.
  • CIt produces random output until it is given examples to imitate.
  • DIt continues text, and does not answer questions, refuse anything or hold a conversation.
Check whether it stuck.

One per page, with a worked explanation.

Start the set
Related material
Book
AI Engineering, On what each training stage contributes.
Book
The Hundred-Page Machine Learning Book, On the mechanics underneath.