A model that answers helpfully and declines harmful requests was made that way in stages, and none of those behaviours is present when training begins. The pipeline that produces an assistant from a text predictor was set out in a 2022 paper from OpenAI describing InstructGPT, and every major laboratory now runs a version of it. Knowing the stages explains most of what goes wrong later, because every safety property of a deployed model was put there by one of them and can be lost through the same door.
The sections below take the stages in order, then explain why refusal behaviour generalises unevenly and what all of it means for a team deploying the model.
Pre training
The model learns to predict the next token across an enormous corpus. This is where capability comes from, and it costs millions of dollars and weeks of compute.
What comes out is knowledgeable and useless as an assistant. A base model continues text. Asked a question it may produce more questions, because that is what often follows a question in its training data. It has no notion of being asked anything, no notion of a user and no reason to refuse anything.
Everything else is added to that.
Supervised fine tuning
The first addition is training on examples of instructions paired with good responses, usually written or curated by people. This is what turns a text continuer into something that answers.
It is straightforward and its limit is coverage. The model imitates the demonstrations, so behaviour on anything unlike them is extrapolation, and a few tens of thousands of demonstrations cannot cover what a public will ask.
Reinforcement learning from human feedback
The second addition works from comparisons. The model produces several responses, people rank them, a reward model is trained to predict those rankings and the language model is then optimised against that reward model.
The reason for the indirection is scale. People cannot rank every output, and a reward model can score millions. A later method called direct preference optimisation, published in 2023, reaches a similar result without the separate reward model, and the consequences below apply to it as well.
Two consequences follow, and both matter.
The model is optimised against a model of human preference, and never against human preference itself. Any gap between the two is a gap the optimiser will find, which is specification gaming arriving by the front door.
And raters prefer what reads well. Confident, complete, fluent answers score higher than hedged ones, so the process quietly rewards sounding right. This is one mechanism behind confident wrong answers, and it is a training artefact with nothing mysterious in it.
Constitutional and AI feedback methods
The third addition replaces some of the human ranking. The model critiques and revises its own output against a written set of principles, and those revisions become the training signal. The method was published in 2022 under the name constitutional AI, and the wider practice of using a model's judgements as the training signal is called reinforcement learning from AI feedback.
The advantages are cost and consistency. A written constitution can be published and argued with, where a rater guideline usually cannot, and the model applies it more consistently than a rotating pool of people.
The limit is circularity. A model judging itself against principles it also interprets can be wrong in a correlated way, so human evaluation stays the reference.
Refusal training and why it leaks
Refusals come from the same stages. Examples of harmful requests paired with declines are included in the fine tuning and preference data, and the model generalises from them.
Generalisation is the whole problem. It extends unevenly to phrasings, languages and framings that were thin in the training data, which is why jailbreaks work at all. A jailbreak bypasses no rule, because there is no rule. It finds a region where the training generalised weakly.
That is also why refusal behaviour is fragile against further training. Research published in 2023 showed that fine tuning an aligned model on a small amount of unrelated data can degrade its refusals substantially, without anybody intending it, and the finding has been repeated since.
What this means for a deployed model
Every safety property of a deployed model is a property of a training process somebody ran, using data the deploying team cannot see, optimising a proxy for a goal.
So it holds on average, over the distribution it was trained against, and it weakens at the edges. That is why controls outside the model, meaning filtering, grounding, permissions and oversight, do work that better training cannot.