A model that answers helpfully and declines harmful requests was made that way in stages. Knowing the stages explains most of what goes wrong later.
Pre-training
The model learns to predict the next token across an enormous corpus. This is where capability comes from, and it costs millions of dollars and weeks of compute.
What comes out is knowledgeable and not useful. A base model continues text. Ask it a question and it may produce more questions, because that is what often follows a question in its training data. It has no notion of being asked anything.
Everything else is added to that.
Supervised fine tuning
Training on examples of instructions paired with good responses, usually written or curated by people. This is what turns a text continuer into something that answers.
It is straightforward and its limit is coverage. The model imitates the demonstrations, so behaviour on anything unlike them is extrapolation.
Reinforcement learning from human feedback
Producing several responses, having people rank them, training a reward model to predict those rankings, then optimising the language model against that reward model.
The reason for the indirection is scale. People cannot rank every output, and a reward model can score millions.
Two consequences follow, and both matter.
The model is optimised against a model of human preference, not against human preference. Any gap between the two is a gap the optimiser will find, which is specification gaming arriving by the front door.
And raters prefer what reads well. Confident, complete, fluent answers score higher than hedged ones, so the process quietly rewards sounding right. This is one mechanism behind confident wrong answers, and it is a training artefact rather than a mystery.
Constitutional and AI feedback methods
Rather than people ranking everything, the model critiques and revises its own output against a written set of principles, and those revisions become the training signal.
The advantages are cost and consistency. A written constitution can be published and argued with, where a rater guideline usually cannot, and the model applies it more consistently than a rotating pool of people.
The limit is circularity. A model judging itself against principles it also interprets can be wrong in a correlated way, so human evaluation stays the reference.
Refusal training and why it leaks
Refusals are trained. Examples of harmful requests paired with declines, and the model generalises from them.
Generalisation is the whole problem. It extends unevenly to phrasings, languages and framings that were thin in the training data, which is why jailbreaks work at all. A jailbreak is not a bypass of a rule, because there is no rule. It is a region where the training generalised weakly.
That is also why refusal behaviour is fragile against further training. Fine tuning an aligned model on unrelated data can degrade its refusals, without anybody intending it.
What this means downstream
Every safety property of a deployed model is a property of a training process somebody ran, using data you cannot see, optimising a proxy for a goal.
So it holds on average, over the distribution it was trained against, and it weakens at the edges. That is why controls outside the model, filtering, grounding, permissions and oversight, do work that better training cannot.