AI safety is the study of whether machine learning systems do what they were built to do, and of what follows where they do not. Such systems learn how to behave from data, and nobody can say in advance everything they have learned. That uncertainty is the field's subject, and it is why the subject lands on anyone who builds, buys or governs a system of this kind.
The field took its modern shape in 2016, when researchers from Google, OpenAI, Stanford and Berkeley published a paper called Concrete Problems in AI Safety. It set out five ways a learning system could cause harm while doing exactly what it was trained to do, and the list still organises much of the work. Avoiding side effects, avoiding reward hacking, oversight that scales, safe exploration and robustness to a change in conditions each have a page of their own on this site. What the paper established was that these were engineering problems, with nothing of science fiction in them, and that they could be worked on at once.
The subject runs from a fraud attempt made easier by a language model, through a classifier that behaves differently on a population it never saw in training, to the effect on a labour market of a capability arriving faster than anybody can absorb it. Problems that far apart do not share a remedy. The sections below describe how the field separates them, the argument inside the field about which deserve attention, the two words everybody uses for the problem and the single fact that makes all of it harder than ordinary engineering.
The three kinds of risk
The field separates risk by where it originates, since the origin is what decides the remedy.
Misuse. The system works as intended and someone uses it to cause harm. Fraud at scale, synthetic media of real people and the generation of material for an attack are all misuse. Nothing in the system has broken. Access control, refusal training, monitoring and law are the remedies, and none of them is a modelling problem.
Accidents. Here something has broken. The system fails, or else it succeeds at something that was never intended. A model that scores well in testing and behaves badly in deployment, a model that learns a shortcut nobody noticed and a model applied to a population it was never trained on are all accidents. Evaluation, oversight and design are the remedies.
Structural harm. Nothing malfunctions, nobody intends harm and the outcome is bad regardless. Neither of the first two categories accounts for it. Competitive pressure to deploy faster than anyone is able to check, a concentration of capability in few hands and a labour effect nobody chose all belong here. The remedy is institutional, which is why this branch is largely policy.
The Center for AI Safety's textbook cuts the same ground into four sources, malicious use, an AI race, organisational failure and rogue systems, and the mapping is direct. Malicious use is misuse, organisational failure and rogue systems are accidents, and the race is structural. Asking which category a given worry belongs to is the useful discipline, since a better model does not solve a misuse problem and a policy does not solve an accident.
Present and future harm
Classifying a risk settles what kind of remedy it needs. It does not settle how much attention each kind deserves, and that question is the field's loudest internal disagreement.
One position holds that attention belongs with the harms already occurring. Discrimination in automated decisions, surveillance, labour displacement and fabricated information are all measurable today, and they fall hardest on those with the least recourse.
The other holds that attention belongs further out. Capability is rising fast enough that future systems will pose risks of a different order, and preparation has to begin before that capability arrives, since afterwards is too late.
The two are frequently presented as opposed camps, and the practical work overlaps heavily. Evaluation, interpretability, oversight and incident response all serve both positions. Knowing that the argument exists is what allows a paper's framing to make sense, and it prevents the field being read as a single view.
Alignment and control
Both positions use the same two words for the problem, and both use them loosely, so the pair is worth separating.
Alignment concerns whether a system pursues what its designers intended. Where a model optimises a proxy that diverges from the goal, that model is misaligned however capable it may be.
Control concerns whether people are able to correct or stop a system, regardless of whether it is aligned. Oversight, interruptibility and permissions are all control problems, and they matter precisely because alignment cannot be assumed.
Most practical safety work is control work. It does not require that alignment be solved, and it does require a decision about what a system is permitted to do.
Why a learned system cannot be specified
Control is a harder problem here than it would be in ordinary software, and the reason is worth stating plainly. Ordinary software is specified. Someone writes down what it should do, and the software can then be tested against that statement.
A learned system carries no specification in that sense. Its behaviour is inferred from data, described statistically and inspected only from the outside. What it did on the cases that were tried can be measured. The cases that were not tried cannot be listed, so nothing can be said about them with certainty.
Everything else in this field is a response to that single fact. Evaluation exists because the specification does not, interpretability exists because the inspection is only from the outside, and oversight exists because the untested cases are where the system will be used.