One strand of AI safety argues that the systems now being built could produce harm of a kind there is no recovering from, and that this differs from the harms already documented, with no larger version of them in it. An equally serious strand argues that the claim is unsupported and that acting on it is costly. The argument became public in 2023, when a one sentence statement organised by the Center for AI Safety, signed by the heads of the largest laboratories and by two of the three researchers who shared the 2018 Turing Award, said that the risk of extinction from AI should be treated as a global priority alongside pandemics and nuclear war. The third Turing laureate declined to sign and has argued the opposite case since. Most writing in this field is positioned against that argument whether it says so or not, so anyone reading in the field has to be able to place it.
The sections below give the argument in its standard form, the two moves that carry most of its weight, the four critiques that land, the reason the two camps end up building the same things and a method for placing a paper without adopting a position.
What the argument from capability says
The starting observation is that capability has risen quickly, largely by increasing scale, and that the increases have been hard to predict before they arrived. If that continues, the argument runs, systems will eventually do things no evaluation was written to look for, and waiting to prepare until then is preparing afterwards.
The word catastrophic carries a second claim, about recovery. Ordinary software failures are bad and bounded. A harm that spreads faster than institutions can respond, or that removes the ability to correct it, is argued to sit in its own category and to justify precaution that normal risk management would not.
What autonomy and opacity add
Two further moves do most of the work.
The first is reach. A model producing text for a person to read is bounded by that person. A system given tools, budgets and the ability to act over many steps has effects with no review point in between, so an unchanged error rate produces a different consequence.
The second is verification. A trained model has no specification it can be checked against, and its behaviour is known only from the cases somebody tried. The argument holds that a capability nobody tested for therefore cannot be ruled out, and that generality grows the untested space faster than testing covers it.
The four critiques
The critiques come from people working on the same systems, and four of them land.
The case is extrapolation. It is built from a trend line and a chain of reasoning, and no link in the chain has been observed. Predicting where a technology goes from the curve it has traced so far has a poor record in every field it has been tried in, and the research on expert forecasting finds that confident long range prediction does little better than chance.
It competes with harm that is already measurable. Discrimination in automated decisions, surveillance and fabricated claims about real people are happening now, to people with the least recourse. Critics argue that speculative risk absorbs funding, regulatory attention and seriousness that documented harm needs more, and the 2021 paper that described large language models as stochastic parrots made this case before the public argument began.
It can serve the people making it. If a technology needs licensing, expensive audits and restricted release, the organisations best placed to comply are the largest incumbents. This is less an accusation of bad faith than an observation about whose position a framing happens to strengthen.
The track record is thin. Confident predictions about which capabilities would arrive when, and about what would follow from them, have been wrong often enough and in both directions that firm expectations about the next decade are hard to justify.
Why the two camps build the same things
Whichever critique lands, the disagreement is about weighting and expectation, and rarely about what to do next. Capability evaluation, interpretability, human oversight, red teaming and incident response are the practical programme on either account, because a system whose behaviour cannot be measured or interrupted is a problem whether the worst case is a wrongly refused loan or something larger.
That overlap is why a product person does not have to settle the argument before acting. Nearly everything else on these two courses is agreed ground.
How to place a paper
Placing a paper is easier than settling the argument, and it takes two questions. The first is which kind of claim the paper is making. A harm already observed, an extrapolation from a trend and a governance conclusion drawn from either are three different things, and papers move between them without marking the change while the confident tone stays constant.
The second question is what would change the author's mind. A position no evidence could revise is a commitment and no longer an argument, and that applies to the confident dismissal exactly as it does to the confident warning.