Most writing about AI risk is about systems nobody has built yet. This page is about the ways a feature built on a language model fails once it is in front of customers, in roughly the order a product team meets them. Six failure modes cover most of what happens, and each has a section below. The first three have the most written about them, and the last three do the most damage.
None of the six is exotic. Each has a cause, usually in how the model was trained or in how the feature was wired to it, and knowing the cause is what says which control has a chance of working. The sections take the six in turn, then ask what each would cost on a particular feature, which is the question the list exists for.
Confident wrong answers
The model produces something plausible and untrue, in the same tone it uses when it is right. There is no signal in the output that separates the two, which is what makes it dangerous where a merely inaccurate system would be an inconvenience.
The cause is the training objective. A model trained to produce the likeliest continuation optimises for plausibility, and plausibility and truth are different properties that usually coincide. The research literature calls the result hallucination and has catalogued it since 2022, and every generation of model has reduced the rate without removing it. A design that depends on the rate reaching zero is unsound.
Answers that are worse for some people
Aggregate accuracy hides subgroup performance completely. A system at 94 per cent overall can be at 97 for one group and 71 for another, and nobody finds out until somebody complains or a regulator asks.
The best known measurement of this was the Gender Shades study of 2018, which tested three commercial face analysis products and found error rates below one per cent for lighter skinned men and above thirty per cent for darker skinned women, on systems whose overall accuracy looked respectable. The cause is usually the data. Historical records encode historical decisions, and groups thin in the training data get worse results because there was less evidence about them.
Instructions arriving as content
Anything the system reads can contain instructions. A customer email, a retrieved document, a web page. The model has one channel for instructions and for data, so no wording makes the developer's instruction authoritative over text that arrives later, and the problem stops being theoretical the moment the system can act as well as answer.
This failure has a name, prompt injection, given to it in 2022, and it sits at the top of the OWASP list of risks for language model applications. It has a page of its own on the Practitioner course.
Misuse
People will use the feature for things it was never intended for, including deliberately. Extracting information it should not give, generating content the organisation would not want associated with the product and automating something at a volume the feature was never designed for are the common forms.
Quiet degradation
Nothing breaks. The world moves away from what the model learned, the questions people ask change, the provider updates the hosted model underneath the feature, and answers get worse while every operational dashboard stays green.
This is the failure most likely to run for months, because the absence of complaints is no evidence of quality. Most people who get a poor answer stop using the feature and never report it.
Over reliance
The last failure is the hardest to control, because it is a property of the people and never of the system. People accept output they should have checked, and they check less as the system gets better, so accuracy and complacency rise together. Human factors research has described this as automation bias since the 1990s, long before language models, and it applies to a spreadsheet as much as to a model.
What each failure costs here
The list is worth little as a scorecard. Its use is to take each mode and ask what it would cost on the feature in hand.
A confident wrong answer in a brainstorming tool costs nothing. The same failure in a tool that tells somebody whether their insurance covers a procedure is a different thing entirely, and no amount of model quality changes which of those is being built.
Practise this
The exercise needs one feature that is live or nearly live, the six failure modes above and fifteen minutes.
Take each mode in turn and write the sentence describing how the team would first hear that it was happening, naming the actual alert, sampling routine or report and never the general idea of one. Then mark the row with the letter for what that answer really is. Write what happens today, because a control that exists in a ticket detects nothing.
Feature ..........................................
Failure mode How the team would first hear Source
Confident wrong answer ............................... A M U N
Worse for some people ............................... A M U N
Instructions in content ............................... A M U N
Misuse ............................... A M U N
Quiet degradation ............................... A M U N
Over reliance ............................... A M U N
A an automated alert fires
M a person looks at something on a schedule
U a user tells us
N nothing, we would not find out
Count of U ____ Count of N ____
Count the U and N rows before anything else. A U means the harm reached somebody who then took the trouble to complain, which makes the users the monitoring and makes the reported cases a small fraction of the real ones. An N is worse and commoner than teams expect, and it lands most often on quiet degradation and over reliance, neither of which produces an error, a latency spike or a complaint.
The quarter of an hour shows which of the feature's failure modes currently have no detection at all, which is a shorter and more useful list than one ranking them by how much they worry the team.