Concept 2 of 3

How AI features fail

2 questions test this

Most writing about AI risk is about systems nobody has built yet. This is about the ways the thing you are shipping next quarter will fail, roughly in the order you will meet them.

Confident wrong answers

The model produces something plausible and untrue, in the same tone it uses when it is right. There is no signal in the output that separates the two, which is what makes it dangerous rather than merely inaccurate.

It follows from the objective. A model trained to produce the likeliest continuation optimises for plausibility, and plausibility and truth are different properties that usually coincide. Newer models do it less often and none of them stop.

Answers that are worse for some people

Aggregate accuracy hides subgroup performance completely. A system at 94 per cent overall can be at 97 for one group and 71 for another, and nobody finds out until somebody complains or a regulator asks.

The cause is usually the data. Historical records encode historical decisions, and groups thin in the training data get worse results because there was less evidence about them.

Instructions arriving as content

Anything the system reads can contain instructions. A customer email, a retrieved document, a web page. Because instructions and data share one channel, no wording makes yours authoritative, and the problem stops being theoretical the moment the system can act rather than only answer.

Misuse

People will use it for things you did not intend, including deliberately. Extracting information it should not give, generating content you would not want associated with the product, or automating something at a volume you did not design for.

Quiet degradation

Nothing breaks. The world moves away from what the model learned, the questions people ask change, the provider updates the hosted model underneath you, and answers get worse while every operational dashboard stays green.

This is the failure most likely to run for months, because the absence of complaints is not evidence of quality. Most people who get a poor answer stop using the feature rather than reporting it.

Over reliance

The one that is hardest to control, because it is not a property of the system at all. People accept output they should have checked, and they check less as the system gets better, so accuracy and complacency rise together.

Reading the list

The useful exercise is not to score a system against these. It is to take each one and ask what it would cost here.

A confident wrong answer in a brainstorming tool costs nothing. The same failure in a tool that tells somebody whether their insurance covers a procedure is a different thing entirely, and no amount of model quality changes which of those you are building.

Common misconceptions

The dangerous failures are the dramatic ones.

The failures that reach customers are mundane. A confident wrong answer, a system that works worse for one group, an output nobody checked. None of them makes a headline on its own and all of them are what a product team will meet.

A failure rate low enough to look acceptable is acceptable.

One per cent of a thousand answers a day is ten wrong answers a day, every day, and whether that is tolerable depends entirely on what a wrong answer does. The rate means nothing without the consequence beside it.

These are bugs, so they can be fixed.

Some are, and several follow from how the technology works and can only be designed around. Hallucination and nondeterminism are properties rather than defects, which is why the answer is grounding, review and rollback rather than a patch.

2 questions test this concept

Which failure is most likely to run for months without anybody noticing?

  • AA model producing confidently wrong answers on a high volume feature.
  • BQuiet degradation, where the world moves away from what the model learned and answers get worse while every dashboard stays green.
  • CAn outage in the model serving endpoint.
  • DA prompt injection that causes the system to reveal its instructions.
Check whether it stuck.

One per page, with a worked explanation.

Start the set
Related material
Book
Weapons of Math Destruction, On failures that scale quietly and are hard to appeal.
Book
AI Engineering, On designing around failure modes rather than hoping past them.
Template
Launch checklist, Everything that has to happen from two weeks out to one week after release, grouped by when it falls due, each line with a named owner and a go or no go decision on the day.
Template
Experiment brief, The belief being tested, a hypothesis with a threshold and a date, how long the test runs and on how much traffic, and a table saying in advance what you will do with each possible result.