Concept 2 of 4

Over reliance and automation bias

2 questions test this

The failure that is hardest to engineer around, because it is not in the system. It is in the person reading the output.

What it is

Automation bias is the tendency to favour a machine's suggestion over your own judgement, and to stop looking for evidence that contradicts it. It shows up in two directions. Accepting a wrong answer the system gave, and failing to act on something the system did not flag.

It is well established outside AI, in aviation, medicine and industrial control, and everything learned there applies. What is new is how quickly a generative system produces fluent, confident output across domains where the reader has no way to judge it.

Why it gets worse as the system improves

People calibrate their attention to how often they have been let down. A system that is wrong often is checked. A system that is wrong rarely teaches its users, correctly and rationally, that checking is not worth the effort.

So reliability and complacency rise together, and the rare failure arrives precisely when nobody is looking for it. This is why "we will improve the model and the problem goes away" is backwards.

Where it bites hardest

When the reader cannot judge. Legal, medical, financial and technical answers given to people without the expertise to challenge them.

When output is fluent. Confidence in the prose is read as confidence in the content, and a model has no way to sound uncertain about the things it is uncertain about.

When volume is high. Attention per item falls as items per hour rise, and review becomes a glance.

When accepting is one click and checking is five. Whatever the policy says, the interface decides.

What a product can do

Show the sources. Grounding an answer and citing what it came from turns checking from a research task into a click. This is the single most effective control available.

Say what it does not know. A system that declines is trusted better than one that always answers, and it trains the right habit.

Make disagreement cheap. Overriding should be fast and unremarkable, and it should not require a justification nobody has time to write.

Vary the friction by stakes. A low consequence suggestion can be one click. A consequential one can require the reader to look at the evidence before accepting, and the small delay is the point.

Watch the acceptance rate. If reviewers accept everything, that is the finding. It means the review is not happening, whatever the process document says.

The honest framing

You cannot remove over reliance, and you can decide how much of your safety case rests on it. A design whose safety depends on people carefully checking every output has no safety case at all, because that is not what people do.

Common misconceptions

Over reliance is a training problem, so telling people to check fixes it.

Warnings are read once and then become furniture. Behaviour follows what is easy and what is measured, so a design that makes checking slow and accepting fast produces acceptance whatever the notice says.

A more accurate system reduces over reliance.

It increases it. People calibrate to how often they have been let down, so a system right ninety nine times in a hundred teaches them not to look, and the hundredth passes through unchecked.

Showing a confidence score lets people judge for themselves.

Only if the score is calibrated and understood, and a generative model's stated confidence is generated text rather than a measurement. An uncalibrated number adds authority without adding information.

2 questions test this concept

A team improves a model so that it is now wrong far less often. What effect should they expect on how carefully reviewers check its output?

  • AReviewers will check less carefully, because people calibrate their attention to how often they have been let down.
  • BReviewers will check more carefully, since the system is now trusted with more consequential work.
  • CReview behaviour is unaffected by model accuracy.
  • DReviewers will check more carefully if they are told the model has improved.
Check whether it stuck.

One per page, with a worked explanation.

Start the set
Related material
Book
Weapons of Math Destruction, On scores people stop questioning.
Book
Thinking with Data, On what a number does to a decision.