Over reliance is the failure that is hardest to engineer around, because it lives in the person reading the output and never in the system. Automation bias is the name human factors research gives to it, and the research is decades old. A generative system adds one new ingredient, which is fluent, confident output across domains where the reader has no way to judge it.
The sections below define the bias, explain why it gets worse as a system improves, name the situations where it bites hardest, list what a product can do about it and end with how much of a safety case can honestly rest on people checking.
What automation bias is
Automation bias is the tendency to favour a machine's suggestion over one's own judgement, and to stop looking for evidence that contradicts it. It shows up in two directions. Accepting a wrong answer the system gave, and failing to act on something the system did not flag.
It was named in the 1990s in studies of pilots and of clinicians using decision support, and a 1983 paper on the ironies of automation had already observed that the more reliable a machine becomes, the less practised and the less attentive the person overseeing it becomes. A 2023 study of radiologists reading mammograms found that when an assistant system gave a wrong suggestion, accuracy fell sharply even among experienced readers. Everything learned in those fields applies to a language model.
Why it gets worse as the system improves
People calibrate their attention to how often they have been let down. A system that is wrong often is checked. A system that is wrong rarely teaches its users, correctly and rationally, that checking is not worth the effort.
So reliability and complacency rise together, and the rare failure arrives precisely when nobody is looking for it. This is why improving the model does not make the problem go away, and why a team that plans to rely on improvement has the argument backwards.
Where it bites hardest
The bias is worst in four situations, and most AI features sit in at least one.
When the reader cannot judge. Legal, medical, financial and technical answers given to people without the expertise to challenge them.
When output is fluent. Confidence in the prose is read as confidence in the content, and a model has no way to sound uncertain about the things it is uncertain about.
When volume is high. Attention per item falls as items per hour rise, and review becomes a glance.
When accepting is one click and checking is five. Whatever the policy says, the interface decides.
What a product can do
The interface deciding is also the opening, because the interface is what a product team controls.
Show the sources. Grounding an answer and citing what it came from turns checking from a research task into a click. This is the single most effective control available.
Say what it does not know. A system that declines is trusted better than one that always answers, and it trains the right habit.
Make disagreement cheap. Overriding should be fast and unremarkable, and it should not require a justification nobody has time to write.
Vary the friction by stakes. A low consequence suggestion can be one click. A consequential one can require the reader to look at the evidence before accepting, and the small delay is the point.
Watch the acceptance rate. If reviewers accept everything, that is the finding. It means the review is not happening, whatever the process document says.
How much of a safety case can rest on people
Over reliance cannot be removed, and a team can decide how much of its safety case rests on it. A design whose safety depends on people carefully checking every output has no safety case at all, because that is not what people do.
Practise this
The exercise needs two colleagues who will not compare notes for ten minutes, and one short piece of writing with two errors planted in it.
Take about a hundred and fifty words on a subject both of them know well, and plant two mistakes, one plainly wrong and one merely unsupported. Give the identical text to each colleague with a different sentence attached, time them without saying they are being timed, and tell them both what was being tested as soon as they hand it back.
Same text, two framings
To colleague A
"Here is a draft our assistant model generated. Could you
mark anything you think is wrong? Take as long as you like."
To colleague B
"Here is a draft someone on the team wrote up. Could you
mark anything you think is wrong? Take as long as you like."
Record for each reader
Seconds spent ......
Errors marked ...... of 2
Checked any claim against a source? Y / N
Marked anything that was in fact correct? Y / N
Compare the time before comparing the errors found. What matters is whether the label moved the effort at all, and in most teams it does. Some readers go harder at the model because they distrust it, others skim it because fluent output reads as settled, and either way the words were identical and the label did the work. If the two readers behave the same, run it again on something longer, where checking would cost more.
Ten minutes of this shows that scrutiny is set by presentation, and the content had nothing to do with it, which is the half of the problem a product can actually design for.