The failure that is hardest to engineer around, because it is not in the system. It is in the person reading the output.
What it is
Automation bias is the tendency to favour a machine's suggestion over your own judgement, and to stop looking for evidence that contradicts it. It shows up in two directions. Accepting a wrong answer the system gave, and failing to act on something the system did not flag.
It is well established outside AI, in aviation, medicine and industrial control, and everything learned there applies. What is new is how quickly a generative system produces fluent, confident output across domains where the reader has no way to judge it.
Why it gets worse as the system improves
People calibrate their attention to how often they have been let down. A system that is wrong often is checked. A system that is wrong rarely teaches its users, correctly and rationally, that checking is not worth the effort.
So reliability and complacency rise together, and the rare failure arrives precisely when nobody is looking for it. This is why "we will improve the model and the problem goes away" is backwards.
Where it bites hardest
When the reader cannot judge. Legal, medical, financial and technical answers given to people without the expertise to challenge them.
When output is fluent. Confidence in the prose is read as confidence in the content, and a model has no way to sound uncertain about the things it is uncertain about.
When volume is high. Attention per item falls as items per hour rise, and review becomes a glance.
When accepting is one click and checking is five. Whatever the policy says, the interface decides.
What a product can do
Show the sources. Grounding an answer and citing what it came from turns checking from a research task into a click. This is the single most effective control available.
Say what it does not know. A system that declines is trusted better than one that always answers, and it trains the right habit.
Make disagreement cheap. Overriding should be fast and unremarkable, and it should not require a justification nobody has time to write.
Vary the friction by stakes. A low consequence suggestion can be one click. A consequential one can require the reader to look at the evidence before accepting, and the small delay is the point.
Watch the acceptance rate. If reviewers accept everything, that is the finding. It means the review is not happening, whatever the process document says.
The honest framing
You cannot remove over reliance, and you can decide how much of your safety case rests on it. A design whose safety depends on people carefully checking every output has no safety case at all, because that is not what people do.
Practise this
You need two colleagues who will not compare notes for ten minutes, and one short piece of writing with two errors planted in it.
Take about a hundred and fifty words on a subject both of them know well, and plant two mistakes, one plainly wrong and one merely unsupported. Give the identical text to each colleague with a different sentence attached, time them without saying you are timing them, and tell them both what you were testing as soon as they hand it back.
Same text, two framings
To colleague A
"Here is a draft our assistant model generated. Could you
mark anything you think is wrong? Take as long as you like."
To colleague B
"Here is a draft someone on the team wrote up. Could you
mark anything you think is wrong? Take as long as you like."
Record for each reader
Seconds spent ......
Errors marked ...... of 2
Checked any claim against a source? Y / N
Marked anything that was in fact correct? Y / N
Compare the time before you compare the errors found. What matters is not which reader caught more, it is whether the label moved the effort at all, and in most teams it does. Some readers go harder at the model because they distrust it, others skim it because fluent output reads as settled, and either way the words were identical and the label did the work. If your two readers behave the same, run it again on something longer, where checking would cost more.
Ten minutes of this shows that scrutiny is set by presentation rather than by content, which is the half of the problem a product can actually design for.