The failure that is hardest to engineer around, because it is not in the system. It is in the person reading the output.
What it is
Automation bias is the tendency to favour a machine's suggestion over your own judgement, and to stop looking for evidence that contradicts it. It shows up in two directions. Accepting a wrong answer the system gave, and failing to act on something the system did not flag.
It is well established outside AI, in aviation, medicine and industrial control, and everything learned there applies. What is new is how quickly a generative system produces fluent, confident output across domains where the reader has no way to judge it.
Why it gets worse as the system improves
People calibrate their attention to how often they have been let down. A system that is wrong often is checked. A system that is wrong rarely teaches its users, correctly and rationally, that checking is not worth the effort.
So reliability and complacency rise together, and the rare failure arrives precisely when nobody is looking for it. This is why "we will improve the model and the problem goes away" is backwards.
Where it bites hardest
When the reader cannot judge. Legal, medical, financial and technical answers given to people without the expertise to challenge them.
When output is fluent. Confidence in the prose is read as confidence in the content, and a model has no way to sound uncertain about the things it is uncertain about.
When volume is high. Attention per item falls as items per hour rise, and review becomes a glance.
When accepting is one click and checking is five. Whatever the policy says, the interface decides.
What a product can do
Show the sources. Grounding an answer and citing what it came from turns checking from a research task into a click. This is the single most effective control available.
Say what it does not know. A system that declines is trusted better than one that always answers, and it trains the right habit.
Make disagreement cheap. Overriding should be fast and unremarkable, and it should not require a justification nobody has time to write.
Vary the friction by stakes. A low consequence suggestion can be one click. A consequential one can require the reader to look at the evidence before accepting, and the small delay is the point.
Watch the acceptance rate. If reviewers accept everything, that is the finding. It means the review is not happening, whatever the process document says.
The honest framing
You cannot remove over reliance, and you can decide how much of your safety case rests on it. A design whose safety depends on people carefully checking every output has no safety case at all, because that is not what people do.