Glossary

Automation bias

Trusting the machine’s answer over your own, and over the evidence in front of you. Nobody arrives at it through laziness. A tool that is right nine times in ten teaches you, quite correctly, that checking it wastes an afternoon. Then the tenth arrives dressed exactly like the other nine.

Also known as over-reliance on automation

Automation bias is the habit of trusting a machine’s answer over your own, and over the evidence in front of you. It comes in two shapes: doing what the system says when it is wrong, and missing what the system failed to flag. The second is quieter and does more damage, because nothing happened. There was no alarm to ignore.

Drivers have followed a satnav into a field. Clinicians have talked themselves out of a correct reading because the software disagreed. The people in both cases were competent and paying attention, which is the part that matters. This is not laziness. It is training. A tool that is right nine times out of ten teaches you, correctly, that checking it wastes effort, and then the tenth arrives dressed exactly like the other nine.

The uncomfortable turn is that the risk grows with reliability rather than against it. A system that failed often would keep you sharp. It is the very good ones that quietly retire your judgement, and they do it while every outcome still looks fine.

See also Cognitive offloading Cognitive surrender Discernment Myth of verification

References

Where it comes up in the sources