Weapons of Math Destruction

O'Neil, a mathematician who worked in finance and advertising technology, examines algorithms that operate at scale, remain opaque to the people they judge, and cause damage that feeds back into their own training data. Case studies cover university rankings, recidivism scoring, personality tests used in hiring, and the pricing of credit and insurance. The argument running through them is that a model encodes the priorities of whoever built it, and that a proxy chosen for convenience quietly becomes the thing being measured. She is specific about the feedback loops that make these systems worsen the conditions they claim to predict. Read it before writing the success metric for anything that scores a person.