Weapons of Math Destruction
About this book
O'Neil, a mathematician who worked in finance and advertising technology, examines algorithms that operate at scale, remain opaque to the people they judge, and cause damage that feeds back into their own training data. Case studies cover university rankings, recidivism scoring, personality tests used in hiring, and the pricing of credit and insurance. The argument running through them is that a model encodes the priorities of whoever built it, and that a proxy chosen for convenience quietly becomes the thing being measured. She is specific about the feedback loops that make these systems worsen the conditions they claim to predict. Read it before writing the success metric for anything that scores a person.
Description via Product Digest.
Topics
Related concepts
- Responsible AI in practiceBias, fairness, robustness, safety and veracity treated as things that can be measured and acted on, plus the legal exposure that arrives with generative output.
- Human oversightWhat it takes for a person in the loop to be oversight and no mere signature, and the three arrangements worth knowing apart.
- Over reliance and automation biasWhy people accept output they should have checked, why it gets worse as the system gets better, and what a product can do about it.
- What AI safety meansThe three kinds of risk the field separates, why each of them requires a different remedy, and where the disagreements inside the field actually lie.
- When not to ship an AI featureThe decision that prevents more harm than any control does, and the four situations where the honest answer is that this should not be built.
- Transparency and explainabilityWhy an accurate model nobody can explain is sometimes the wrong choice, the difference between the two words, and what a model card is for.
- Catastrophic risk arguments and their critiquesThe case that rising capability could produce harm of a different order, the serious objections to it, and why both camps ask for much the same work.
- The alignment problemTwo questions hiding inside one word, whether the objective specified was the right one and whether the trained system actually acquired it.
- Robustness and distribution shiftWhy every accuracy figure belongs to the data it was measured on, and what happens to a deployed model as the world drifts away from that data.
- Specification gaming and goal misgeneralisationTwo ways a system does exactly what it was trained to do and still fails, and why that is harder to fix than disobedience would be.
- AI in project managementWhat a language model genuinely does for a project manager today, what it does not, and the accountability, confidentiality and data questions that come with putting project material into one.
- How AI features failThe ways an AI feature goes wrong in production, ordered by how often a product team actually meets them, with the cause behind each and the control that has a chance against it.