Adversarial machine learning studies attacks aimed at a model itself, rather than at the software around it, and the defences against them. Ordinary security protects code, networks and credentials. These attacks go after something else. Many of them exploit the way learning works rather than any mistake anyone has made, which is also why some have no clean fix. The field opened with a 2013 paper showing that an image could be altered imperceptibly and still change a classifier's answer. By 2024 the United States standards body had published a taxonomy of the field for practitioners.
The sections below take the attacks in the order the taxonomy uses, evasion, poisoning, extraction and membership inference, then the ordinary security failures behind most real incidents, and end with what all of it means in practice.
Evasion attacks, also called adversarial examples
An evasion attack changes an input slightly, often imperceptibly, so the model gets it wrong while a person sees nothing unusual. A few pixels can alter a classification. In a 2017 experiment a few stickers on a stop sign made a vision system read it as a speed limit.
What makes this hard to fix is where it comes from. A classifier divides its input space with a boundary, and in a space of thousands of dimensions almost every point sits close to that boundary. A small push in the right direction crosses it. The behaviour follows from the way these models separate one class from another, so a decade of work has produced defences against particular attacks and no general fix.
Adversarial training feeds attacked examples back into training. It helps against the attacks it was shown and costs accuracy on ordinary input, which is the trade every defence in this area makes.
Attacks also transfer. An attack built against a model the attacker can see often works against a different model trained for the same task, so keeping weights private protects less than it appears to.
For language models the same family arrives as prompt injection and jailbreaking. The medium is text and the attack has the same shape, an input crafted to move the model somewhere its designers did not intend.
Poisoning
Poisoning corrupts what the model learns from. Where an attacker can influence training data or a retrieval index, they can plant behaviour that surfaces later.
A backdoor is the most deliberate form. The model behaves normally until a trigger appears, and then it does whatever the attacker chose. It passes evaluation, because the evaluation set does not contain the trigger.
Poisoning grows as a risk as systems pull in more outside content. Anything scraped, crowdsourced or accepted from users is a surface, and a retrieval index built from documents anybody can edit is poisonable without touching the model at all. Research published in 2023 put a price on it. Poisoning a small fraction of a web scale dataset turned out to be within an ordinary budget, because a crawler's route through the web can be predicted and some of the expired domains on that route can be bought.
Extraction
Extraction rebuilds a model by querying it. With enough input and output pairs an attacker can train a copy that approximates the original, which matters commercially wherever the model is the product. A 2016 paper showed the attack working against commercial prediction services with a few thousand queries.
Rate limiting and monitoring for systematic querying are the practical defences, and neither is complete.
Membership inference
Membership inference determines whether a particular record was in the training data. It sounds minor. It is a privacy breach on its own, because membership can be the sensitive fact. Knowing that somebody's record was in a dataset of patients with a condition discloses the condition. The attack was demonstrated in 2017 against models served through ordinary prediction interfaces, using nothing beyond the confidence scores they returned.
Model theft and supply chain
The ordinary security problems are also the most likely. Weights left in accessible storage, a compromised dependency in the training pipeline, a model downloaded from a public hub with nobody checking what is inside it.
These are less interesting than adversarial examples and far more likely to be how a real incident starts.
What this means in practice
Two things carry over from the whole list.
Treat model input as untrusted, always. Whether it arrives as an image, a document or a prompt, it may have been constructed to move the model.
Assume defences are partial. None of these attacks has a complete fix, so the security case cannot rest on the model resisting them. It rests on limiting what a manipulated model is permitted to do, which is why least privilege and oversight do the work that robustness cannot.