Memorisation is what a model retains from its training data closely enough to reproduce it, and extraction is the family of attacks that pull it back out. A model does not contain a copy of its training data, and it can reproduce parts of it. Both statements are true, and the gap between them is where the privacy problem lives. The subject matters to anyone who trains or fine tunes a model on data they would not publish, which is most organisations.
The sections below explain what memorisation is and what drives it, how extraction works, what membership inference adds, what differential privacy does and costs and what actually reduces the risk.
What memorisation is
Training on a corpus produces a model that generalises. Along the way it also memorises, retaining some sequences closely enough to reproduce them.
The strongest driver is repetition. Text appearing many times across a corpus is far more likely to come back verbatim, which is why licence headers, common code and widely copied passages emerge readily.
Rarity matters too, for the opposite reason. An unusual record has nothing similar for the model to blend it with, so it is retained more distinctly than a typical one.
And larger models memorise more, so the property strengthens with the scale that also brings capability.
Extraction
Researchers have repeatedly pulled training data out of production models using nothing but prompting, recovering personal information, code and long passages of text. The first large demonstration came in 2021 against a model whose training data was known, and in 2023 a group extracted training text from a widely used chat service with an inelegant technique, asking the model to repeat a single word indefinitely until it drifted into reciting what it had been trained on.
Two things follow. Training data is not private simply because the corpus is unpublished. And the same applies to anything a team fine tunes on, so a model tuned on customer records is a system that may recite customer records.
Membership inference
A weaker attack has a sharper edge. The attacker recovers nothing and establishes only whether a particular record was in the training set.
That can be the sensitive fact by itself. If the training set was patients with a diagnosis, membership discloses the diagnosis, and no content needs to be recovered at all.
Differential privacy
The principal formal defence was defined in 2006 and adapted to deep learning in 2016. Noise is added during training so that the finished model is provably almost unchanged by any single record, which bounds what can be inferred about any one person.
It is a real guarantee, with mathematics behind it, and it costs accuracy. Stronger privacy means more noise means a weaker model, and the tradeoff is a decision. It is used where the data is sensitive enough to justify the loss, and several large technology companies use it for telemetry and keyboard prediction.
What actually reduces the risk
The formal guarantee is expensive, and five cheaper measures do most of the work.
Not collecting it. The only complete defence, and the one most often skipped.
Deduplicating the corpus. Since repetition drives memorisation, removing duplicates reduces it substantially and costs nothing in capability.
Removing sensitive records before training, because output filtering afterwards catches only what it recognises.
Retrieval ahead of fine tuning. Facts kept in an index can be deleted, access controlled and corrected, where facts in weights cannot. This is the practical answer for most product teams, and it is why the choice between retrieval and fine tuning is a privacy decision as much as an engineering one.
Anonymisation and pseudonymisation, which help and are insufficient, because combinations of ordinary fields identify people even after direct identifiers are gone.