Concept 2 of 3

Memorisation and training data extraction

3 questions test this

A model does not contain a copy of its training data, and it can reproduce parts of it. Both statements are true and the gap between them is where the privacy problem lives.

What memorisation is

Training on a corpus produces a model that generalises. Along the way it also memorises, retaining some sequences closely enough to reproduce them.

The strongest driver is repetition. Text appearing many times across a corpus is far more likely to come back verbatim, which is why licence headers, common code and widely copied passages emerge readily.

Rarity matters too, for the opposite reason. An unusual record has nothing similar for the model to blend it with, so it is retained more distinctly than a typical one.

And larger models memorise more, so the property strengthens with the scale that also brings capability.

Extraction

Researchers have repeatedly pulled training data out of production models using nothing but prompting, recovering personal information, code and long passages of text. Some techniques are inelegant, including asking a model to repeat a word indefinitely until it drifts into reciting training data.

Two things follow. Training data is not private simply because the corpus is not published. And the same applies to anything you fine tune on, so a model tuned on customer records is a system that may recite customer records.

Membership inference

A weaker attack with a sharper edge. Rather than recovering a record, an attacker establishes whether a particular record was in the training set.

That can be the sensitive fact by itself. If the training set was patients with a diagnosis, membership discloses the diagnosis, and no content needs to be recovered at all.

Differential privacy

The principal formal defence. Noise is added during training so that the finished model is provably almost unchanged by any single record, which bounds what can be inferred about any individual.

It is a real guarantee rather than a heuristic, and it costs accuracy. Stronger privacy means more noise means a weaker model, and the tradeoff is a decision rather than a setting. It is used where the data is sensitive enough to justify the loss.

What actually reduces the risk

Not collecting it. The only complete defence, and the one most often skipped.

Deduplicating the corpus. Since repetition drives memorisation, removing duplicates reduces it substantially and costs nothing in capability.

Removing sensitive records before training rather than filtering output afterwards, because output filtering catches what it recognises.

Retrieval instead of fine tuning. Facts kept in an index rather than in weights can be deleted, access controlled, and corrected. This is the practical answer for most product teams, and it is why the choice between retrieval and fine tuning is a privacy decision as much as an engineering one.

Anonymisation and pseudonymisation, which help and are not sufficient, because combinations of ordinary fields identify people even after direct identifiers are gone.

Common misconceptions

A model does not store its training data, so it cannot leak it.

It does not store it as a copy, and it can reproduce passages verbatim. Researchers have extracted training text from production models by prompting, so the absence of a database is not the absence of the data.

Only rare or unusual records are at risk of memorisation.

Repetition drives memorisation, so anything duplicated across a corpus is most exposed. Rare records are at risk for a different reason, because there is nothing similar for the model to blend them into.

Removing names from training data prevents disclosure.

Anonymisation removes direct identifiers and leaves the combinations that identify people. It reduces the risk and does not remove it, which is why the stronger technique is not collecting the data.

3 questions test this concept

A team argues that because a model does not store its training data as a database, it cannot leak that data. What is wrong?

  • ANothing, since a model holds only weights.
  • BModels do store a compressed copy of the corpus, which can be decompressed.
  • CThe risk exists only for models trained on fewer than a million documents.
  • DTraining data has repeatedly been extracted from production models by prompting, so the absence of a database is not the absence of the data.
Check whether it stuck.

One per page, with a worked explanation.

Start the set
Related material
Book
Data and Goliath, On what is retained and who ends up holding it.