Concept 3 of 3

Content provenance and watermarking

2 questions test this

As generated content becomes indistinguishable from the rest, the question of where a piece of media came from stops being trivia. Three approaches exist and they are not equally sound.

Detection after the fact

Training a classifier to spot generated content. This is the approach people reach for first and it is the weakest.

For text it barely works. Detectors produce false positives on non native English writers and on formal prose, they are defeated by light editing, and several well known ones were withdrawn after being used to accuse students of cheating. A detector's output is not evidence and should not be treated as any.

For images and audio it works better and still degrades quickly as generators improve, which is a race the detectors are structurally behind in.

Watermarking

Marking content at the point of generation, so the mark travels with it.

For images, imperceptible patterns embedded in the pixels, designed to survive cropping, compression and recompression. Google's SynthID is the best known.

For text it is harder, because there is much less room to hide anything. The approach biases the model's token choices in a pattern detectable statistically, which needs a reasonable length of text and is weakened by paraphrasing.

The property to hold on to is the asymmetry. A watermark that is present is evidence. Its absence proves nothing, because it can be removed and because most content was made by something that never applied one.

Provenance and signing

The stronger idea, and where the standards work is. Rather than detecting fakes, cryptographically sign genuine content and record what happened to it.

C2PA, the Coalition for Content Provenance and Authenticity, is the standard. Metadata records the camera or tool that produced a file and each edit applied, signed so tampering is detectable.

The advantage is direction. Verifying that something is what it claims is a tractable problem, where proving something is fake is not.

The weakness is coverage. Provenance only helps where it exists, and metadata is routinely stripped by platforms that recompress uploads, so it depends on an ecosystem rather than a vendor.

What the law is starting to require

The EU AI Act places transparency duties on systems that generate synthetic content, requiring that it be marked as artificially generated in a machine readable form, and that people be told when they are interacting with a system rather than a person. Other jurisdictions are moving similarly, particularly on political content and on synthetic likenesses of real people.

For a product team the practical reading is that if you generate media at scale, marking it is becoming an obligation rather than a courtesy, and the mechanism is the provider's watermarking plus provenance metadata you do not strip.

Common misconceptions

Detectors can tell whether text was written by a model.

Not reliably. They produce false positives on non native English writing and on formal prose, and they are defeated by light editing. Several were withdrawn after being used to accuse students, and a detector's confident output is not evidence.

A watermark proves content was generated.

A watermark that survives is evidence it was. Its absence proves nothing, because it can be stripped by cropping, recompressing or paraphrasing, and because most content is produced by systems that never applied one.

Provenance is about catching people who use AI.

The more durable use is the other direction. Signing genuine content so it can be verified is more robust than trying to detect fakes, which is why the standards work is about proving authenticity rather than spotting generation.

2 questions test this concept

A university proposes using an AI text detector to identify submitted work written by a model. What is the strongest objection?

  • ADetectors are too slow to run at the volume of a university's submissions.
  • BDetectors require access to the model that generated the text.
  • CDetectors produce false positives on non native English writing and formal prose, are defeated by light editing, and their output is not evidence.
  • DDetection is prohibited under data protection law.
Check whether it stuck.

One per page, with a worked explanation.

Start the set
Related material
Book
Data and Goliath, On metadata and what it establishes.