Content provenance and watermarking

Content provenance is the practice of establishing where a piece of media came from and what has been done to it, and watermarking is one of the techniques used. As generated content becomes indistinguishable from the rest, the question of origin stops being trivia, and in the European Union it has become a legal obligation. Three approaches exist, and they are far from equally sound.

The sections below take the three in order of soundness, from detection after the fact through watermarking to signed provenance, and end with what the law now requires.

Detection after the fact

The first approach trains a classifier to spot generated content. It is the approach people reach for first and it is the weakest.

For text it barely works. Detectors produce false positives on non native English writers and on formal prose, they are defeated by light editing, and several well known ones were withdrawn after being used to accuse students of cheating. The developer of the most widely used chat service withdrew its own text classifier in 2023, citing its low accuracy. A detector's output is no evidence and should not be treated as any.

For images and audio it works better and still degrades quickly as generators improve, which is a race the detectors are structurally behind in.

Watermarking

The second approach marks content at the point of generation, so the mark travels with it.

For images, imperceptible patterns are embedded in the pixels, designed to survive cropping, compression and recompression. Google's SynthID, released in 2023, is the best known.

For text it is harder, because there is much less room to hide anything. The approach biases the model's token choices in a pattern detectable statistically, which needs a reasonable length of text and is weakened by paraphrasing. Google published its text watermarking method openly in 2024, and the major providers have since agreed to mark generated images and audio, with text marking still uneven.

The property to hold on to is the asymmetry. A watermark that is present is evidence. Its absence proves nothing, because it can be removed and because most content was made by something that never applied one.

Provenance and signing

The third approach is the stronger idea, and it is where the standards work is. Genuine content is signed cryptographically and a record is kept of what happened to it, so that proving something is fake is never required.

C2PA, the Coalition for Content Provenance and Authenticity, is the standard, founded in 2021 by a group including camera makers, software companies and a broadcaster. Metadata records the camera or tool that produced a file and each edit applied, signed so tampering is detectable, and camera manufacturers have begun shipping devices that sign at the moment of capture.

The advantage is direction. Verifying that something is what it claims is a tractable problem, and proving something is fake is not.

The weakness is coverage. Provenance only helps where it exists, and metadata is routinely stripped by platforms that recompress uploads, so it depends on an ecosystem and never on a single vendor.

What the law now requires

The EU AI Act places transparency duties on systems that generate synthetic content, requiring that it be marked as artificially generated in a machine readable form, and that people be told when they are interacting with a system and never a person. Those duties have applied since August 2026, and they were left on their original date when the high risk obligations in the rest of the Act were pushed back. Other jurisdictions are moving similarly, particularly on political content and on synthetic likenesses of real people, and California passed a law in 2024 requiring providers of large generative systems to offer detection tools and to embed provenance data, which takes effect during 2026.

For a product team the practical reading is that a product generating media at scale carries an obligation to mark it, and the mechanism is the provider's watermarking plus provenance metadata the product does not strip.

Common misconceptions

“Detectors can tell whether text was written by a model.”

Not reliably. They produce false positives on non native English writing and on formal prose, and they are defeated by light editing. Several were withdrawn after being used to accuse students, and a detector's confident output is no evidence.

“A watermark proves content was generated.”

A watermark that survives is evidence it was. Its absence proves nothing, because it can be stripped by cropping, recompressing or paraphrasing, and because most content is produced by systems that never applied one.

“Provenance is about catching people who use AI.”

The more durable use is the other direction. Signing genuine content so it can be verified is more robust than trying to detect fakes, which is why the standards work is about proving authenticity and leaves spotting generation to one side.

Where this is examined
AI Safety Foundation
Adversarial Attacks and Data Privacy, 12 per cent of the exam.
Related material
Book
Data and Goliath, On metadata and what it establishes.
Concepts