Concept 1 of 2

Interpretability

4 questions test this

We can build systems whose behaviour nobody can account for. Interpretability is the attempt to change that, and it is worth knowing both what it has achieved and how far it has to go.

Why it is hard

A large model holds billions of parameters with no labels and no structure anybody designed. Nothing corresponds to a variable a person named.

Worse, representations are distributed and superposed. A single concept is spread across many neurons, and a single neuron participates in many unrelated concepts, because a network with limited width packs more features into it than it has dimensions. So there is rarely a neuron to point at.

What the field has established

Features can be found. Techniques including sparse autoencoders have pulled interpretable features out of real models, corresponding to recognisable things such as a concept, a tone, or a piece of code syntax.

Behaviour can be steered. Having found a feature, amplifying or suppressing it changes output in the predicted direction. That is a strong result, because it shows the feature is doing work rather than merely correlating.

Circuits exist. Small, specific computations have been traced end to end in smaller models, showing how a result is produced rather than only what it correlates with.

Probes work. A simple classifier trained on internal activations can often tell whether a model internally represents something, including whether a statement is true, which is interesting when the output says otherwise.

What it has not established

No complete account of any frontier model. The findings are islands, and the work does not currently scale to explaining a whole system's behaviour on an arbitrary input.

So interpretability cannot yet certify a model as safe, or guarantee an absence, or generally answer why one particular output happened.

Stated reasoning is not interpretability

The most useful thing for a product person to hold on to.

When a model shows its working, that text is generated the same way the answer was. It is a plausible account, not a report from inside. Research has found cases where the stated reasoning omits the factor the model was demonstrably responding to, while sounding entirely coherent.

Chain of thought genuinely improves accuracy on multi step problems, because the intermediate tokens give the model something to condition on. That is a performance benefit and not a window.

Post hoc attribution

For models that cannot explain themselves, feature attribution methods such as SHAP approximate an explanation by measuring how much each input contributed.

They are approximations. They can be unstable, they mislead when inputs are correlated, and they describe the model rather than the world. Better than nothing, and not a faithful account.

What to do while the research runs

Where a decision needs a reason, use a model that can give one rather than a model plus an explanation technique. Where that is not possible, document the system rather than the decision, which is what a model card is for, and keep a person able to overrule it.

Common misconceptions

A model explaining its reasoning is interpretability.

That explanation is generated the same way the answer was. It can be fluent and unrelated to what actually drove the output, and studies have found stated reasoning that does not match the factor the model was demonstrably responding to.

Open weights make a model interpretable.

They make it inspectable. Being able to read billions of numbers does not tell you why one input produced one output, which is why interpretability is a research field rather than a matter of access.

Interpretability research has not produced anything usable.

It has produced real results, including identifying features inside models and steering behaviour by manipulating them directly. What it has not produced is a complete account of any frontier model, which is a different claim.

4 questions test this concept

A model is asked to show its reasoning before answering, and produces a clear step by step account. What has been obtained?

  • AA faithful record of the computation that produced the answer.
  • BGenerated text about the answer, which can be fluent and unrelated to what actually drove the output.
  • CAn audit trail sufficient to satisfy a right to explanation.
  • DNothing useful, since chain of thought does not improve accuracy.
Check whether it stuck.

One per page, with a worked explanation.

Start the set
Related material
Book
The Hundred-Page Machine Learning Book, On what the layers are actually doing.