We can build systems whose behaviour nobody can account for. Interpretability is the attempt to change that, and it is worth knowing both what it has achieved and how far it has to go.
Why it is hard
A large model holds billions of parameters with no labels and no structure anybody designed. Nothing corresponds to a variable a person named.
Worse, representations are distributed and superposed. A single concept is spread across many neurons, and a single neuron participates in many unrelated concepts, because a network with limited width packs more features into it than it has dimensions. So there is rarely a neuron to point at.
What the field has established
Features can be found. Techniques including sparse autoencoders have pulled interpretable features out of real models, corresponding to recognisable things such as a concept, a tone, or a piece of code syntax.
Behaviour can be steered. Having found a feature, amplifying or suppressing it changes output in the predicted direction. That is a strong result, because it shows the feature is doing work rather than merely correlating.
Circuits exist. Small, specific computations have been traced end to end in smaller models, showing how a result is produced rather than only what it correlates with.
Probes work. A simple classifier trained on internal activations can often tell whether a model internally represents something, including whether a statement is true, which is interesting when the output says otherwise.
What it has not established
No complete account of any frontier model. The findings are islands, and the work does not currently scale to explaining a whole system's behaviour on an arbitrary input.
So interpretability cannot yet certify a model as safe, or guarantee an absence, or generally answer why one particular output happened.
Stated reasoning is not interpretability
The most useful thing for a product person to hold on to.
When a model shows its working, that text is generated the same way the answer was. It is a plausible account, not a report from inside. Research has found cases where the stated reasoning omits the factor the model was demonstrably responding to, while sounding entirely coherent.
Chain of thought genuinely improves accuracy on multi step problems, because the intermediate tokens give the model something to condition on. That is a performance benefit and not a window.
Post hoc attribution
For models that cannot explain themselves, feature attribution methods such as SHAP approximate an explanation by measuring how much each input contributed.
They are approximations. They can be unstable, they mislead when inputs are correlated, and they describe the model rather than the world. Better than nothing, and not a faithful account.
What to do while the research runs
Where a decision needs a reason, use a model that can give one rather than a model plus an explanation technique. Where that is not possible, document the system rather than the decision, which is what a model card is for, and keep a person able to overrule it.