Interpretability is the attempt to account for what happens inside a model, so that its behaviour can be explained from its internals and checked against them. Systems whose behaviour nobody can account for can be built, and are, and interpretability is the research field trying to change that. It is worth knowing both what it has achieved and how far it has to go, because a great deal of what is sold as explanation is nothing of the kind.
The sections below say why the problem is hard, what the field has established, what it has not, why a model's stated reasoning is a separate thing, what the older attribution methods can and cannot do and what to do while the research runs.
Why it is hard
A large model holds billions of parameters with no labels and no structure anybody designed. Nothing corresponds to a variable a person named.
Worse, representations are distributed and superposed. A single concept is spread across many neurons, and a single neuron participates in many unrelated concepts, because a network with limited width packs more features into it than it has dimensions. The phenomenon was described formally in 2022, and it is the reason there is rarely a neuron to point at.
What the field has established
Features can be found. Techniques including sparse autoencoders have pulled interpretable features out of real models, corresponding to recognisable things such as a concept, a tone or a piece of code syntax. In 2024 one laboratory extracted millions of such features from a production model and published a map of them.
Behaviour can be steered. Having found a feature, amplifying or suppressing it changes output in the predicted direction. The best known demonstration made a model talk about a particular bridge in answer to almost anything. That is a strong result, because it shows the feature is doing work and is more than a correlation.
Circuits exist. Small, specific computations have been traced end to end, first in small models and by 2025 in production ones, showing how a result is produced and how a model plans several words ahead when writing a rhyme.
Probes work. A simple classifier trained on internal activations can often tell whether a model internally represents something, including whether a statement is true, which is interesting when the output says otherwise.
What it has not established
No complete account of any frontier model exists. The findings are islands, and the work does not currently scale to explaining a whole system's behaviour on an arbitrary input.
So interpretability cannot yet certify a model as safe, or guarantee an absence, or generally answer why one particular output happened.
Stated reasoning and interpretability
The gap between the two is the most useful thing on this page for a product person to hold on to. When a model shows its working, that text is generated the same way the answer was. It is a plausible account, and no report from inside. Research published in 2023 found cases where the stated reasoning omitted the factor the model was demonstrably responding to, such as the order of the answer options, while sounding entirely coherent.
Chain of thought genuinely improves accuracy on multi step problems, because the intermediate tokens give the model something to condition on. That is a performance benefit, and it opens no window.
Post hoc attribution
For models that cannot explain themselves, feature attribution methods such as SHAP approximate an explanation by measuring how much each input contributed to the output.
They are approximations. They can be unstable, they mislead when inputs are correlated, and they describe the model and never the world. Better than nothing, and no faithful account.
What to do while the research runs
Where a decision needs a reason, a model that can give one is used, and a model plus an explanation technique is a poor substitute. Where that is not possible, the system is documented where the decision cannot be, which is what a model card is for, and a person is kept able to overrule it.