You cannot ask a model how it works. You have to open it up and look.
The field is called mechanistic interpretability. It treats a trained model like an unknown machine and works out what each part does, then proves it by changing the part and watching what happens.
- 1
Record what happens inside
Run text through the model and save the numbers it produces at each layer. These are called activations. There are millions for every word, and on their own they mean nothing to a person.
- 2
Untangle the numbers into features
Single units in the network respond to many unrelated things, because the model packs more ideas in than it has units. A second network, a sparse autoencoder, is trained to pull the activations apart into features that each stand for one recognizable idea.
- 3
Name the features
For each feature, look at the text that activates it most strongly and write down what it seems to track: a city, a tone of apology, code with a bug in it. Much of this labeling is now done by other models and spot-checked by people.
- 4
Trace how features connect
Follow which features switch on which others on the way to a particular answer. The result is a map, called an attribution graph, of the steps the model took. In March 2025 Anthropic used this to follow how one of its models did arithmetic, planned a rhyme and reasoned in two hops.
- 5
Test it by intervening
A map is only a guess until it is tested. Switch a feature off, turn it up, or copy it in from another run, and see whether the answer changes the way the map predicts. If it does not, the explanation was wrong.
- 6
Say what is still unexplained
Current methods explain only part of what a model does on any prompt, and can produce tidy stories that are not what really happened. Careful work reports how much is left over.
Terms in this guide are defined in the glossary.
- Safety. When an agent does something nobody intended, tests show that it happened. Internals show why.
- Trust. A model's own explanation of its reasoning can differ from what it computed. Reading the computation is the check.
- Consciousness research. It is the only way to look for the internal mechanisms that scientific theories of consciousness describe. See how we rate the evidence.
In January 2025, 29 researchers from 18 organizations published a list of what the field cannot yet do: check that an explanation is faithful, scale the methods to the largest models, and explain what the model's weights do rather than only its activations. Open Problems in Mechanistic Interpretability ↗