Reading models · Glossary

The key terms, in plain English.

28 terms
Ablation
Switching off a part of the model to see what stops working.
Activation
A number produced inside the model while it processes text. A model produces millions of them for each word.
Activation patching
Copying activations from one run into another to find which internal parts cause a behavior. Also called causal tracing.
Attention head
A component that moves information between positions in the text, for example from a name to a later pronoun.
Attribution graph
A map of which features influenced which others on the way to a particular answer.
Circuit
A set of features and connections that together carry out one identifiable job, such as adding two numbers or tracking who a pronoun refers to.
Dictionary learning
The general method behind sparse autoencoders: finding a set of building blocks that the activations can be written as combinations of.
Faithfulness
Whether an explanation matches what the model actually computed, as opposed to a plausible story.
Feature
A direction in the model's activations that stands for one recognizable idea, such as a city, a tone of voice or a kind of error.
Global workspace
A theory in which consciousness is information being broadcast to many specialist processes through a limited shared space.
Indicator property
A feature of how a system works that a scientific theory of consciousness treats as a sign of consciousness.
Introspection (in models)
Whether a model's statements about its own internal states are caused by those states, which can be tested by changing the states and seeing whether the reports change.
Logit lens
Reading the model's partial guess at the next word from an intermediate layer.
Mechanistic interpretability
Working out what a neural network is doing by studying its internal parts, the way you would reverse-engineer a machine, instead of only watching its outputs.
Model organism
A model deliberately built or trained to have a known property, so methods can be tested against a known answer.
Model welfare
The study of whether AI systems have interests that matter morally, and what to do given the uncertainty.
Moral patienthood
Being the kind of thing whose interests count morally for its own sake.
Neuron
One unit inside the network. Single neurons usually respond to many unrelated things, which makes them hard to read directly.
Phenomenal consciousness
There being something it is like to be the system: experience itself, as opposed to intelligence or behavior.
Polysemantic
Describes a neuron that fires for several unrelated ideas.
Probe
A simple classifier trained on activations to test whether some piece of information is present inside the model.
Residual stream
The shared channel that every layer of a transformer reads from and writes to.
Self-report
What a model says about itself when asked. Easy to collect, hard to interpret.
Sentience
The capacity for experiences that feel good or bad.
Sparse autoencoder (SAE)
A second, smaller network trained to untangle a model's activations into separate features, only a few of which are active at once.
Steering
Adding or amplifying a feature during a run to change behavior, which tests whether the feature really does what its label says.
Superposition
The model packing more ideas into its activations than it has neurons, by overlapping them. The main reason single neurons are hard to read.
Transcoder
A variant of a sparse autoencoder that learns to imitate one step of the model's computation using readable features.