Reading models · Glossary
The key terms, in plain English.
28 terms
- Ablation
- Switching off a part of the model to see what stops working.
- Activation
- A number produced inside the model while it processes text. A model produces millions of them for each word.
- Activation patching
- Copying activations from one run into another to find which internal parts cause a behavior. Also called causal tracing.
- Attention head
- A component that moves information between positions in the text, for example from a name to a later pronoun.
- Attribution graph
- A map of which features influenced which others on the way to a particular answer.
- Circuit
- A set of features and connections that together carry out one identifiable job, such as adding two numbers or tracking who a pronoun refers to.
- Dictionary learning
- The general method behind sparse autoencoders: finding a set of building blocks that the activations can be written as combinations of.
- Faithfulness
- Whether an explanation matches what the model actually computed, as opposed to a plausible story.
- Feature
- A direction in the model's activations that stands for one recognizable idea, such as a city, a tone of voice or a kind of error.
- Global workspace
- A theory in which consciousness is information being broadcast to many specialist processes through a limited shared space.
- Indicator property
- A feature of how a system works that a scientific theory of consciousness treats as a sign of consciousness.
- Introspection (in models)
- Whether a model's statements about its own internal states are caused by those states, which can be tested by changing the states and seeing whether the reports change.
- Logit lens
- Reading the model's partial guess at the next word from an intermediate layer.
- Mechanistic interpretability
- Working out what a neural network is doing by studying its internal parts, the way you would reverse-engineer a machine, instead of only watching its outputs.
- Model organism
- A model deliberately built or trained to have a known property, so methods can be tested against a known answer.
- Model welfare
- The study of whether AI systems have interests that matter morally, and what to do given the uncertainty.
- Moral patienthood
- Being the kind of thing whose interests count morally for its own sake.
- Neuron
- One unit inside the network. Single neurons usually respond to many unrelated things, which makes them hard to read directly.
- Phenomenal consciousness
- There being something it is like to be the system: experience itself, as opposed to intelligence or behavior.
- Polysemantic
- Describes a neuron that fires for several unrelated ideas.
- Probe
- A simple classifier trained on activations to test whether some piece of information is present inside the model.
- Residual stream
- The shared channel that every layer of a transformer reads from and writes to.
- Self-report
- What a model says about itself when asked. Easy to collect, hard to interpret.
- Sentience
- The capacity for experiences that feel good or bad.
- Sparse autoencoder (SAE)
- A second, smaller network trained to untangle a model's activations into separate features, only a few of which are active at once.
- Steering
- Adding or amplifying a feature during a run to change behavior, which tests whether the feature really does what its label says.
- Superposition
- The model packing more ideas into its activations than it has neurons, by overlapping them. The main reason single neurons are hard to read.
- Transcoder
- A variant of a sparse autoencoder that learns to imitate one step of the model's computation using readable features.