“Interpretability” names three research programmes that share a word and almost nothing else.
One wants to reverse-engineer the computation into an algorithm.
One wants to know what information a representation carries.
One wants an explanation that satisfies somebody outside the lab.
They use different methods, and — the part that matters — they would accept different things as proof.
Three programmes, not one field Mechanistic interpretability tries to recover the algorithm.
The claim it aims at has the form: this set of attention heads and MLP neurons, wired this way, computes this function, and here is the intervention that confirms it.
The unit of output is a circuit.
Work in this programme is dominated by small models — two-layer attention-only transformers, GPT-2 small, toy networks trained on modular arithmetic — because a claim about an algorithm has to be checked exhaustively and that is only tractable when the model is small.
Representation analysis asks a weaker and much more answerable question: what information is present in this activation, and is it used?
Probing classifiers, representation-similarity measures and concept-vector methods live here.
The output is not an algorithm but a map — part-of-speech information is linearly recoverable by layer four, sentiment by layer eight, and here is the direction along which it lies.
Post-hoc explanation takes a model as fixed and produces, for a single prediction, an account of what drove it.
Saliency maps, feature attributions, counterfactual explanations and surrogate models are in this family.
Its consumers are frequently outside research entirely: a clinician, a loan applicant, a regulator.
Its success criterion is correspondingly different, and the gap between what a regulator means by an explanation and what a researcher means is where most of the confusion in this area comes from.
A fourth strand cuts across all three: training dynamics, the study of when in training a capability appears and what changes in the weights when it does.
Grokking and the phase-change literature belong here.
It is not a separate programme so much as a different axis — the same questions asked of a checkpoint sequence rather than a final model.
What each would count as success Programme Description Mechanistic A circuit whose description predicts the outcome of interventions it was not fitted to.
Ablate this head and this specific error appears; patch this activation from a different prompt and the output flips in the direction the story says it should.
Prediction under intervention is the bar.
Representational A property that is decodable and causally used.
Decodability alone is weak — a strong enough classifier finds structure in noise.
Success is decodability plus an intervention on the identified direction that changes behaviour in the predicted way.
Post-hoc An explanation that is faithful (it reflects what the model did, verified by removing what it highlighted) and useful (somebody makes a better decision with it).
Both halves are tested separately, and most published methods only ever get tested on the second.
Training dynamics A progress measure that moves before the loss does — a quantity computed from the weights or activations that anticipates a capability appearing, rather than describing it afterwards.
The question that separates them Ask of any interpretability claim: what result would show this is wrong?
The three programmes answer very differently, and the answer tells you which one you are reading.
A mechanistic claim is falsified by an intervention.
If head 9.6 is said to move information about the subject token to the final position, then zeroing its output should break exactly the behaviours that depend on that information and leave the others alone.
If it breaks everything, the claim was too specific; if it breaks nothing, the head was not necessary.
A representational claim is falsified by a control.
If a probe reads syntactic depth out of layer six at high accu