Almost every production LLM bug is diagnosed from inputs, outputs and logs.
Interpretability earns its keep in a narrow set of cases, and knowing which ones is worth more than knowing the techniques.
This page is that list, with the cases where it is the wrong tool named first.
The honest starting position If your model is producing bad output, the ordered list of things to do begins: read the actual prompt that was sent, including everything your framework appended; check the retrieved context for the failing case; check whether the failure reproduces at temperature 0; and check whether it started at a specific deploy.
Those four steps resolve the overwhelming majority of real incidents.
The internals of the model are the last place to look, not the first, because they are the part that did not change — and the systematic version of that checklist is a better use of an afternoon than any technique on this page.
What follows is for the residue: problems that survive that pass, and where a signal from inside the model is genuinely the cheapest way forward.
The gate: do you have the weights?
Almost everything in this cluster requires local weights.
A hosted API returns tokens and, sometimes, log-probabilities.
It does not return activations, and it will not.
So the practical question is not “is this technique good” but “is this problem worth self-hosting an open model to investigate”.
Usually it is not.
Occasionally — a classifier at the centre of a product, a safety-relevant behaviour you must characterise, a model you fine-tuned and now cannot explain — it is, and the self-hosting trade-off changes when internals access is part of what you are buying.
One technique survives the gate: log-probability analysis works through any API that exposes it, and it is the cheapest useful signal about a model’s internal state that exists.
What pays for itself Log-probabilities, on any model The token distribution tells you whether the model was confident or picking between near-ties.
A wrong answer at probability 0.95 and a wrong answer at 0.31 are different bugs: the first is a representation problem, the second is often a prompt or retrieval problem where the right answer was in contention and lost.
For any constrained output — classification, routing, extraction from a fixed set — reading the distribution over the permitted tokens gives you a usable confidence signal and an abstention threshold.
That is a production feature, not an analysis: abstaining below a threshold converts a class of silent wrong answers into a handled case.
See log-probabilities for the mechanics.
Embedding-space diagnosis for retrieval When a retrieval system returns the wrong documents, the internals in question are the embeddings, and they are fully available to you.
Check the similarity between the query embedding and the embedding of the document that should have won, then check the documents that actually won.
This distinguishes three failures that look identical from the outside: the document was not in the index at all, it was there but scored below the cut, or it scored well and was displaced by near-duplicates.
Each has a different fix, and no amount of prompt iteration separates them.
A probe as a runtime classifier If you self-host, a linear probe on a mid-layer hidden state is a genuinely practical component.
It costs a dot product on a tensor you already computed, it runs on the prompt before generation, and it can flag categories — this request looks like the class we route elsewhere — more cheaply than a second model call.
The evaluation obligations from the probing page apply in full: a control task, a held-out split at document level, and a measured false positive rate on real traffic.
A probe deployed on the strength of its training accuracy is a liability.
Attention patterns for long-context routing bugs Narrow but real.
If a model ignores instructions placed in the middle of a long document, the attention pattern shows the positional structur