Activation patching
Sparse Autoencoders Find Highly Interpretable Features in Language Models — inherited
A causal mediation analysis technique (Vig et al., 2020) in which a model's internal activations are edited toward values observed on a counterfactual input and the resulting change in output is measured. This paper adapts it to dictionary features -- patching feature-encoded activation differences into the residual stream and measuring KL divergence from a target output -- to test how precisely SAE features localize the Indirect Object Identification behavior, compared to patching PCA components.
A direction inside a network can look interpretable and still have nothing to do with what the model actually does — the only way to tell the difference is to intervene on it and watch whether the output moves. Activation patching is the technique built for that test: swap in activations from a counterfactual input and measure how far the model's behavior shifts toward it, a much harder bar to clear than simply looking suggestive.
The page traces the technique from its original use investigating bias in language models, through its adoption for reverse-engineering circuits component by component, to a later adaptation that patches individual learned features instead — and the precision that buys over an alternative decomposition.
Anchor of the causal-proof-by-patching theme
Activation patching is the causal-mediation technique that makes the causal-proof-by-patching theme falsifiable rather than merely suggestive: instead of asking whether a direction looks interpretable, it asks whether forcing that direction to a counterfactual value actually moves the model's output, the harder causal question the paper needs answered before claiming dictionary features are more than a labeling exercise. The technique originates with Vig et al. (2020), who introduced it as a form of causal mediation analysis for investigating gender bias in language models: run the model on a counterfactual input, cache its internal activations, transplant those cached activations into a run on the base input at a chosen intervention point, and measure how far the resulting output shifts toward the counterfactual's own output. The size of that shift is read as a measure of how much the intervened-on component causally mediates the behavior, as opposed to merely correlating with it.
From attention heads to a scalable circuit-discovery toolkit
In the years after Vig et al. (2020), activation patching (and close relatives such as path patching) became a standard tool for reverse-engineering transformer circuits component by component -- for example, Wang et al. (2022)'s hand-built circuit for Indirect Object Identification was itself established largely through patching individual attention heads, and Conmy et al. (2023)'s ACDC algorithm was built to automate exactly that kind of head-by-head patching search. This paper inherits the technique from that lineage but changes what gets patched: instead of transplanting whole attention-head or layer activations, Section 4.1 transplants the differences in dictionary feature activations between a counterfactual and a base sentence, via Dictionary-feature patching procedure -- patching individual learned directions rather than architectural components (sparse-autoencoders, §"4.1 Adapting activation patching to dictionary features", p. 6).
The metric: how close does patching get you to the target?
The intuition carries over unchanged from the original technique: patch, then check how far the output moved toward the counterfactual. What differs is that this paper needs a single scalar summarizing that movement across an entire output distribution, so it measures the Kullback-Leibler divergence between the patched model's output logits $z$ and the target (counterfactual) model's output logits $y$, $D_{\mathrm{KL}}(z \parallel y)$, on the IOI behaviour introduced in Section 4 (sparse-autoencoders, §"4.1 Adapting activation patching to dictionary features", p. 6). A smaller divergence means the patched features accounted for more of what separates the base sentence's prediction from the counterfactual's.
What the adaptation bought: precision over PCA
The payoff of running Vig et al.'s technique on dictionary features rather than raw activations or components is a precision result: patching the resulting feature subsets reaches a given KL divergence from the target using fewer patched features and a smaller mean edit magnitude than patching an equal number of principal components (sparse-autoencoders, §"4.2 Precise localisation of IOI dictionary features", p. 6). That comparison is only meaningful because both decompositions are patched under the same causal-mediation logic Vig et al. defined -- activation patching supplies the common yardstick that lets a feature-level intervention and a component-level intervention be compared on equal footing.