Dictionary-feature patching procedure
Sparse Autoencoders Find Highly Interpretable Features in Language Models — introduced
The paper's specific patching procedure for dictionary features: run the model on a counterfactual sentence, cache the encoded feature activations, then re-run on the base sentence and add the encoded activation differences for a chosen feature subset F into the residual stream, measuring KL divergence between patched and target output logits. Found to reach a given KL divergence with fewer patched features and smaller edit magnitude than patching an equivalent number of PCA components.
Deciding to test causal importance by swapping in activations from a counterfactual input is one thing; deciding which of several thousand learned features to touch, and by how much, is another problem entirely. Dictionary-feature patching answers it: run the model twice, once on the real input and once on a counterfactual variant, then nudge only a chosen subset of features toward their counterfactual values and see how close the output gets.
This page walks through the patching procedure, how the subset of features to touch gets chosen rather than picked by hand, and the headline result — fewer patched features and smaller edits than an alternative decomposition needs to reach the same target — along with its caveats.
The paper's own instrument for the causal-proof theme
Dictionary-feature patching is Section 4.1's answer to a question the causal-proof-by-patching theme needs answered concretely: given a learned feature dictionary, exactly how do you intervene on it to test causal importance? It is the specific, executable procedure underneath the more general idea of Activation patching, built to run Vig et al. (2020)'s causal-mediation logic on individual dictionary features rather than on attention heads or layers.
The procedure: run twice, cache once, patch the difference
Concretely, fix a layer to intervene on. First run the model on a counterfactual target sentence -- a variant of the base IOI sentence with the indirect object swapped, e.g. "Bob" replaced by "Vanessa" -- and save both its output logits $y$ and its per-token encoded feature activations $\bar{c}_{i,1},\dots,\bar{c}_{i,k}$. Then run the model on the base sentence up through the intervention layer, compute its own encoded features $c_{i,1},\dots,c_{i,k}$, and at each token position $i$ replace the residual-stream vector $x_i$ with $x'_i = x_i + \sum_{j \in F} (\bar{c}_{i,j} - c_{i,j}) f_j$, where $F$ is a chosen subset of features and $f_j$ is dictionary feature $j$'s learned direction (sparse-autoencoders, §"4.1 Adapting activation patching to dictionary features", p. 6). Finally the patched model's output logits $z$ are compared to $y$ via $D_{\mathrm{KL}}(z \parallel y)$.
Choosing which features to patch: borrowing ACDC's ordering
The subset $F$ is not chosen by hand. The paper reuses ACDC (Conmy et al., 2023)'s Algorithm 4.1, treating every dictionary feature as an independent node in a flat computational graph: ablate each feature, measure the resulting change in $D_{\mathrm{KL}}$ averaged over a 50-example IOI test set, and order features by how much patching each one reduces the divergence, so that $F$ for a given budget $k$ is simply the first $k$ features under that ordering (sparse-autoencoders, §"4.1 Adapting activation patching to dictionary features", p. 6). The same ordering procedure is applied separately to the PCA baseline so the two decompositions are ranked and compared like for like.
The headline result, and its asterisks
Run on layer 11 of Pythia-410M's residual stream, with dictionaries trained on the first 10,000 elements of the Pile (about 7 million activations) (sparse-autoencoders, §"4 Identifying causally-important dictionary features for indirect object identification", p. 5), dictionary-feature patching reaches a given KL divergence from the target using fewer patched features, and a smaller mean edit magnitude, than an equivalent number of patched PCA components -- an advantage that disappears for a non-sparse ($\alpha=0$) dictionary, showing the effect depends on sparsity rather than merely on having a learned dictionary at all (sparse-autoencoders, §"4.2 Precise localisation of IOI dictionary features", p. 6). The comparison also has a floor: dictionaries trained with a larger sparsity coefficient $\alpha$ have lower reconstruction accuracy, which shows up as a larger minimum achievable KL divergence that no amount of patching can close (sparse-autoencoders, §"4.2 Precise localisation of IOI dictionary features", p. 6).