Features proved by intervention

Members — a guided walk

1. Activation patching

   Activation patching is the causal mediation technique, borrowed from Vig et al. (2020), that this theme's entire argument rests on: edit a model's internal activations toward values observed on a counterfactual input, then measure the resulting change in output. Cunningham et al. adapt it specifically to dictionary features, patching feature-encoded activation differences into the residual stream rather than raw neuron or coordinate values, to test how precisely those features localize a real model behavior. That behavior is a previously-studied task chosen for being simple enough to check the method against a known answer.

2. Indirect Object Identification (IOI) task

   The Indirect Object Identification task has the model complete sentences like 'Then, Alice and Bob went to the store. Alice gave a snack to ___' by correctly naming the indirect object, a behavior Wang et al. (2022) had already reverse-engineered into a known circuit in GPT-2-small. Cunningham et al. reuse it as a case study specifically because their dictionaries were trained task-agnostically, on generic activations with no emphasis on IOI at all, so any precision the features show here is not an artifact of training on the task. Turning that precision into a number requires a specific patching procedure built for dictionary features.

3. Dictionary-feature patching procedure

   The dictionary-feature patching procedure runs the model on a counterfactual sentence, caches the encoded activations of a chosen feature subset, then reruns on the base sentence and adds the encoded activation differences for that subset back into the residual stream, measuring the KL divergence between the patched output and the target output. On layer 11 of Pythia-410M, this procedure reaches a given KL divergence with fewer patched features and smaller edit magnitude than patching an equivalent number of PCA components, and a non-sparse (alpha=0) dictionary shows no such advantage. Which features belong in that patched subset in the first place is decided by a separate algorithm.

4. Automated Circuit Discovery (ACDC)

   Automated Circuit Discovery supplies that algorithm: Cunningham et al. reuse its ordering procedure, Algorithm 4.1 from Conmy et al. (2023), treating dictionary features as a flat computational graph and ranking each by its individual contribution to reducing KL divergence, averaged over a test set of 50 IOI data points. The resulting ordering is what defines the feature subsets patched in the dictionary-feature-patching experiments, applied separately to each decomposition being compared. That same causal logic, applied at the level of a single feature rather than a whole subset, is what the paper's finer-grained ablation technique isolates next.

5. Less-than-rank-one ablation

   Less-than-rank-one ablation lowers a residual-stream activation's component along a feature's direction only as far as that feature's own activation threshold, rather than subtracting the full magnitude of the direction. Used in the apostrophe-feature case study, this precision is what lets the paper isolate a single feature's causal effect on next-token logits cleanly, showing that removing the apostrophe feature specifically suppresses the logit for the following 's' token. The same ablate-and-measure logic, applied not to logits but to another feature's activation, is how the paper traces causal dependencies between features across layers.

6. Automatic feature circuit detection

   Automatic feature circuit detection extends activation-patching-style causal testing across layers: given a target feature, sample contexts where it strongly activates, then ablate each previous-layer feature in turn and rank features by how much their ablation lowers the target's activation, optionally recursing to build a multi-layer causal tree. This is the method that turns individual causally-validated features into circuits, an interpretable chain of upstream causes rather than an isolated case study. Its central demonstration is a single layer-5 feature chosen because its role in the network is unusually easy to verify independently.

7. Closing-parenthesis dictionary feature

   The closing-parenthesis dictionary feature is that demonstration: a layer-5 feature whose unembedding directly predicts closing-parenthesis tokens, letting the paper check its circuit-detection method against a feature whose output-side meaning is already unambiguous. Tracing upstream causes turns up features detecting dates, acronyms, and other parenthesis-preceding phrases, exactly the contexts that intuitively precede a closing parenthesis, giving the ablation-based method a legible, checkable success. Not every attempt to trace causal structure between features succeeded this cleanly, and the paper reports one that did not.

8. Weight-based feature connection attempt (failed method)

   The weight-based feature connection attempt was a different, faster-to-compute alternative: multiply a layer-4 feature through the MLP weights and check cosine similarity with layer-5 features, hoping weights alone would reveal the same connections ablation found. It found no meaningful connections, likely because the resulting vector went out-of-distribution, and Cunningham et al. report this failure alongside the ablation-based method's success rather than omitting it. The contrast closes the theme's argument: an interpretable-looking feature, or a plausible-looking connection between features, earns trust only once an actual intervention -- ablation, patching -- has been shown to change model behavior in the predicted way, not merely once its weights look suggestive.

Superthemes

• Decomposing the model into features

Edges explored within this theme

• Activation patching — is adapted into → Dictionary-feature patching procedure
• Dictionary-feature patching procedure — is tested against → Indirect Object Identification (IOI) task
• Less-than-rank-one ablation — edits with finer granularity than → Dictionary-feature patching procedure
• Automated Circuit Discovery (ACDC) — lends its ranking logic to → Automatic feature circuit detection
• Weight-based feature connection attempt (failed method) — is the failed precursor to → Automatic feature circuit detection

Super edges touching this theme

• Interpretability itself becomes a benchmark — passes the burden of proof to → Features proved by intervention
• Features proved by intervention — runs its proof on separate features than → Single features under the microscope
• Features proved by intervention — installs one expensive link in → Feature enumeration as a safety audit

Sources

Every passage behind this theme’s member concepts.