Automatic feature circuit detection
Sparse Autoencoders Find Highly Interpretable Features in Language Models — introduced
A method for tracing causal dependencies between dictionary features in adjacent layers: given a target feature, sample contexts where it strongly activates, then ablate each previous-layer feature and rank features by how much their ablation decreases the target's activation, optionally recursing to build a multi-layer causal tree. Demonstrated on a layer-5 feature that predicts closing parentheses, yielding an interpretable circuit of upstream features.
Showing that a single feature is interpretable is one thing; showing how a whole chain of features causes each other to fire is a different and harder claim, closer to what an actual map of a model's internal computation would need to look like. This method builds that chain by brute force: ablate every candidate feature in the previous layer one at a time, measure how much each removal weakens the target feature's own activation, and rank the results.
The page covers the ablate-and-rank procedure itself, the worked example it's demonstrated on, the cheaper weights-only method that was tried first and abandoned, and the larger ambition this kind of causal tracing is offered as one small step toward.
Where three of the paper's themes meet
Feature-circuit-detection sits at an unusual crossing point: it belongs to causal-proof-by-patching as an ablation-based causal method, to single-features-under-the-microscope as the technique demonstrated on the closing-parenthesis feature, and to enumerating-features-for-safety as the paper's only worked example of what a safety-relevant map of causal dependencies between features could actually look like in practice.
The procedure: ablate backward from a target feature
Given a target dictionary feature, the method first finds its maximum activation $M$ across the dataset and samples 20 contexts in which it activates within $[M/2, M]$. Then, for every feature in the previous layer, it reruns the model with that feature ablated and records how much its removal decreased the target feature's own activation; sorting previous-layer features by that decrease produces a ranked list of upstream causes, and the procedure can recurse into the previous layer's own high-impact features to build a multi-layer causal tree (sparse-autoencoders, §"5.3 Intermediate features: dictionary features allow automatic circuit detection", p. 7-8).
The demonstration: layer 5's closing-parenthesis feature
The method is demonstrated on feature 52027 in layer 5, unembedded to show its top tokens are all closing-parenthesis variants -- an unusually solid anchor, since layer 5 is the model's final layer and therefore writes directions the unembedding matrix reads out directly as tokens (sparse-autoencoders, §"5.3 Intermediate features: dictionary features allow automatic circuit detection", p. 8). Applying the procedure backward from that anchor produces Figure 5, a causal tree of layer-4 features for dates, acronyms, and other parenthesis-preceding phrases, with edge thickness in the figure encoding the strength of each causal effect as measured by ablation.
What it replaced, and what it points toward
The ablation-based approach was not the first thing tried: a cheaper, purely weights-based method -- multiplying a feature through the MLP's weights and checking cosine similarity with the next layer -- was attempted first and abandoned after turning up no meaningful connections (sparse-autoencoders, §"D.4 Failed interpretability methods", p. 17). The paper explicitly frames feature-circuit-detection's payoff as a first step toward its most ambitious goal: tracing causal dependencies between features is offered as "a step towards the eventual goal of building an end-to-end understanding of how a model computes its outputs" (sparse-autoencoders, §"6.2 Limitations and future work", p. 9), which the very next paragraph names as Enumerative safety (sparse-autoencoders, §"6.3 Conclusion", p. 9).