Superposition and the case against the neuron basis

Members — a guided walk

1. Polysemanticity

   Polysemanticity is the paper's opening problem: individual neurons activate in multiple, semantically unrelated contexts, blocking any concise human-understandable account of what a network's units are doing. Cunningham et al. name it in their abstract as the central roadblock their sparse-autoencoder method is built to remove, and every technique in this theme traces back to explaining or curing it. The paper's proposed explanation for why polysemanticity happens at all is superposition.

2. Superposition

   Superposition, hypothesized by Elhage et al. (2022), is a network representing more features than it has dimensions by packing them into an overcomplete set of non-orthogonal directions, relying on activation sparsity to keep interference between them manageable. If superposition is the reason neurons are polysemantic, then recovering the true, overcomplete set of directions -- via sparse dictionary learning -- should recover monosemantic features instead of polysemantic neurons. The paper's main testing ground for this claim is the residual stream.

3. Residual stream

   The residual stream, the transformer's running sum of layer outputs, is this paper's primary object of study: sparse autoencoders are trained mainly on residual-stream activations, rather than on individual neurons or MLP units. Studying it raises an immediate question the paper has to address before its results mean anything: does this space even have a meaningful default coordinate system to compare its learned features against?

4. Privileged basis

   A privileged basis is a coordinate system distinguished as unusually meaningful for a given vector space, the way the standard basis is meaningful immediately after an elementwise nonlinearity. Cunningham et al. note that the residual stream is not usually expected to have one; nothing about how it accumulates layer outputs picks out any particular set of coordinates as special. That expectation turns out to be only partially true, because of outlier dimensions.

5. Outlier dimensions

   Outlier dimensions are specific residual-stream coordinates that transformers write disproportionately large, information-bearing values to, a pattern Dettmers et al. (2022) attribute to the Adam optimizer storing gradients with finite precision in the residual basis. The paper finds its top-activating default-basis dimensions for both the most positive and most negative apostrophe-related directions were outlier dimensions rather than typical coordinates -- a partial exception to the 'no privileged basis' expectation that the paper has to control for when it builds its baseline. That baseline is the default basis itself.

6. Default (neuron/residual-stream) basis baseline

   The default basis baseline treats individual neurons, or for the residual stream individual coordinates, as the units worth analyzing directly, exactly the naive hypothesis that features are neurons taken literally. Scored for autointerpretability against the paper's learned dictionary features, this baseline is shown, through the apostrophe-dimension case study, to be polysemantic exactly where the corresponding dictionary feature is monosemantic. That contrast is the paper's cleanest empirical demonstration of what superposition predicts and sparse coding is meant to fix.

7. Monosemanticity

   Monosemanticity, activating for one coherent, human-interpretable concept, is the property the default basis lacks and the learned dictionary feature has: the same apostrophe-related direction that is polysemantic as a raw residual-stream coordinate becomes monosemantic once expressed as a dictionary feature. The paper's case studies extend this same contrast beyond apostrophes, to a closing-parenthesis feature and others, arguing that monosemanticity is a general property of the learned decomposition rather than one lucky example. Taken together, the theoretical account and this empirical demonstration argue for a broader methodological conclusion about how interpretability research should proceed.

8. Mechanistic interpretability

   Mechanistic interpretability is that broader research agenda: reverse-engineering a network's internal computations into human-understandable parts, in service of verifying, editing, and mitigating risks such as deceptive behavior. Cunningham et al. frame their sparse-autoencoder method as a foundation for future work in this agenda precisely because it validates the theme's premise -- that neurons and coordinate bases are not privileged units of analysis, and meaningful units have to be discovered rather than assumed. The paper's remaining sections extend that discovered basis into scoring, baselines, and causal tests, but the case against the neuron basis is made in full by this point.

Superthemes

• Decomposing the model into features

Edges explored within this theme

• Superposition — manifests as → Polysemanticity
• Polysemanticity — blocks → Mechanistic interpretability
• Superposition — motivates → Sparse autoencoders (SAEs)
• Monosemanticity — benchmarks → Dictionary feature
• Privileged basis — is absent from → Residual stream
• Outlier dimensions — partially restores → Privileged basis
• Outlier dimensions — fails to elevate → Default (neuron/residual-stream) basis baseline
• SAE training on the MLP sublayer — confronts → Privileged basis

Super edges touching this theme

• Superposition and the case against the neuron basis — dictates the sparsity mechanics of → Unmixing activations with a sparsity penalty
• Steering doubles as interpretability — surfaces artifacts explained by → Superposition and the case against the neuron basis

Sources

Every passage behind this theme’s member concepts.