Unmixing activations with a sparsity penalty

Members — a guided walk

1. Sparse dictionary learning / sparse coding

   Sparse dictionary learning is the decades-old mathematical framework this theme scales up: Olshausen and Field (1997) and Lee et al. (2006) showed that data could be expressed as sparse linear combinations of an overcomplete set of learned basis vectors, originally to model sparse coding in the visual cortex. Cunningham et al. (2023) treat a language model's internal activations as exactly this kind of data, hypothesizing that a network's true features sit as an overcomplete, non-orthogonal set of directions that ordinary neurons or coordinates cannot represent one-to-one. Sparse autoencoders are how that decades-old framework gets implemented as a trainable neural network.

2. Sparse autoencoders (SAEs)

   A sparse autoencoder is a single-hidden-layer, ReLU network trained directly on a language model's cached activations, with tied encoder/decoder weights and an overcomplete hidden layer. Cunningham et al. train these autoencoders on Pythia-70M and Pythia-410M residual streams to demonstrate, alongside Anthropic's concurrent 'Towards Monosemanticity' work, that this simple architecture recovers features more interpretable, more monosemantic, and more causally precise than PCA, ICA, or the neuron basis. Every subsequent step in this walk is a design choice or training-time fact about that one small network.

3. Dictionary feature

   Each row of the sparse autoencoder's weight matrix M is a dictionary feature, a direction in activation space that the network uses, in sparse linear combination with others, to reconstruct its input. The paper calls this feature its central unit of analysis: Sections 3 through 5 examine dictionary features for interpretability, causal importance, and monosemanticity respectively. What makes a set of dictionary features good or bad is entirely determined by the training objective that produces them.

4. Reconstruction loss

   That objective's first term is reconstruction loss, the squared error ||x - x-hat||^2 between an input activation and the autoencoder's reconstruction of it, Equation 4 in the paper. Cunningham et al. are explicit that this loss never reaches zero: the dictionary always fails to capture some information in a layer's activations, a gap the reconstruction-substitution perplexity metric will later quantify directly. Reconstruction loss alone, though, would just reproduce ordinary autoencoding; the second term in the objective is what makes the decomposition a dictionary.

5. Sparsity loss (L1 penalty on feature activations)

   Sparsity loss is that second term, alpha times the L1 norm of the hidden-layer coefficients c, added to reconstruction loss to encourage each input to be explained by only a few active dictionary features at a time. Alpha is the paper's single most consequential hyperparameter: at alpha=.00086 with a two-fold expansion factor, the residual-stream dictionaries used for the main interpretability comparison in Figure 2 are trained for 10 epochs, and every sweep in the paper varies alpha to trace how much sparsity a given amount of reconstruction accuracy costs. How much room that sparsity penalty has to work with is set by a separate hyperparameter, the dictionary's expansion factor.

6. Dictionary expansion factor (R)

   The expansion factor R fixes the ratio between the autoencoder's hidden dimension and the model's activation dimension, dhid = R x din, controlling how overcomplete the learned dictionary is. Sweeping R shows interpretability gains hold even for undercomplete dictionaries with R<1, while larger dictionaries, unsurprisingly, explain more of a layer's overall activation variance. R sets how many dictionary features exist to compete for sparsity budget; the weight matrix mapping between them and the model's own activations is itself a further design choice, tied encoder/decoder weights.

7. Tied encoder/decoder weights

   Tied encoder/decoder weights are the choice of using a single weight matrix, transposed between the encoder and decoder halves of the network, rather than two independent ones. Cunningham et al. adopt this because it encodes the expectation that a feature's detecting and defining directions coincide, halves memory cost, and removes an otherwise-arbitrary ambiguity between encoder and decoder directions, but they find it hurts performance enough on MLP-sublayer activations to motivate an untied variant there. That MLP setting is also where a second cost of undertrained dictionaries shows up: dead features.

8. Dead features

   Dead features are dictionary rows that never activate across the training corpus, wasted capacity in an already-overcomplete dictionary. Residual-stream dictionaries stay largely alive, overcomplete up to roughly a 4x expansion factor before features start dropping out, but MLP-sublayer dictionaries suffer many dead features even with no sparsity penalty at all (alpha=0), in some cases leaving less than half a nominally twice-as-large dictionary usably active. That imbalance between the residual stream and the MLP sublayer is worth following into the MLP setting directly.

9. SAE training on the MLP sublayer

   SAE training on the MLP sublayer extends the same architecture and objective to a different set of activations, with mixed results: many MLP features are more interpretable than individual neurons, but the training pipeline does not yet robustly learn overcomplete bases in the intermediate or later MLP layers, making untied weights and reinitializing dead features necessary workarounds. This mixed outcome on the MLP is a smaller-scale version of the residual stream's own unresolved gap, the amount of information the dictionary still fails to reconstruct.

10. Reconstruction-substitution perplexity metric

    The reconstruction-substitution perplexity metric puts a number on that gap: replace a layer's activations with the SAE's reconstruction of them and measure how much the model's perplexity on the Pile rises. Substituting Pythia-70M's layer-2 residual stream with its reconstruction raises Pile perplexity from 25 to 40, direct evidence that reconstruction loss, even after training, is far from zero. That residual gap turns out not to be a bug to be swept away by tuning alpha harder, but a structural feature of the whole tradeoff the objective is built on.

11. Sparsity-reconstruction tradeoff (no single correct decomposition)

    The sparsity-reconstruction tradeoff is what sweeping alpha actually reveals: a smooth curve between the number of active features and unexplained reconstruction variance, with no knee or bump marking a single best operating point. Cunningham et al. read this as weak evidence against there being one uniquely correct sparse decomposition of a given activation space, only a family of decompositions at different points along the curve. Every other choice this walk has covered, R, tied weights, alpha itself, is a choice of where on that curve to sit, not a search for a single ground truth waiting to be found.

Superthemes

• Decomposing the model into features

Edges explored within this theme

• Sparsity loss (L1 penalty on feature activations) — trades off against → Reconstruction loss
• Sparsity loss (L1 penalty on feature activations) — forces unit norm on → Dictionary feature
• Dictionary expansion factor (R) — governs emergence of → Dead features
• Tied encoder/decoder weights — is the default configuration of → Sparse autoencoders (SAEs)
• Dictionary expansion factor (R) — sets the operating point on → Sparsity-reconstruction tradeoff (no single correct decomposition)
• Sparsity-reconstruction tradeoff (no single correct decomposition) — rules out a single canonical dictionary for → Sparse autoencoders (SAEs)
• Reconstruction-substitution perplexity metric — independently corroborates → Reconstruction loss
• SAE training on the MLP sublayer — stress tests → Sparse autoencoders (SAEs)
• Dead features — hollows out the count of → Dictionary feature
• Tied encoder/decoder weights — gives way to an untied variant in → SAE training on the MLP sublayer

Super edges touching this theme

• Superposition and the case against the neuron basis — dictates the sparsity mechanics of → Unmixing activations with a sparsity penalty
• The dictionary-learning lineage — leaves the architecture for → Unmixing activations with a sparsity penalty
• Unmixing activations with a sparsity penalty — hands off only what fires to → Interpretability itself becomes a benchmark

Sources

Every passage behind this theme’s member concepts.