Sparsity loss (L1 penalty on feature activations)
Sparse Autoencoders Find Highly Interpretable Features in Language Models — introduced
The alpha*||c||_1 term in the autoencoder's training objective (Equation 4), penalizing the L1 norm of the hidden-layer coefficients c to encourage sparse feature activations. The sparsity coefficient alpha is the paper's key hyperparameter, swept throughout to trade off sparsity against reconstruction accuracy.
Nothing about reconstructing an activation well forces an autoencoder to use only a few of its directions at once, even when its hidden layer is far wider than the input feeding it. Sparsity loss is the term added specifically to demand that restraint, penalizing how many directions switch on rather than just how faithfully they reconstruct.
What comes next picks apart the term's exact form, why penalizing outright magnitude rather than mere smallness matters, and how turning its strength up or down trades reconstruction accuracy against the causal precision that separates a sparse dictionary from an ordinary one.
Why the loss needs a sparsity term at all
The whole premise of using sparse autoencoders to resolve Superposition rests on activations being sparse linear combinations of a larger set of true features; sparsity loss is the term in training that actually enforces this, rather than leaving it as a hoped-for side effect of a wide hidden layer. Without it, an autoencoder with $d_{hid} > d_{in}$ has no pressure to use only a few dictionary features per input, and the resulting decomposition would not resemble the sparse coding problem of Olshausen & Field (1997) that the method is built to solve. The paper cites both empirical results (Sharkey et al., 2023) and theory (Wright & Ma, 2022) for the claim that reconstruction under an L1 penalty specifically -- not just any regularizer -- can recover the actual ground-truth directions that generated the data (sparse-autoencoders, §"2 TAKING FEATURES OUT OF SUPERPOSITION WITH SPARSE DICTIONARY LEARNING", p. 3).
What turning sparsity up or down buys and costs
Sparsity loss is the dial the paper turns to demonstrate that sparsity itself, not just dictionary learning generally, is responsible for its results. Training a dictionary with no sparsity penalty at all (alpha=0) reconstructs activations well but loses the causal-localization advantage over PCA entirely: on the IOI task, its features need as many patches as PCA components to reach a given KL divergence, showing that sparsity rather than reconstruction fidelity is what makes a patched feature set small and precise (sparse-autoencoders, §"4.2 PRECISE LOCALISATION OF IOI DICTIONARY FEATURES", p. 6). Turning alpha up instead raises the floor: dictionaries trained with a larger sparsity coefficient reach lower reconstruction accuracy, which shows up as a higher minimum-achievable KL divergence in the same patching experiments no matter how many features are patched. Sweeping alpha continuously across this range, with no distinguished 'knee,' is the direct empirical basis for the Sparsity-reconstruction tradeoff (no single correct decomposition) (sparse-autoencoders, §"B SPARSE AUTOENCODER TRAINING AND HYPERPARAMETER SELECTION", p. 12).
The term itself, and what it holds fixed
Sparsity loss is the second term of the training objective $$L(x) = \|x - \hat{x}\|_2^2 + \alpha\|c\|_1$$ where $c = \mathrm{ReLU}(Mx+b)$ is the hidden code and $\alpha$ is a hyperparameter controlling how strongly sparsity is weighted against reconstruction accuracy (sparse-autoencoders, §"2 TAKING FEATURES OUT OF SUPERPOSITION WITH SPARSE DICTIONARY LEARNING", p. 3). Penalizing the L1, rather than L2, norm of $c$ specifically encourages exact zeros in the code rather than merely small values, which is what produces a sparse rather than merely small activation pattern. Because $c$ and $M$ can trade off against each other -- doubling a feature's row norm in $M$ lets the optimizer halve its coefficient in $c$ without truly sparsifying anything -- the loss only works as intended once each row of $M$ is held to unit norm, so dictionary features cannot be rescaled to game the penalty (sparse-autoencoders, §"2 TAKING FEATURES OUT OF SUPERPOSITION WITH SPARSE DICTIONARY LEARNING", p. 3). Working values of alpha in the paper's main experiments are around 8.6e-4 for the residual stream and 3.2e-4 for MLP sublayers, different enough between the two settings that alpha cannot be tuned in isolation from where in the model the dictionary is trained (sparse-autoencoders, §"E NUMBER OF ACTIVE FEATURES", p. 17).