Dead features
Sparse Autoencoders Find Highly Interpretable Features in Language Models — introduced
Dictionary features that never activate across the training corpus, wasting dictionary capacity. The paper finds residual-stream dictionaries stay overcomplete (few dead features) up to roughly a 4x expansion factor, while MLP-sublayer dictionaries suffer many dead features even without a sparsity penalty, in some cases leaving less than half the dictionary usably 'alive'.
Building a dictionary wider than the layer it's meant to explain sounds like a free way to capture more of a model's hidden structure, but width bought this way isn't guaranteed to get used — some of those extra directions can simply sit there, never once switching on across an entire training run.
The entry turns next to how bad that waste becomes and where, how a feature earns the label 'dead' in the first place, and the fix proposed for reclaiming that stranded capacity rather than treating it as an unavoidable cost of building an overcomplete dictionary.
The cost of pushing a dictionary too wide
Because the whole point of an overcomplete dictionary is to give sparse autoencoder training more candidate directions than the model has neurons, there is an obvious failure mode: rows of the weight matrix that the optimizer simply never uses. A dead feature is exactly this -- a dictionary feature that never activates across the entire training corpus, occupying capacity that the expansion factor R nominally counts but that contributes nothing to any reconstruction and is never scored for interpretability. Dead features are the concrete reason R cannot simply be pushed arbitrarily high in the hope of capturing more of a model's true feature set: past some point, added capacity stops translating into added live, useful directions (sparse-autoencoders, §"E NUMBER OF ACTIVE FEATURES", p. 17).
How severe the problem gets, and where
On the residual stream, the paper finds the problem is mild within a wide operating range: raising R keeps essentially the whole dictionary alive up to roughly 4x overcompleteness, with dead features only beginning to appear past that point (sparse-autoencoders, §"E NUMBER OF ACTIVE FEATURES", p. 17). On the MLP sublayer, the picture is much worse: dead features appear in large numbers even at alpha=0, with no sparsity pressure at all, so the failure is driven by the ReLU nonlinearity and MLP activation geometry rather than by the sparsity penalty or R alone (sparse-autoencoders, §"C.3 INTERPRETING THE MLP SUBLAYER", p. 15). The gap between nominal and effective dictionary size can be severe there: in a dictionary with twice as many features as neurons, less than half might be active enough to perform automatic interpretability, except in early layers, where most features stay alive (sparse-autoencoders, §"C.3 INTERPRETING THE MLP SUBLAYER", p. 15). This is part of why MLP-sublayer training uses a different sparsity coefficient and untied encoder/decoder matrices -- both partial responses to the same underlying dead-feature problem.
How a dead feature is identified, and the proposed fix
The paper operationalizes activity with a concrete threshold: a feature counts as active if it fires more than 10 times across 10 million datapoints, and anything below that line is dead (sparse-autoencoders, §"E NUMBER OF ACTIVE FEATURES", p. 17). Rather than leaving dead rows permanently inert, the paper proposes periodically reinitializing them during training so that wasted capacity can be recovered and put toward learning additional useful directions -- explicitly framed as future work rather than something implemented in the reported experiments (sparse-autoencoders, §"E NUMBER OF ACTIVE FEATURES", p. 17).