Unmixing activations with a sparsity penalty — hands off only what fires to → Interpretability itself becomes a benchmark

The autointerpretability pipeline does not grade every dictionary feature the design produces -- it can only grade features that fire enough to generate text fragments with non-zero variation, and the protocol explicitly skips any feature with fewer than 20 such fragments among 50,000 lines of OpenWebText (sparse-autoencoders, §A, p. 12). That requirement collides directly with dead features, the failure mode this theme documents as common under MLP training: in a dictionary twice the size of the neuron layer, "less than half might be active enough to perform automatic interpretability" (sparse-autoencoders, §"C.3 Interpreting the MLP Sublayer", p. 15), meaning scoring coverage silently shrinks precisely where the sparsity-reconstruction tradeoff pushes hardest toward overcompleteness. Reconstruction fidelity, by contrast, turns out to be nearly irrelevant to the score once a feature clears that firing bar: smaller, less accurate dictionaries score just as interpretable per feature as larger ones, so a larger dictionary only explains more variance rather than scoring better feature-by-feature (sparse-autoencoders, §"C.1 Interpretability Is Consistent Across Dictionary Sizes", p. 13). What neither theme states is how asymmetric the handoff is: the grading apparatus is blind to reconstruction quality but not to activation frequency, so the design axis theme 1 spends most of its attention on (the sparsity-reconstruction curve) barely touches the score, while the failure mode it treats as a side detail (dead features) determines how much of the dictionary even reaches grading.