The dictionary-learning lineage — leaves the architecture for → Unmixing activations with a sparsity penalty

Both direct predecessors leave a specific engineering gap that this theme's design choices fill in. Yun et al. (2021) apply sparse dictionary learning "to all residual stream layers... simultaneously" (sparse-autoencoders, §"1 Introduction", p. 2) -- one dictionary asked to explain every layer at once -- while this paper instead trains a fresh autoencoder per layer and per sublayer (separate dictionaries reported for, e.g., the layer 1 residual stream, layer 2 and layer 11 elsewhere), which is what makes the per-layer alpha/R hyperparameter sweep in Appendix B a coherent operation in the first place, one that a single all-layers-at-once dictionary could not support. Sharkey et al.'s (2023) unpublished interim report already showed "empirically... that reconstruction with an l1 penalty can recover the ground-truth features" (sparse-autoencoders, §2, p. 3), establishing that sparsity works in principle and motivating this paper directly; what it left unresolved is architecture at the scale of a full transformer's sublayers, which is exactly where tied weights (halving the parameter count) and the discovery of dead features specific to MLP training (sparse-autoencoders, §"C.3 Interpreting the MLP Sublayer", p. 15) enter -- problems that only surface once the idea is pushed past a single demonstrated recovery result and into a sweepable, per-sublayer training pipeline. Neither narrative states this handoff: the lineage theme credits Sharkey with the motivating idea and Yun with the simultaneous-layers precedent, while the instrument theme presents tied weights and dead features as this paper's own design decisions without naming the specific gap in scale and per-layer decomposition that made them necessary.