Transformer visualization via dictionary learning (Yun et al., 2021)
Sparse Autoencoders Find Highly Interpretable Features in Language Models — inherited
Prior work applying sparse dictionary learning to residual stream activations across all layers of a language model simultaneously, framing token representations as a linear superposition of transformer factors. Cited as closely related prior work alongside Sharkey et al. (2023), applying a similar technique but without the interpretability and causal-localization evaluations this paper contributes.
Applying sparse dictionary learning to a transformer's internal representations wasn't a new idea when this paper did it — an earlier paper had already tried something close, decomposing token representations the same way, years before anyone asked whether the resulting directions were actually interpretable or causally meaningful.
This entry covers what that earlier work did differently, fitting one shared dictionary across every layer rather than training one dictionary per layer, what it never tested that gets added here, and how a single shared co-author threads a decades-old signal-processing idea through both projects.
Direct prior work, cited by name
This concept names a specific prior paper, Yun, Chen, Olshausen & LeCun (2021), 'Transformer visualization via dictionary learning,' rather than a general technique. It applies sparse dictionary learning, the same Olshausen & Field (1997) framework this paper also inherits, to a transformer's token representations, treating each token's representation as a linear superposition of a shared set of learned 'transformer factors.' The paper cites it directly alongside Sharkey et al. (2023) interim research report as the closest prior work to its own approach: 'Our approach is also similar to Yun et al. (2021), who apply sparse dictionary learning to all residual stream layers in a language model simultaneously' (sparse-autoencoders, §"1 INTRODUCTION", p. 2).
What differs: granularity and evaluation
The key difference from this paper is granularity, not method: Yun et al. (2021) fit one shared dictionary across every layer of the Residual stream simultaneously, while this paper trains a separate Sparse autoencoders (SAEs) instance per layer, and separately again for the MLP sublayer in Appendix C.3, which is what makes it possible to track interpretability and dead-feature rates layer by layer rather than only at the level of the whole model. A larger gap is evaluative: Yun et al. never test whether their learned factors are more interpretable than baselines, more monosemantic than neurons, or more causally precise on a behavioral task, which is exactly the Autointerpretability score, case-study, and Indirect Object Identification (IOI) task-patching apparatus this paper layers on top of essentially the same underlying idea.
A continuity link in the dictionary-learning lineage
Concretely, Yun et al. (2021) express each token's representation at every depth as a combination of a common set of 'transformer factors,' a visualization tool for how a token's meaning is built up through the network rather than an interpretability-scoring framework. Because Bruno Olshausen, co-author of the 1997 paper that originated sparse coding, is also a co-author here, the work functions as a continuity link running through this corpus's lineage: the same mathematical framework moves from modeling receptive fields in the visual cortex, to visualizing transformer representations as superposed factors, to this paper's per-layer, per-feature interpretability and causal analysis of language model activations. The paper's related-work section places both Yun et al. (2021) and the Sharkey et al. (2023) interim research report together as the limited set of prior works that 'have learned dictionaries of sparsely-activating features in pretrained models' (sparse-autoencoders, §"6.1 RELATED WORK", p. 8).