Dictionary expansion factor (R)
Sparse Autoencoders Find Highly Interpretable Features in Language Models — introduced
The hyperparameter R controlling the ratio of the autoencoder's hidden dimension to the model's activation dimension (dhid = R x din), governing how overcomplete the learned feature dictionary is. The paper sweeps R and finds interpretability gains hold even for undercomplete dictionaries (R<1), with larger dictionaries explaining more of the overall activation variance.
If a model really does pack more features into a layer than it has dimensions for, recovering them means building a dictionary deliberately bigger than the layer itself — and how much bigger turns out to matter more than it might first seem.
Ahead is the ratio itself, how far it can be pushed before the extra capacity stops adding anything useful, and how it interacts with the sparsity setting to decide which balance of accuracy and interpretability a given trained dictionary ends up landing on.
Turning the superposition hypothesis into a hyperparameter
If Superposition is right that a model packs more features into a layer than it has dimensions, then a dictionary meant to recover those features needs to be allowed to be larger than the layer itself. The expansion factor R is exactly the knob that allows this: it sets the ratio between the autoencoder's hidden dimension and the model's activation dimension, so that R>1 gives an overcomplete dictionary -- more candidate dictionary features than the model has neurons or residual-stream coordinates -- directly operationalizing what the superposition hypothesis requires the method to search for (sparse-autoencoders, §"2 TAKING FEATURES OUT OF SUPERPOSITION WITH SPARSE DICTIONARY LEARNING", p. 2).
What sweeping R actually buys, and where it breaks
The paper finds interpretability gains from dictionary learning are not an artefact of having an overcomplete dictionary specifically: they hold even for undercomplete dictionaries with R<1, though each such smaller dictionary explains less of the model's overall activation variance than a larger one at the same sparsity level, because there is simply less capacity to work with (sparse-autoencoders, §"C.1 INTERPRETABILITY IS CONSISTENT ACROSS DICTIONARY SIZES", p. 13). Pushed the other direction, R has a ceiling before the extra capacity goes to waste: for residual-stream dictionaries, raising R keeps essentially the whole dictionary alive up to roughly 4x overcompleteness, with dead features only beginning to appear past that point (sparse-autoencoders, §"E NUMBER OF ACTIVE FEATURES", p. 17). R also determines which tradeoff curve a given training run sits on in the first place: the smooth Sparsity-reconstruction tradeoff (no single correct decomposition) traced out by sweeping the sparsity coefficient alpha is measured at a fixed dictionary size, and a larger R shifts the whole curve toward explaining more variance at a given sparsity level without making any individual feature less interpretable on average (sparse-autoencoders, §"C.1 INTERPRETABILITY IS CONSISTENT ACROSS DICTIONARY SIZES", p. 13).
The ratio itself
Formally, the autoencoder's hidden dimension is $$d_{hid} = R \cdot d_{in}$$ where $d_{in}$ is the dimension of the language model activation being modeled -- 512 for Pythia-70M's residual stream and 1024 for Pythia-410M's -- and R is a free hyperparameter chosen per experiment (sparse-autoencoders, §"2 TAKING FEATURES OUT OF SUPERPOSITION WITH SPARSE DICTIONARY LEARNING", p. 2). The paper's main interpretability comparisons use R=2, a mild 2x overcomplete dictionary (sparse-autoencoders, §"3.2 SPARSE DICTIONARY FEATURES ARE MORE INTERPRETABLE THAN BASELINES", p. 5), while the dictionary-size sweep in Appendix C.1 tests R values from well below 1 up through several multiples, and the dead-feature sweep in Appendix E covers R up to and beyond the roughly 4x point where residual-stream dictionaries start to lose live capacity (sparse-autoencoders, §"C.1 INTERPRETABILITY IS CONSISTENT ACROSS DICTIONARY SIZES", p. 13).