Sparsity-reconstruction tradeoff (no single correct decomposition)
Sparse Autoencoders Find Highly Interpretable Features in Language Models — introduced
The paper's empirical finding, from sweeping the sparsity coefficient alpha, that there is a smooth tradeoff between the number of active features and unexplained reconstruction variance, with no distinct 'knee' or single best operating point. Taken as weak evidence against there being one uniquely correct sparse decomposition of a given activation space.
If a layer's activations decomposed into one uniquely correct set of features, pushing a dictionary to be sparser should eventually hit a wall — some point past which sparsity starts costing accuracy in a way that marks the true boundary. Sweeping that dial in practice finds no such wall, only a smooth curve of equally defensible tradeoffs.
What follows works out what a curve without a knee implies about whether a model has one uniquely correct decomposition at all, why the paper reports results at several settings rather than tuning to one, and how two separate hyperparameters, one sliding a dictionary along the curve and the other redrawing it, together fix where a given dictionary lands.
What a tradeoff curve with no knee implies
If sparse coding could recover one single, uniquely correct set of ground-truth features from a layer's activations, sweeping how strongly reconstruction is traded against sparsity should reveal some special operating point -- a bend or 'knee' where further sparsification starts costing disproportionately more reconstruction accuracy. The paper looks for exactly this by varying the sparsity coefficient alpha and finds a smooth curve instead, with no such distinguished bend (sparse-autoencoders, §"B SPARSE AUTOENCODER TRAINING AND HYPERPARAMETER SELECTION", p. 12). Read at face value, this is weak evidence against there being one uniquely correct sparse decomposition of a given activation space: every (alpha, R) setting on the curve looks like a defensible, merely different, decomposition, not an approximation converging toward some single ground-truth dictionary that a knee would mark.
Why the paper reports many settings instead of tuning to one
This finding is why the paper evaluates its method across a range of alpha and R values rather than presenting one canonical, best-tuned dictionary: Sections 3 and 4 report results at multiple settings -- for instance alpha=.00086 with R=2 for the main interpretability comparison, and a separate alpha sweep for the IOI patching results -- treating the choice of operating point as a modeling decision with real consequences (sparsity vs. fidelity) rather than a nuisance parameter to be optimized away (sparse-autoencoders, §"3.2 SPARSE DICTIONARY FEATURES ARE MORE INTERPRETABLE THAN BASELINES", p. 5). It also reframes what 'the model's features' means as a target for the whole method: not a fixed list to be recovered exactly, but a family of similarly-valid sparse bases that trade sparsity and fidelity against each other, all consistent with the underlying activations to varying degrees.
The shape of the curve, and what sets which curve you're on
The tradeoff is measured by plotting the average number of active features against the proportion of reconstruction variance left unexplained as alpha varies, for a fixed dictionary size (sparse-autoencoders, §"B SPARSE AUTOENCODER TRAINING AND HYPERPARAMETER SELECTION", p. 12). That dictionary size is itself a second free parameter: the expansion factor R sets which curve a given training run traces out in the first place, since a smaller R gives the tradeoff less capacity to work with and a larger R shifts the whole curve toward explaining more variance at a given sparsity level (sparse-autoencoders, §"C.1 INTERPRETABILITY IS CONSISTENT ACROSS DICTIONARY SIZES", p. 13). So alpha moves a trained dictionary along a fixed curve, while R redraws the curve itself -- the two hyperparameters are not interchangeable knobs on the same axis, and both are needed to describe where in this space a particular sparse autoencoder sits.