Reconstruction loss
Sparse Autoencoders Find Highly Interpretable Features in Language Models — introduced
The ||x - x-hat||^2 term in the SAE training objective (Equation 4), the standard squared-error measure of how well the reconstructed activation vector matches the original. The paper notes this loss is never driven to zero, showing the dictionary fails to capture all information in a layer's activations, a gap it also quantifies via the reconstruction-perplexity metric.
Penalizing an autoencoder for using too many directions creates an easy way to cheat: switch nothing on, and the penalty disappears along with any real reconstruction work. Reconstruction loss is what keeps that cheat from paying off, rewarding an autoencoder only when its sparse code still manages to reproduce the activation it was given.
The rest of this entry walks through the loss in its exact form, the tug-of-war it wages against sparsity during training, and why the paper treats its own inability to drive this term to zero as the clearest sign that its method still leaves real information about the model unexplained.
The half of the objective that keeps the dictionary honest
Reconstruction loss is what stops sparse autoencoder training from degenerating into a decomposition that is sparse but meaningless: without a term rewarding faithful reconstruction, the L1 penalty alone could be minimized by simply never activating any dictionary feature. Reconstruction loss is the standard squared-error measure of how far the reconstructed activation $\hat{x}$ falls from the original $x$, and it is what ties the sparse code back to the actual activations of the language model being studied, rather than to some arbitrary sparse pattern unrelated to what the model computes (sparse-autoencoders, §"2 TAKING FEATURES OUT OF SUPERPOSITION WITH SPARSE DICTIONARY LEARNING", p. 3).
Why the paper treats this loss as its clearest limitation
The paper reports plainly that this loss is never driven to zero, meaning the learned dictionaries fail to capture all the information present in a layer's activations (sparse-autoencoders, §"6.2 LIMITATIONS AND FUTURE WORK", p. 8). Because squared-error numbers on cached activations have no inherent interpretation, the paper backs this claim with a concrete downstream check, the Reconstruction-substitution perplexity metric: substituting Pythia-70M's layer-2 residual stream with its SAE reconstruction during a real forward pass raises Pile perplexity from 25 to 40, evidence that the gap reconstruction loss reports is not a rounding error but a real, measurable loss of information the model actually uses (sparse-autoencoders, §"6.2 LIMITATIONS AND FUTURE WORK", p. 9). This is also why the paper's enumerative-safety framing in its conclusion is careful to call the method 'a step towards' rather than an achievement of a complete feature enumeration -- an incomplete reconstruction is direct evidence the dictionary is not yet capturing everything relevant (sparse-autoencoders, §"6.3 CONCLUSION", p. 9).
The term itself and its tug-of-war with sparsity
Reconstruction loss is the first term of $$L(x) = \underbrace{\|x - \hat{x}\|_2^2}_{\text{Reconstruction loss}} + \underbrace{\alpha\|c\|_1}_{\text{Sparsity loss}}$$ where $\hat{x} = M^Tc$ is the autoencoder's output (sparse-autoencoders, §"2 TAKING FEATURES OUT OF SUPERPOSITION WITH SPARSE DICTIONARY LEARNING", p. 3). Because both terms sum into one objective, the optimizer is free to trade one against the other: dictionaries trained with a larger sparsity coefficient alpha reach lower reconstruction accuracy, visible in the IOI patching experiments as a higher floor on the minimum KL divergence no amount of further patching can close, while a non-sparse dictionary (alpha=0) reconstructs well but loses the causal-precision advantage over PCA that motivates using sparsity at all (sparse-autoencoders, §"4.2 PRECISE LOCALISATION OF IOI DICTIONARY FEATURES", p. 6). Sweeping alpha traces a smooth curve between the two losses with no distinguished 'knee,' the same empirical basis behind the Sparsity-reconstruction tradeoff (no single correct decomposition)'s finding that no single alpha setting is uniquely correct.