Reconstruction-substitution perplexity metric
Sparse Autoencoders Find Highly Interpretable Features in Language Models — introduced
A fidelity check in which a layer's activations are replaced with the SAE's reconstruction of them and the resulting increase in the model's perplexity on the Pile is measured, quantifying information lost by the dictionary. Reported as rising from 25 to 40 when substituting Pythia-70M's layer-2 residual stream with its reconstruction, evidencing that reconstruction loss is not yet near zero.
Squared-error units don't tell a reader much about a reconstruction on their own — is losing that much information a rounding error, or does it break something the model actually relies on? The only way to know is to make the model use the imperfect reconstruction and watch what happens to its real behavior.
Laid out next is how that substitution is carried out, what a sizeable jump in prediction error from replacing just one layer's activations reveals about how much a dictionary still leaves out, and why this downstream check answers a different question than scoring how well one feature's own activation pattern can be explained in isolation.
Grounding an abstract loss number in real model behavior
Reconstruction loss is measured in squared-error units on cached activations, a quantity with no inherent meaning for how much a dictionary's imperfection actually matters to what the model does. The reconstruction-perplexity metric is the paper's way of converting that abstract number into something concretely interpretable: instead of just reporting a loss value, it substitutes a sparse autoencoder's reconstruction directly into the model's forward pass, in place of the real activations, and measures the resulting change in next-token prediction quality (sparse-autoencoders, §"6.2 LIMITATIONS AND FUTURE WORK", p. 9).
What it shows, and what it does not measure
Applied to Pythia-70M's layer-2 residual stream, substituting the SAE reconstruction for the true activations raises perplexity on the Pile from 25 to 40 -- roughly a 60% increase from replacing a single layer (sparse-autoencoders, §"6.2 LIMITATIONS AND FUTURE WORK", p. 9). The paper treats this as the clinching evidence that reconstruction loss, despite looking small in absolute squared-error terms, is not yet near zero in any sense that matters to the model's actual computation. This metric answers a different question than the Autointerpretability score does: interpretability scoring asks whether an automated system can explain and predict one feature's own activation pattern from text (sparse-autoencoders, §"3.1 INTERPRETABILITY AT SCALE", p. 3), while reconstruction-substitution perplexity asks how much information the whole dictionary loses when it stands in for a layer outright. A dictionary could in principle score well on the first while still failing the second, since interpretability scoring never checks what a reconstruction does to downstream model behavior.
How the substitution works
The procedure runs the language model as normal up to the target layer, replaces that layer's activation vector $x$ with the autoencoder's reconstruction $\hat{x} = M^Tc$, and continues the forward pass from there, then measures the resulting perplexity on held-out Pile text (sparse-autoencoders, §"6.2 LIMITATIONS AND FUTURE WORK", p. 9). Because it requires an actual forward pass through the rest of the model rather than just comparing $x$ and $\hat{x}$ directly, the metric is non-differentiable and computed only after training, unlike Reconstruction loss, which is the differentiable quantity optimized throughout. That distinction is deliberate: the perplexity check is a downstream validation of what the training loss curve alone cannot show -- whether the dictionary's reconstruction is good enough for the rest of the model to keep working normally.