Reconstruction-substitution perplexity metric — measures a different axis than → Autointerpretability score

These two metrics are run as separate report cards on the same dictionaries and are never checked against each other. The interpretability score (Section 3) asks whether an automated system can explain and predict a feature's activations from text; the reconstruction-perplexity metric (Section 6.2) instead asks how much information the whole dictionary loses when it replaces a layer's activations outright, and finds a large gap: substituting Pythia-70M's layer-2 residual stream with its SAE reconstruction raises Pile perplexity from 25 to 40, nearly doubling it. A dictionary could in principle score well on interpretability, each active feature having a clean story, while still being unfaithful in this stronger sense, because interpretability scoring never looks at what the reconstruction does to downstream behavior, only at whether a feature's own activation pattern is describable. The paper flags reconstruction loss never reaching zero as a limitation precisely because it means the dictionary omits information the interpretability score has no way of noticing was missing.