Principal Component Analysis (PCA) — trails → Sparse autoencoders (SAEs)

On the IOI activation-patching task, sparse dictionary features reach a given KL divergence from the target output using fewer patched features and smaller edit magnitude than an equal-size set of PCA components, improving the Pareto frontier of edit magnitude versus thoroughness. Crucially, the paper isolates why: training the same autoencoder architecture with no sparsity penalty (alpha=0) removes the advantage over PCA entirely, so the gap is not explained by dictionary learning's flexibility to fit an arbitrary rotation of the data the way PCA cannot -- it is specifically the sparsity penalty forcing few features to matter per token that produces cleaner causal localization. PCA's variance-maximizing objective, by design, spreads a behaviour's signal across whichever directions carry the most variance in the corpus overall, which need not coincide with the few directions that causally drive one specific behaviour like IOI.