SAE training on the MLP sublayer — stress tests → Sparse autoencoders (SAEs)

Every core result on the residual stream (Sections 3-4) uses a single working recipe: tied weights, alpha near 8.6e-4, and dictionaries that stay overcomplete up to roughly 4x expansion. Porting that recipe unchanged to MLP-sublayer activations (Appendix C.3) breaks two of its assumptions at once: many dictionary features still turn out more interpretable than individual neurons, but large numbers go dead even without any sparsity penalty, and the training pipeline does not yet reliably learn an overcomplete basis in the middle or later MLP layers -- only the earliest layers keep a large fraction of features alive. The fix that partially restores results is itself informative: MLPs need a different sparsity coefficient (alpha=3.2e-4 rather than 8.6e-4) and untied encoder/decoder matrices, meaning the sparse-autoencoder recipe validated on the residual stream is not activation-agnostic -- it has hidden dependencies on which part of the model it is applied to.