SAE training on the MLP sublayer — confronts → Privileged basis

The MLP sublayer is the paper's own worked example of a space that does have a privileged basis -- it is literally the definitional example given elsewhere in the paper, 'the standard basis after a non-linearity in an MLP' -- so training a dictionary there means competing against neurons that are already somewhat meaningfully aligned with the coordinate axes, unlike in the residual stream. Even so, the paper reports 'mixed success': many MLP dictionary features are more interpretable than the neurons themselves, showing the privileged basis does not make neurons optimal. But capacity is wasted differently here than in the residual stream -- MLP dictionaries develop large numbers of dead features that never activate, severely enough that 'in a dictionary with twice as many features as neurons, less than half might be active enough to perform automatic interpretability,' except in early layers where most features stay alive. Tied encoder/decoder weights, which cost nothing on residual-stream data, also measurably hurt MLP performance, forcing the paper to use separate encoder and decoder matrices specifically for this setting.