Tied encoder/decoder weights — gives way to an untied variant in → SAE training on the MLP sublayer

Tied weights are the default everywhere in the paper except one place: training dictionaries on MLP-sublayer activations. There, the paper reports retaining a larger number of live features by switching to separate encoder and decoder matrices, c = ReLU(M_e x + b) and x-hat = M_d^T c (Equations 5-6 in Appendix C.3), abandoning the single shared matrix M used on the residual stream. This is a direct response to the dead-features problem: MLP dictionaries lose so many features to inactivity that recovering some of that lost capacity outweighs the memory savings and detecting/defining-direction guarantee that motivated tying in the first place. The tradeoff the paper accepts is that MLP dictionary features, unlike residual-stream ones, no longer have a single canonical direction -- their encoder (detecting) and decoder (defining) directions can differ, reintroducing exactly the ambiguity tied weights were designed to remove.