Tied encoder/decoder weights
Sparse Autoencoders Find Highly Interpretable Features in Language Models — introduced
The design choice of using a single weight matrix (transposed between encoder and decoder) for the SAE, adopted because it encodes the expectation that a feature's detecting and defining directions coincide, halves memory cost, and removes encoder/decoder direction ambiguity. The paper finds tied weights hurt performance on MLP activations, motivating an untied variant with separate encoder/decoder matrices for that setting.
Nothing forces an autoencoder's reading weights and writing weights to be the same matrix, yet a generic one keeps them separate: one set for reading a feature in, another for writing it back out. Tying them into a single shared matrix here is a deliberate choice, one that also settles an otherwise awkward question — which direction, the reading one or the writing one, actually counts as the feature.
Taken up next is why that choice was made, what it costs to give up once results start slipping on a harder kind of activation, and why untying the two weight sets there sacrifices exactly the clean, single-direction bookkeeping that tying was meant to provide.
Why the encoder and decoder share one matrix
A generic autoencoder has two separate weight matrices, one mapping activations into the hidden code and another mapping the code back out; tying them -- using a single matrix M, transposed between encoder and decoder -- is a specific design choice this paper makes for its sparse autoencoders, not a structural necessity. The paper gives three reasons: tying encodes the expectation that the direction which detects a feature and the direction that defines it should coincide, it halves the model's memory cost, and it removes any ambiguity about whether a 'dictionary feature' direction should be read from the encoder or the decoder, since with tied weights the two are the same by construction (sparse-autoencoders, §"2 TAKING FEATURES OUT OF SUPERPOSITION WITH SPARSE DICTIONARY LEARNING", p. 2).
What that equivalence buys the rest of the paper
Every main-text result -- the interpretability comparisons of Section 3 and the IOI patching results of Section 4 -- depends on being able to treat a dictionary feature as a single, well-defined direction, for autointerpretability scoring, for ablation, and for patching, rather than needing to track separate detecting and defining directions for the same feature. Tied weights are what make that simplification valid: the paper reports that tying does not reduce performance when training on residual-stream data, meaning nothing is sacrificed in exchange for the analytical convenience (sparse-autoencoders, §"2 TAKING FEATURES OUT OF SUPERPOSITION WITH SPARSE DICTIONARY LEARNING", p. 2).
Where tying stops being free
Tied weights are the default everywhere in the paper except one place: training dictionaries on MLP-sublayer activations, where the paper reports observing reduced performance and retains more live features by switching to separate encoder and decoder matrices instead (sparse-autoencoders, §"C.3 INTERPRETING THE MLP SUBLAYER", p. 15). Concretely, the shared-matrix forward pass $c = \mathrm{ReLU}(Mx+b)$, $\hat{x} = M^Tc$ becomes, for this one setting, $$c = \mathrm{ReLU}(M_e x + b), \qquad \hat{x} = M_d^T c$$ with independent encoder matrix $M_e$ and decoder matrix $M_d$ (sparse-autoencoders, §"C.3 INTERPRETING THE MLP SUBLAYER", p. 15). The cost of this switch is exactly the ambiguity tying was designed to remove: an MLP dictionary feature's detecting direction (a row of $M_e$) and its defining direction (a row of $M_d$) can now differ, so recovering some of the lost dead-feature capacity comes at the price of no longer having one canonical direction per feature.