Default (neuron/residual-stream) basis baseline
Sparse Autoencoders Find Highly Interpretable Features in Language Models — inherited
Treating individual neurons, or for the residual stream individual basis coordinates, as the units of analysis, used as a naive baseline against which the paper's dictionary features are compared for autointerpretability. The residual-stream default basis is shown, via the apostrophe-dimension case study, to be polysemantic where the corresponding dictionary feature is monosemantic.
Before reaching for any decomposition technique at all, it's worth asking what interpretability looks like if you do nothing to the data — just take the individual coordinates a network already computes in and treat each one as a candidate feature. That's the default basis, and it matters as a baseline precisely because it's the option every more elaborate technique has to justify improving on, not a strawman built to lose easily.
The page covers why treating raw coordinates as meaningful units is a genuinely open question rather than an obvious mistake, and a worked comparison — a single coordinate against its closest learned-feature counterpart — that shows how much is lost by skipping decomposition altogether.
Treating the model's own coordinates as the features
The default basis is the baseline that asks what interpretability looks like if you do nothing: take the units the model already computes in -- individual neurons for the MLP, individual residual-stream coordinates elsewhere -- and treat each one as a candidate feature, with no decomposition applied at all. It stands in for the pre-dictionary-learning default in interpretability research, against which the paper measures how much sparse dictionary features, Principal Component Analysis (PCA), Independent Component Analysis (ICA), and the Random directions baseline actually improve on doing nothing (sparse-autoencoders, §"3.2 SPARSE DICTIONARY FEATURES ARE MORE INTERPRETABLE THAN BASELINES", p. 3).
A basis with no strong claim to being privileged
Whether treating individual coordinates as meaningful units makes sense at all depends on whether the space has a Privileged basis -- a coordinate system distinguished as unusually meaningful, the way the standard basis is right after an elementwise nonlinearity like ReLU. The Residual stream has no comparable reason to be privileged, yet transformers still tend to write disproportionately large values into specific coordinates, so-called Outlier dimensions, hypothesized to arise from the Adam optimizer storing gradients at finite precision in the residual basis (Dettmers et al., 2022). Despite that partial privilege, the paper reports it "does not find residual stream basis directions to be any more interpretable than random directions" (sparse-autoencoders, §"C.3 INTERPRETING THE MLP SUBLAYER", p. 15) -- carrying disproportionate information is not the same as being interpretable one coordinate at a time.
Losing head-to-head to the matched dictionary feature
The clearest illustration is the Apostrophe dictionary feature (feature 556) case study: searching the residual stream's raw coordinates for the dimension most associated with apostrophes found one (dimension 21) that was only the 10th-highest dimension the dictionary's own apostrophe feature reads from, since the nine stronger-weighted coordinates didn't show apostrophes among their own top activations (sparse-autoencoders, §"D.1 RESIDUAL STREAM BASIS", p. 16). Even that best-available coordinate stays polysemantic where the dictionary feature does not: it shows a large share of apostrophes at its highest activation values but "only explains a very small fraction of the variance for middle-to-lower activation ranges," meaning it means something else most of the time it fires (sparse-autoencoders, §"D.1 RESIDUAL STREAM BASIS", p. 16). This single comparison is a worked instance of the aggregate finding across Section 3.2: the default basis is far less interpretable than dictionary features on the same metric.