Privileged basis
Sparse Autoencoders Find Highly Interpretable Features in Language Models — inherited
The property of a particular coordinate basis being distinguished as unusually meaningful for a given vector space (e.g. the standard basis immediately after an elementwise nonlinearity), as opposed to a generic space with no privileged direction. The paper notes the residual stream is not usually expected to have a privileged basis, but discusses evidence (Dettmers et al., 2022; Elhage et al., 2023) that it partially does due to outlier dimensions, which is relevant to interpreting the default-basis baseline's behavior.
Whether a network's raw coordinates mean anything, or are just one arbitrary rotation among infinitely many equally valid ones, isn't a philosophical question — it decides whether comparing a learned feature against the model's own neurons is a fair fight or one rigged from the start.
Spelled out next is what makes a basis privileged in the first place, why one part of a network earns that status while another mostly doesn't, and a documented, partial exception that complicates an otherwise clean story about which coordinates in a model are worth treating as meaningful on their own.
A basis that means something, and one that doesn't
A privileged basis is a coordinate system for a vector space that carries genuine meaning, as opposed to being one arbitrary choice among infinitely many equivalent rotations. Whether a basis is privileged is a property of the operations applied to the space, not of the space itself: a generic linear layer treats all directions symmetrically, so no basis is singled out, but an elementwise nonlinearity like ReLU is not rotation-invariant, since it is defined coordinate by coordinate. The paper's own working definition is a basis 'in which we would expect changes to be unusually meaningful, such as the standard basis after a non-linearity in an MLP' (sparse-autoencoders, §"C.3 INTERPRETING THE MLP SUBLAYER", p. 15). The framing was developed for transformers specifically by Elhage et al. (2023), which this paper cites directly.
Why this matters for choosing a baseline
Whether a space has a privileged basis determines whether treating its raw coordinates as the unit of analysis is a principled choice or an arbitrary one. Because the paper finds the Residual stream lacks a privileged basis in general, individual residual-stream coordinates, the Default (neuron/residual-stream) basis baseline the paper compares its Dictionary features against, have no special claim to interpretability over any other direction, including a randomly sampled one, and the paper tests this directly. The concept plays the opposite role for the MLP sublayer, where the standard basis genuinely is privileged by the nonlinearity that immediately precedes it, so training a dictionary there means competing against neurons that already have some architectural claim to meaningfulness, unlike in the residual stream.
Where the neuron basis is privileged, and where it partially isn't
Mechanically, a linear operation such as the residual stream's running sum of layer outputs commutes with any rotation of coordinates: rotate the inputs and the output rotates the same way, so no coordinate axis is distinguished from any other. An elementwise nonlinearity breaks this symmetry, since $\mathrm{ReLU}(z)_i = \max(z_i, 0)$ depends on the value in coordinate $i$ specifically, which is what makes the post-nonlinearity standard basis privileged. By that logic the residual stream, being additive rather than the output of a nonlinearity, should have no privileged basis at all, yet the paper reports a documented, partial exception: transformers show 'a tendency ... to store information in the residual stream basis' (Dettmers et al., 2022), attributed to the Adam optimiser's finite-precision gradient storage in that basis, concentrated specifically in Outlier dimensions rather than restored uniformly across all coordinates.