Random directions baseline
Sparse Autoencoders Find Highly Interpretable Features in Language Models — inherited
Randomly sampled directions in activation space, used as a lower-bound baseline for autointerpretability comparisons alongside the default basis, PCA, and ICA.
Before you can say a learned feature is genuinely interpretable, you need some idea of what a completely meaningless direction in the same space would score — otherwise a modest-looking number has no way to be judged good or bad. Random directions supply that floor: vectors chosen with no reference to any data at all, projected onto a model's activations purely to see what an uninformed guess looks like next to the real thing.
This page covers how these directions are generated, the small accommodation made so they aren't unfairly penalized for a mismatch in how activations are typically shaped, and why the floor they set still comes out low even with that accommodation in their favor.
The floor the other results are measured against
Of the four baselines this paper compares against its learned dictionaries, randomly sampled directions in activation space are the deliberately unstructured one: no data, no optimization, nothing about the model informs their choice. That is exactly the point -- they exist to define the bottom of the Autointerpretability score's usable range, so that Principal Component Analysis (PCA), Independent Component Analysis (ICA), and the Default (neuron/residual-stream) basis baseline can be judged not just against the sparse dictionary but against a baseline with no claim to meaning anything at all (sparse-autoencoders, §"3.2 SPARSE DICTIONARY FEATURES ARE MORE INTERPRETABLE THAN BASELINES", p. 3).
What 'random' means here
A random direction is simply a vector drawn without reference to any activation data -- intuitively, a point sampled uniformly on the unit hypersphere in the model's activation space, $u \sim \mathrm{Unif}(\mathbb{S}^{d-1})$, with no preference for any particular orientation. Projecting a language model's activations onto such a direction and reading off the result is the null hypothesis for feature-finding: whatever pattern of activation shows up is, by construction, not the product of the direction having been chosen for anything, so any interpretability it exhibits has to come from the activations themselves rather than the projection.
Given the benefit of the doubt, and still losing
The comparison is not run against random directions in the rawest possible form: because the Autointerpretability score and the dictionary features it evaluates both assume nonnegative, ReLU-shaped activations, the paper applies the same accommodation to this baseline that it gives the Default (neuron/residual-stream) basis baseline -- "for the random directions and for the default basis in the residual stream, we replace negative activations with zeros so that all feature activations are nonnegative" (sparse-autoencoders, §"3.2 SPARSE DICTIONARY FEATURES ARE MORE INTERPRETABLE THAN BASELINES", p. 3). Random directions are therefore scored on a version of themselves already reshaped to look more like a genuine feature, which if anything should inflate their apparent interpretability. That the floor still sits low despite this handicap in its favor is what gives the score room to distinguish better and worse decompositions at all.