Dictionary feature
Sparse Autoencoders Find Highly Interpretable Features in Language Models — introduced
A single learned row of the sparse autoencoder's weight matrix M (equivalently, one hidden unit of the autoencoder), representing a direction in activation space used to reconstruct inputs as a sparse linear combination. Individual dictionary features are the paper's central unit of analysis, examined for interpretability (Section 3), causal importance (Section 4), and monosemanticity (Section 5).
A sparse autoencoder hands back thousands of learned directions after training, but the case for the method rests on what happens when just one is isolated: does it track a single idea, or stay as tangled as the neuron it was meant to replace? A dictionary feature is that isolated unit — one row of the trained matrix, standing in for one of the model's real, hidden features.
From here, the entry unpacks its exact mathematical form, why isolating one feature at a time lets researchers test and trace it much like earlier work tested individual neurons, and what it means to earn a monosemantic label narrowly, with related features often splitting a concept a person would treat as just one.
The atomic unit the SAE produces
A dictionary feature is the basic output of training a sparse autoencoder: one row of the learned weight matrix M, equivalently one hidden unit of the autoencoder, treated as a single direction in a language model's activation space (sparse-autoencoders, §"2 TAKING FEATURES OUT OF SUPERPOSITION WITH SPARSE DICTIONARY LEARNING", p. 2). Where the underlying Superposition hypothesis holds that a model's true, ground-truth features outnumber its neurons and so cannot be read off individual coordinates, a dictionary feature is the method's candidate stand-in for one such ground-truth feature: a learned direction that the training objective pushes toward representing one coherent, sparsely-active concept rather than several unrelated ones smeared across a coordinate. Every subsequent claim the paper makes about interpretability, causal precision, and monosemanticity is a claim about properties of individual dictionary features, making it the paper's central unit of analysis rather than the dictionary or the model as a whole.
What having many of them enables
Because a dictionary typically holds thousands of features, individual ones can be tested and used the way individual neurons are studied in classical interpretability work, but with far less polysemanticity to contend with. A feature's autointerpretability score is not evenly earned across depth: it shrinks the further a feature sits into the model, becoming merely comparable to ICA by layer 4 and showing minimal improvement by the model's final layer, partly because later features are best explained by their effect on outputs rather than by the input text the scoring protocol relies on (sparse-autoencoders, §"3.2 SPARSE DICTIONARY FEATURES ARE MORE INTERPRETABLE THAN BASELINES", p. 5). Individual features can also be ablated to read off a causal effect on next-token logits (sparse-autoencoders, §"5.2 OUTPUT: DICTIONARY FEATURES HAVE INTUITIVE EFFECTS ON THE LOGITS", p. 7), and traced to the upstream features that cause them to fire, which is how the paper builds automatic feature circuits such as the closing-parenthesis case study (sparse-autoencoders, §"5.3 INTERMEDIATE FEATURES: DICTIONARY FEATURES ALLOW AUTOMATIC CIRCUIT DETECTION", p. 8). None of this is possible at the level of the whole dictionary; it depends on being able to isolate and interrogate one feature at a time.
How a dictionary feature is defined, and what keeps it well-formed
Formally, for an activation vector $x \in \mathbb{R}^{d_{in}}$, the autoencoder's hidden code is $c = \mathrm{ReLU}(Mx+b)$ and its reconstruction is $\hat{x} = M^Tc = \sum_i c_i f_i$, where each $f_i$ -- a row of $M$ -- is a dictionary feature and $c_i$ is that feature's activation (coefficient) on this particular input (sparse-autoencoders, §"2 TAKING FEATURES OUT OF SUPERPOSITION WITH SPARSE DICTIONARY LEARNING", p. 3). A dictionary feature has only one free parameter, its direction, because the L1 sparsity penalty forces every row of $M$ to unit norm; without that constraint the optimizer could shrink the sparsity loss for free by growing a feature's row norm in $M$ and shrinking its coefficient in $c$ correspondingly, without genuinely sparsifying anything. A feature that never activates across the training corpus is a dead feature -- it still exists structurally as a row of $M$, occupying dictionary capacity, but contributes to no reconstruction and is never scored. L4-norm maximization theory predicts, and the paper finds weakly but consistently (correlations of 0.19-0.24), that features whose activations are more heavy-tailed -- mostly near zero with occasional large spikes -- tend to score higher on interpretability, linking a feature's statistical shape to how cleanly it satisfies Monosemanticity (sparse-autoencoders, §"C.4 INTERPRETABILITY SCORES CORRELATE WITH KURTOSIS AND SKEW OF ACTIVATION", p. 15).
Monosemantic, but narrowly so
Meeting the paper's monosemanticity bar does not mean one dictionary feature captures a whole human concept end to end. The technique the paper uses to test it -- histogramming which tokens activate a feature -- 'only works for dictionary features that activate for a small set of tokens' (sparse-autoencoders, §"5.1 INPUT: DICTIONARY FEATURES ARE HIGHLY MONOSEMANTIC", p. 7), and applied at that resolution the dictionary resolves polysemanticity not by finding one feature per concept but by proliferating many narrower, context-specific features: apostrophes alone are split across separate features for '[I/We/They]'ll'-type contexts and '[don/won/wouldn]'t'-type contexts, in addition to the feature that covers apostrophes generally (sparse-autoencoders, §"5.1 INPUT: DICTIONARY FEATURES ARE HIGHLY MONOSEMANTIC", p. 7). A dictionary feature, in other words, is monosemantic at the grain the method actually delivers -- a narrow, reliably-triggered pattern -- rather than at the grain of an everyday word like 'apostrophe.'