Monosemanticity
Sparse Autoencoders Find Highly Interpretable Features in Language Models — inherited
The property of a feature or direction activating for a single, coherent, human-interpretable concept -- the converse of polysemanticity. The paper's case studies (the apostrophe and closing-parenthesis features) are framed as demonstrating that learned dictionary features are highly monosemantic compared to neurons or other baseline directions.
Getting a single unit inside a network to mean just one thing sounds like a modest ask, but it's the property individual neurons routinely fail at, and showing that a different kind of unit can actually achieve it, reliably and at scale, turns out to be surprisingly hard to pin down.
The entry then turns to how the attempt to measure this property at scale runs into a blind spot in its own scoring method, and what a convincingly monosemantic feature turns out to look like up close: often several narrower features quietly dividing up a concept a person would just call one thing.
The target property superposition prevents
Monosemanticity is the converse of Polysemanticity: a feature or direction is monosemantic if it activates for one coherent, human-nameable concept rather than several unrelated ones. As a target property it belongs to the same Anthropic interpretability line that named polysemanticity and superposition as its diagnosis (Elhage et al., 2022b), and this paper is one of two concurrent 2023 empirical demonstrations, alongside Anthropic's own 'Towards Monosemanticity' work, that training a sparse dictionary over a language model's activations yields directions measurably closer to this ideal than neurons or generic matrix decompositions. The paper's own case studies, the apostrophe feature and the layer-5 closing-parenthesis feature, are framed explicitly as evidence that Dictionary features achieve reduced polysemanticity relative to the baselines it tests.
How the paper measures it, and where the measurement breaks down
Because monosemanticity cannot be checked by hand across thousands of features, the paper relies on the Autointerpretability score as a scalable stand-in: a feature earns a high score only if its activations follow a pattern a language model can state and then use to predict held-out activations, something difficult for a feature that fires on more than one unrelated concept. The proxy has a structural blind spot the paper acknowledges directly: current interpretability protocols 'are unable to verify outputs by looking at changes in output or other data' (sparse-autoencoders, §"3.2 SPARSE DICTIONARY FEATURES ARE MORE INTERPRETABLE THAN BASELINES", p. 5). So the paper's most direct monosemanticity evidence instead comes from hand-run case studies: an activation histogram technique that plots which tokens cause a feature to fire, which 'only works for dictionary features that activate for a small set of tokens' (sparse-autoencoders, §"5.1 INPUT: DICTIONARY FEATURES ARE HIGHLY MONOSEMANTIC", p. 7), together with directly measuring a feature's causal effect on output logits.
What monosemanticity looks like up close: the apostrophe feature
The paper's central worked example, dictionary feature 556, activates almost exclusively on apostrophe tokens (sparse-autoencoders, §"5.1 INPUT: DICTIONARY FEATURES ARE HIGHLY MONOSEMANTIC", p. 7), in sharp contrast to the corresponding Residual stream default-basis dimension, which the paper shows encodes different information at different activation ranges and is therefore polysemantic. Yet even this feature 'does not activate on all apostrophes' (sparse-autoencoders, §"5.1 INPUT: DICTIONARY FEATURES ARE HIGHLY MONOSEMANTIC", p. 7): two further features cover apostrophes in contraction contexts like '[I/We/They]'ll' and negation contexts like '[don/won/wouldn]'t' respectively. The dictionary does not resolve polysemanticity by mapping one feature onto one human-level word category; it does so by learning several narrower, context-specific monosemantic features that jointly cover what a person would casually call a single concept.