Less-than-rank-one ablation
Sparse Autoencoders Find Highly Interpretable Features in Language Models — introduced
A specific ablation technique that lowers a residual-stream activation's component along a feature's direction only up to the point where that feature is no longer active, rather than a full-magnitude subtraction. Used to measure a dictionary feature's causal effect on next-token logits in the apostrophe-feature case study.
Removing a feature from a network's activations sounds like a simple subtraction, but a full-strength ablation edits further than it needs to, which weakens any claim that the resulting change in output is really about that one feature. Less-than-rank-one ablation is a more careful version: instead of zeroing a feature's contribution outright, it walks the activation back only as far as the exact point where the feature stops firing, and no further.
The page covers the geometric reasoning behind that stopping point, the case study — a feature tied to apostrophes — where this method shows a clean, predictable effect on what the model predicts next, and how the same geometric distinction shows up again in an unrelated fairness check elsewhere in the paper.
The causal-proof theme's second surgical tool
Where dictionary-feature patching proves causal importance by transplanting activations from a counterfactual run, less-than-rank-one ablation proves it by subtraction within a single run: it is the technique behind the apostrophe feature's output analysis, the piece of evidence that a monosemantic-looking feature also has an intuitive, predictable causal effect on what the model predicts next.
Lowering an activation to its own boundary, not past it
The intuition is a stopping rule rather than a fixed subtraction. A standard ablation of a feature would zero out its entire encoded coefficient; less-than-rank-one ablation instead lowers the residual-stream activation along the feature's direction only as far as the point where the feature stops being active, and no further (sparse-autoencoders, §"5.2 Output: dictionary features have intuitive effects on the logits", p. 7). The name reflects the geometry: because a dictionary feature fires only after $c = \mathrm{ReLU}(Mx+b)$ clears zero, feature $i$ is active only when $(Mx)_i > -b_i$, and the learned bias $b_i$ is in practice always negative -- so a feature's "on" region is a half-space bounded by a hyperplane that does not pass through the origin, unlike a PCA or ICA direction's full-space split. Removing a feature therefore means walking back to that boundary, which is strictly less editing than subtracting a full-magnitude direction would be.
What it found on the apostrophe feature
Applied to the apostrophe feature (feature 556), less-than-rank-one ablation shows that removing the feature mainly reduces the logit for the following "s" token, matching the story that this feature detects apostrophes and is used to predict the "s" that follows in possessives and contractions such as "let's" (sparse-autoencoders, §"5.2 Output: dictionary features have intuitive effects on the logits", p. 7). Out of roughly 12,000 logits negatively affected in total, this single downstream token shows the largest drop -- evidence that the feature's causal role lines up with its input-side interpretation rather than diverging from it.
A finer edit than patching, for a reason grounded in fairness
Compared to Dictionary-feature patching procedure, which pushes a feature's activation the full distance from a base value to a counterfactual target, less-than-rank-one ablation only ever edits down to a feature's own activation boundary. That asymmetry is not incidental: the paper's fairness check in Appendix G, comparing dictionary features against fixed-K-active variants of PCA and ICA, is built around exactly this observation -- that a sparse-coding feature is active on less than a full direction while baseline directions like PCA split the whole space through the origin, a consequence of the bias term applied before the ReLU always being negative in practice (sparse-autoencoders, §"G Top K comparisons", p. 18). Ablating exactly to a feature's own boundary keeps the intervention consistent with that same geometric distinction.