L4-norm maximization (sparse dictionary search reformulation)
Sparse Autoencoders Find Highly Interpretable Features in Language Models — inherited
A prior theoretical result (Qu et al., 2019) showing that searching for sparse, overcomplete dictionaries can be reformulated as searching for directions that maximize the L4 norm. Cited to motivate the paper's finding that interpretability correlates with the heavy-tailedness (skew/kurtosis) of feature activations, and to help explain ICA's relatively strong baseline performance.
Searching directly for a sparse, overcomplete dictionary is a combinatorial nightmare, but a piece of pure theory offers a shortcut: instead of searching over discrete sparsity patterns, search continuously for directions where the data spikes rarely and dramatically rather than varying smoothly around some average.
Picked apart next is that reformulation and the statistical signature it predicts, and how the paper leans on it after the fact to make sense of two separate findings: why more heavy-tailed learned features tend to score as more interpretable, and why a technique built to chase that exact signature consistently beats simpler baselines.
A pure theory result the paper leans on
L4-norm maximization refers to a prior, purely theoretical result of Qu et al. (2019) showing that the combinatorial search for a sparse, overcomplete dictionary can be recast as a continuous optimization problem: searching, among unit-norm directions, for the one that maximizes the fourth moment of its projections onto the data. It belongs to the broader optimization-landscape literature on overcomplete representation learning rather than to interpretability research specifically, and this paper cites it only in an appendix, to interpret results it did not design the study to produce (sparse-autoencoders, §"C.4 INTERPRETABILITY SCORES CORRELATE WITH KURTOSIS AND SKEW OF ACTIVATION", p. 15).
What it buys the paper: an explanation for two separate findings
The paper leans on this reformulation to explain two otherwise-separate empirical observations. First, it predicts that dictionary features closer to the 'true' sparse-coding directions should show up as more heavy-tailed, so the paper checks whether its own trained Dictionary features bear this signature by correlating each feature's Autointerpretability score with its activation distribution's skew and kurtosis (see Interpretability-vs-kurtosis/skew correlation analysis). Second, it supplies a reason why Independent Component Analysis (ICA), a technique built to explicitly maximize the non-Gaussianity of its found directions, consistently outperforms Principal Component Analysis (PCA) and the Default (neuron/residual-stream) basis baseline and Random directions baseline as the strongest non-dictionary-learning baseline the paper tests.
Why heavy tails, not variance, mark a sparse direction
Where an $\ell_2$-based method like PCA finds directions that maximize the variance of projected data, $\ell_4$-norm maximization finds directions maximizing $\mathbb{E}[(w \cdot x)^4]$ over unit vectors $w$. A direction whose projected values are mostly near zero with occasional large spikes, exactly the profile produced by a sparse generative code where a feature is usually absent and occasionally strongly present, scores far higher on this criterion than a direction whose projected values are smoothly, normally distributed, because the fourth power disproportionately rewards rare extreme values over many small ones. This is the same intuition captured by kurtosis, $$\mathrm{Kurt}(z) = \frac{\mathbb{E}[(z-\mu)^4]}{\sigma^4}$$ which is large exactly when a distribution is heavy-tailed and peaked rather than Gaussian; $\ell_4$-norm maximization is, in effect, searching directly for high-kurtosis directions, which is why the paper treats measured kurtosis and skew as the observable trace of the property Qu et al. (2019) characterize in closed form.