Interpretability itself becomes a benchmark

Members — a guided walk

1. Autointerpretability score

   The autointerpretability score, adapted from Bills et al. (2023), is the paper's primary measure of whether a dictionary feature means anything: a language model is shown text where the feature activates, writes a natural-language explanation of it, and a second language-model pass predicts the feature's per-token activations from that explanation alone. The correlation between predicted and true activations is the score, and Cunningham et al. compute it across thousands of features and four baselines. Two different language models carry out the two halves of that pipeline, starting with the one that writes the explanation.

2. GPT-4

   GPT-4 is the explainer: given a dictionary feature's highly-activating text fragments and their rescaled per-token activations, it writes the natural-language description of what the feature detects, such as 'instances of the letter W and words beginning with w' for one layer-1 residual-stream feature. Table 1 in the paper shows five such GPT-4-written descriptions alongside their scores, ranging from -0.11 to 0.57 for the first five features in that dictionary. GPT-4's explanation is only half the pipeline; a second model has to test whether that explanation actually predicts the feature's behavior.

3. GPT-3.5

   GPT-3.5 is that second model, the simulator: given GPT-4's explanation, it predicts the feature's per-token activations on held-out text, and those predictions are correlated against the ground truth to produce the score. Cunningham et al. substitute GPT-3.5 for GPT-4 at this step, departing from Bills et al.'s original protocol, purely because OpenAI's public GPT-3.5 API exposed the logprobs the simulation step needs while its GPT-4 API did not at the time. Which text fragments get shown to these two models is itself a protocol choice with consequences for the score.

4. Top-and-random vs. random-only interpretability scoring

   Top-and-random vs. random-only interpretability scoring names that choice: the default 'top-and-random' scoring mixes highly-activating fragments with randomly sampled ones, since with only 640 tokens of context a purely random sample would rarely contain a highly-activating example for anything but the most common features. Appendix C.2 reruns the whole comparison using stricter random-only scoring and finds the learned dictionary features still outperform every baseline, especially in early layers, ruling out top-scoring as an artifact explaining the paper's results. A second, independent control checks a different possible artifact, this time in how the baselines themselves are set up.

5. Top-K-active baseline control

   The top-K-active baseline control, run in Appendix G, addresses an asymmetry: a PCA direction is active across an entire half-space, while a sparse-coding feature activates only past a learned bias threshold, so the PCA and ICA baselines get restricted to a fixed number K of simultaneously active directions per datapoint, set equal to the SAE's own average active-feature count at R=1 and alpha=8.6e-4. This handicap explains only a small fraction of the interpretability gap between the learned dictionary and its baselines, meaning sparsity of activation alone does not account for the advantage. What does help explain both the size of that advantage and which baseline comes closest to closing it is a property of the activation distributions themselves.

6. Interpretability-vs-kurtosis/skew correlation analysis

   The interpretability-vs-kurtosis/skew correlation analysis correlates a feature's autointerpretability score with statistical moments of its activation distribution, finding weak positive correlations of 0.19 with skew and 0.24 with kurtosis. Heavier-tailed features, in other words, tend to score as more interpretable, an intuitive fit with the idea that interference from other active features should be roughly normally distributed by the central limit theorem, leaving genuine features to stand out as outliers. This is also the analysis that explains ICA's position among the four baselines: because ICA directly maximizes non-Gaussianity, it lands closest to the SAE's scores of any baseline tested, closing the loop between what this pipeline measures and why some methods score better on it than others.

Superthemes

• Decomposing the model into features

Edges explored within this theme

• GPT-4 — generates explanations for → Autointerpretability score
• GPT-3.5 — simulates activations for → Autointerpretability score
• Top-and-random vs. random-only interpretability scoring — shapes fragment sampling in → Autointerpretability score
• Top-K-active baseline control — tests a separate confound from → Top-and-random vs. random-only interpretability scoring
• Interpretability-vs-kurtosis/skew correlation analysis — explains variation in → Autointerpretability score
• Interpretability-vs-kurtosis/skew correlation analysis — supplies empirical evidence for → Sparse dictionary learning / sparse coding
• Reconstruction-substitution perplexity metric — measures a different axis than → Autointerpretability score
• Dictionary feature — scores unevenly across depth in → Autointerpretability score
• Monosemanticity — is imperfectly proxied by → Autointerpretability score
• Top-and-random vs. random-only interpretability scoring — is vindicated by → Interpretability-vs-kurtosis/skew correlation analysis

Super edges touching this theme

• Unmixing activations with a sparsity penalty — hands off only what fires to → Interpretability itself becomes a benchmark
• The ladder of decomposition baselines — ranks every rung through → Interpretability itself becomes a benchmark
• Interpretability itself becomes a benchmark — passes the burden of proof to → Features proved by intervention
• Small open models as interpretability testbeds — outsources its grading to → Interpretability itself becomes a benchmark

Sources

Every passage behind this theme’s member concepts.