← Explorean AI safety wiki themeHelp

Small open models as interpretability testbeds

Every interpretability result in this paper is scoped by the size of what it was tested on, and the authors chose small and open over large and proprietary. A full training run finishes on a single GPU in under an hour.

This theme covers the infrastructure that choice buys: a fully open model family, a public training corpus, and a separate public source for the text fragments used to explain features, cheap enough that sweeping many settings is affordable, and modest enough that the paper is explicit about being a proof of concept rather than a claim about frontier models.

Cunningham et al. (2023) train and evaluate their sparse autoencoders on modest, fully open infrastructure: Pythia-70M and Pythia-410M, an open, reproducible model suite with residual-stream dimensions of 512 and 1024 respectively, trained on the Pile, the public webtext corpus also used to measure reconstruction-substitution perplexity. Autointerpretability text fragments are drawn from OpenWebText, another public corpus, sampled from its first 50,000 lines. The choice keeps every stage of the pipeline small and open: a full training run completes in under an hour on a single A40 GPU, cheap enough that the paper can afford to sweep dictionary-size ratios and sparsity coefficients rather than report a single configuration. The scale is explicitly a proof of concept rather than a claim about frontier models, the same honesty about scope that the 2017 Deep RL from Human Preferences paper's own small testbed environments show for a different method.

Members — a guided walk

  1. Pythia (model suite)

    Cunningham et al. (2023) run their sparse autoencoders on Pythia-70M and Pythia-410M, an open, reproducible suite of language models (Biderman et al., 2023) whose residual streams are 512- and 1024-dimensional respectively. Working with a fully open model suite means every later claim in the paper -- about reconstruction quality, interpretability scores, or causal precision -- can be checked directly against public weights, rather than resting on models only the paper's own authors can access. Training the autoencoders themselves requires activations to train on, drawn from another open resource: the Pile.

  2. The Pile

    The Pile (Gao et al., 2020) is the large public webtext corpus whose text the paper runs through Pythia to build its training sets: 5-50M activation vectors per autoencoder, trained for 1-3 epochs with the Adam optimizer. The same corpus doubles as the paper's fidelity check, the reconstruction-substitution perplexity metric, where swapping in the SAE's reconstructed layer-2 activations for Pythia-70M raises Pile perplexity from 25 to 40. A separate, smaller public corpus supplies the text used for scoring features rather than training them: OpenWebText.

  3. OpenWebText

    OpenWebText supplies the text fragments for the autointerpretability protocol: the first 50,000 of its lines are each cut into a 64-token fragment, on which a feature's activations are measured before GPT-4 writes an explanation and GPT-3.5 simulates it for scoring. Keeping this corpus public, like the models and training data before it, means the paper's interpretability scores can be reproduced from openly available pieces, with GPT-4 and GPT-3.5 the only closed components in the whole pipeline. A full autoencoder training run under this setup completes in under an hour on a single A40 GPU, cheap enough that the paper affords wide hyperparameter sweeps rather than a single committed configuration -- an explicit proof of concept rather than a claim about frontier models, the same honesty about scope that the 2017 Deep RL from Human Preferences paper shows for its own small testbed environments elsewhere in the corpus.

Superthemes

Super edges touching this theme

Sources

Every passage behind this theme’s member concepts.

3 source passages in 1 paper