Autointerpretability score — prefigures the llm judge pattern of → Open-ended generation evaluation with GPT-4 rating

Both papers solve an evaluation-at-scale problem with the same trick: hand the judging to a capable language model instead of a human, because there is too much to grade by hand. The sparse-autoencoder paper’s autointerpretability score asks GPT-4 to read activating text fragments for a dictionary feature and write a natural-language explanation, then asks GPT-3.5 to use that explanation to predict the feature’s activation on held-out text, scoring the feature by how well predicted and actual activations correlate — a procedure run across "thousands of dictionary features" that no team of human annotators could read individually (sparse-autoencoders, §"3.1 Interpretability at scale", p. 3). Contrastive Activation Addition’s open-ended generation evaluation, published three months later, hands the identical model the same kind of job at the opposite end of the pipeline: rather than explaining what an internal feature detects, GPT-4 reads a free-form generated response and scores it 0-10 against a rubric for how strongly it displays a targeted behavior (contrastive-activation-addition, §"4.2 Open-ended generation", p. 4). Both papers hedge the same worry rather than treating LLM judgment as free of cost: the SAE paper notes that "current LLMs are limited in the kinds of patterns that they can find" as explainers (sparse-autoencoders, §"3.2 Sparse dictionary features are more interpretable than baselines", p. 5), while CAA’s limitations section flags that GPT-4’s open-ended scores "can be sensitive to rubric wording" and "may carry systematic bias," mitigated only by a manual spot-check rather than a formal inter-rater statistic (contrastive-activation-addition, §"10 Limitations", p. 10). The SAE paper could not have framed its autointerpretability score as an instance of a recurring "LLM-as-judge" pattern, since CAA’s own use of the same model for a structurally similar job did not exist yet; the parallel is visible only once the later paper supplied its own half of it.