Top-and-random vs. random-only interpretability scoring — shapes fragment sampling in → Autointerpretability score

The autointerpretability pipeline's entire scoring budget for one feature is small (10 fragments of 64 tokens, 640 tokens total), and most dictionary features fire on only a tiny fraction of tokens in the corpus. Appendix A spells out why this forces a top-and-random default rather than uniform sampling: 'a random sample will likely not contain any highly activating examples for all but the most common features,' so a purely random draw would give GPT-3.5 nothing but near-zero activations to correlate against, making the score uninformative for anything but the most frequently-firing features. Mixing in fragments already known to activate the feature guarantees the simulator sees the feature in action at least once, at the cost of making the comparison less like a blind test and more like an open-book one. The paper treats this as a real methodological compromise rather than a free lunch: Appendix C.2 reruns everything with random-only scoring specifically to check how much top-random inflates the numbers.