Dictionary feature — scores unevenly across depth in → Autointerpretability score

Being autointerpretability-scored well is not evenly distributed across a dictionary: Section 3.2 finds the SAE's advantage over baselines like PCA, ICA, and the default basis shrinks as depth increases, becoming merely 'comparable to ICA' by layer 4 and showing 'minimal improvement' by the model's final layer. The paper offers two candidate reasons tied to what dictionary features actually are at each depth: later-layer features build on many earlier-layer features and so may encode more complex, composite concepts, and later features are often best explained by their effect on the model's output rather than by the input text the autointerpretability protocol relies on. That second point is a real gap in the metric, not just a hard case: the paper has a method (Section 5's ablation analysis) for reading a feature's effect on output logits, but notes 'we currently do not send this information to the language model for hypothesis generation,' so GPT-4 is asked to explain late-layer features using exactly the kind of evidence, input text, least suited to them. 'A dictionary feature that scores well' is therefore layer-dependent almost by construction: early features are graded on terms the protocol is good at, late features on terms it is not.