Independent Component Analysis (ICA) — converges with → Sparse autoencoders (SAEs)

ICA is not a uniformly weaker baseline: the paper reports that the sparse-autoencoder advantage on autointerpretability score 'declines as we move through the model, being comparable to ICA in layer 4 and showing minimal improvement in the final layer.' That is, the gap that clearly favors dictionary features in early layers largely disappears by the last layer studied, where ICA catches up. The paper offers two possible explanations rather than picking one: sparse autoencoders may simply work less well on later-layer representations, or later-layer features may be intrinsically harder for the autointerpretability pipeline to score well, since by that point they are 'often best explained by their effect on the output' rather than by input tokens, which the GPT-4/GPT-3.5 pipeline is not set up to check.