Principal Component Analysis (PCA) — gets more training data than → Independent Component Analysis (ICA)

The paper's two classical baselines are not fit on equal amounts of data: PCA is estimated online 'on the same quantity of data we used for training the autoencoders' (the paper's full activation datasets, 5-50M vectors), while ICA, run on only 2GB of data, about 4 million residual-stream activations or 1 million MLP activations, is deliberately given far less, explicitly 'due to the slower convergence times' of ICA algorithms at scale. This means the interpretability-score comparison between the two baselines is not apples-to-apples on data budget alone, which is worth knowing before reading too much into ICA narrowing the gap with sparse autoencoders in later layers: ICA is doing comparably well while working from a noticeably smaller sample.