Top-K-active baseline control — density matches → Independent Component Analysis (ICA)

The paper identifies a structural asymmetry that could inflate the sparse-autoencoder's apparent advantage: 'a PCA direction is active to an entire half-space on one side of a hyperplane through the origin, whereas a sparse coding feature activates on less than a full direction,' only firing past a hyperplane offset by a negative bias -- so on any given datapoint, far fewer PCA/ICA directions register as simultaneously inactive than dictionary features do. The top-K control directly neutralizes this by restricting ICA (and PCA) to a fixed number K of simultaneously active directions per datapoint, set equal to the average number of active features in a sparse dictionary with R=1 and alpha=8.6e-4 on the layer in question. With activation density equalized this way, the paper finds the adjustment 'does not explain more than a small fraction of the improvement in scores,' ruling out density mismatch as the source of ICA's underperformance and pointing instead to a genuine difference in feature quality.