The ablation is the argument
The 2017 paper rarely defends a design choice with a first-principles argument; it removes the choice and shows the method breaking in some specific way. Strip the reward-predictor ensemble and query selection has nothing left to disagree over; drop the regularization term and only dropout holds the model steady; freeze the predictor mid-training and the Pong result is a policy that stops losing without ever learning to score.
The walk moves through five such removals in turn, arriving at the paper's own admission that its query-selection heuristic is a crude stand-in for a method it never built, one its ablations show can lose to picking pairs at random, and then, six years later, at a 2023 paper running the same kind of argument over a single hyperparameter, sweeping every network layer to justify, empirically rather than theoretically, where its own intervention should go.
The 2017 paper backs up nearly every major design decision not with a first-principles argument but with an ablation that removes the choice and shows what happens when it is gone. The reward predictor ensemble supplies the disagreement signal that uncertainty-based query selection needs to pick which pairs to query; the paper's own ablation removes the ensemble, leaving query selection nothing to disagree over and forcing it to fall back on picking pairs at random. Adaptive L2 regularization keeps the reward model's validation loss inside a fixed band above training loss as the comparison dataset grows; a paired 'no regularization' ablation drops the term and relies on dropout alone, isolating the adaptive scheme as a distinct safeguard. The comparisons-vs-absolute-scores ablation swaps in mean-squared-error regression against an oracle's true score in place of the Bradley-Terry comparison loss, and finds comparisons win clearly on continuous control but not consistently on Atari, complicating rather than simply vindicating the choice of Bradley-Terry. Iterated online training's necessity is demonstrated by removing it: freezing the reward predictor after an initial batch of labels produces the paper's own Pong reward-hacking example, an agent that avoids losing points but never learns to score. Clip-length effects calibrates the length of the trajectory segment, and the 'no segments' ablation collapses a segment to a single frame, producing large performance drops that show segment duration is carrying real information rather than just amortizing overhead. Uncertainty-based query selection is conceded by its own authors to be a crude approximation of the more principled expected-value-of-information query selection they leave to future work, and their ablations show the crude version can sometimes perform worse than querying at random. CAA runs an analogous sweep six years later, over a hyperparameter rather than a design choice already made: multiple-choice behavioral evaluation supplies the layer at which the steering vector is added, scoring every layer's effect on 50 held-out questions per behavior with the multiplier held fixed, and converging on a narrow answer, layer 13 or its near neighbors in the 7B model, layer 14 or 15 in the 13B model, consistent across every behavior tested. As with the 2017 paper's ablations, the argument is empirical rather than derived, made by sweeping the choice rather than reasoning from theory about which layer should matter. But the search stays sequential rather than joint: only once the layer is fixed does the paper tune multiplier magnitude, watching open-ended text quality degrade rather than rerunning the multiple-choice sweep, so where the 2017 paper isolates one design choice per ablation, CAA's two hyperparameters are each settled by a different instrument instead of a single joint search.