Comparisons vs. Absolute Scores (Target Ablation)
Deep Reinforcement Learning from Human Preferences — introduced
An ablation ('target') replacing the comparison-based cross-entropy loss with direct regression of the reward predictor to an oracle's true total reward over a segment via mean squared error, testing whether eliciting comparisons rather than absolute scores is important to the method's success.
Asking a person which of two things is better is often easier than asking them to put a number on either one. This ablation tests whether that difference actually matters, by swapping the paper's comparison-based loss for plain regression to a true score.
The page describes how the ablation replaces the loss with mean-squared-error regression against an oracle's reward, then goes through the mixed, task-dependent results, a clear win for comparisons on continuous control, no consistent winner on Atari.
The ablation: regression against an oracle instead of comparisons
This ablation ("target") replaces the paper's comparison-based cross-entropy loss with direct regression: an oracle supplies the true total reward over a trajectory segment, and the predictor is fit to that value using mean squared error instead of the Bradley-Terry loss over pairwise judgments (deep-rl-human-prefs, §"3.3 Ablation Studies", p. 9). It is a direct test of whether eliciting comparisons, rather than absolute scores, is actually load-bearing for the method's results.
A mixed, task-dependent answer
The authors' stated motivation was that they found it "much easier for humans to provide consistent comparisons than consistent absolute scores," especially on continuous-control and qualitative tasks (deep-rl-human-prefs, §"3.3 Ablation Studies", p. 10). The ablation's results complicate any simple story, however: on continuous-control tasks comparisons clearly outperformed score regression, likely because varying reward scales make direct regression harder while comparisons sidestep that difficulty; on Atari, where rewards were clipped and effectively reduced to a sign, comparisons and score targets performed differently across games with neither consistently better (deep-rl-human-prefs, §"3.3 Ablation Studies", p. 10). The ablation does not deliver a clean verdict in favor of comparisons — it shows the choice matters, and matters differently depending on the domain.