Adaptive L2 Regularization for the Reward Predictor
Deep Reinforcement Learning from Human Preferences — introduced
A regularization scheme in which the l2 penalty coefficient on the reward predictor is adaptively adjusted during training to keep held-out validation loss between 1.1 and 1.5 times training loss, with 1/e of the preference data held out as a validation set; dropout is additionally used in some domains.
A regularization strength fixed once at the start of training can quietly become wrong as training continues, too loose early or too tight later. Here the penalty on the reward predictor isn't fixed at all; it moves in response to how the model performs on data it hasn't seen.
The page describes how the coefficient is tuned against a held-out slice of preference data to keep validation loss within a fixed band above training loss, then turns to why a reward model trained on a constantly shifting policy needs this kind of adaptivity more than an ordinary classifier would.
Tuning the penalty against held-out loss
Rather than fixing an l2 penalty coefficient in advance, the paper adjusts it continuously during training against a held-out slice of the reward predictor's own data. A fraction of 1/e of the preference database $D$ is set aside as a validation set for each ensemble member, and the regularization coefficient is adjusted to keep validation loss between 1.1 and 1.5 times training loss (deep-rl-human-prefs, §"2.2.3 Fitting the Reward Function", p. 5). In some domains dropout is applied as well, alongside this l2 term (deep-rl-human-prefs, §"2.2.3 Fitting the Reward Function", p. 5).
Why the reward model needs this more than a typical classifier does
The Reward model (RM) is unusually exposed to distribution shift: it trains on comparisons of clips from a policy that keeps changing as training proceeds, so a fixed regularization strength chosen once at the start could become too weak or too strong as the policy's behavior — and the query distribution it induces — shifts. Keeping validation loss in a fixed ratio to training loss throughout is the paper's way of bounding the reward model's generalization gap adaptively rather than gambling on a hyperparameter fixed before the data reflects the policy's actual trajectory distribution. The "no regularization" ablation removes the l2 term and relies on dropout alone (deep-rl-human-prefs, §"3.3 Ablation Studies", p. 9).