Adaptive L2 Regularization for the Reward Predictor — bounds the generalization gap of → Reward model (RM)

Rather than fix an l2 penalty coefficient in advance, the paper tunes it continuously against a held-out slice of the reward model's own training data: 1/e of the preference database is set aside as validation, and the regularization coefficient is adjusted so validation loss stays between 1.1 and 1.5 times training loss (deep-rl-human-prefs, §"2.2.3 Fitting the Reward Function", p. 5). This keeps the reward model's fit tethered to a fixed, small band of acceptable overfitting rather than letting the penalty go stale as the preference database keeps growing through iterated online training. The paper's 'no regularization' ablation removes the l2 term and relies on dropout alone (deep-rl-human-prefs, §"3.3 Ablation Studies", p. 9), isolating this adaptive scheme as a distinct safeguard rather than a redundant one. Because the reward model's only training signal is a comparison loss with no absolute scale, a coefficient that under- or over-penalizes would either let the model memorize idiosyncratic comparisons or wash out the reward differences it needs to preserve.