Keeping the reward model on-distribution

Members — a guided walk

1. Asynchronous Reward-Learning Architecture

   The asynchronous reward-learning architecture is the design in the 2017 paper that makes every other countermeasure in this theme possible: a policy, a human-preference-collection process, and a reward predictor run as three concurrent loops, with trajectories flowing to comparisons, comparisons flowing to the predictor, and the predictor's updated parameters flowing back to the policy. Because the predictor keeps refitting while the policy keeps acting, the reward it produces is a moving target rather than a fixed function.

2. Iterated (Online) Training

   Iterated (online) training is the discipline that architecture enables: preference labels are collected continuously throughout training rather than only at the start. The paper's own ablation shows this matters, since training only on early labels lets the policy learn to exploit gaps in a reward predictor that never gets to see where the policy goes next.

3. Non-Stationary Reward Function Challenge

   The non-stationary reward function challenge names the cost of that same choice: because the reward predictor is repeatedly refit during training, the policy is effectively solving a reinforcement learning problem whose reward keeps changing underneath it. The paper responds by favoring policy-gradient methods that tolerate a shifting reward and by raising the entropy bonus to preserve exploration.

4. Reward Predictor Ensemble

   The reward predictor ensemble is one concrete safeguard against a predictor that overfits before it has caught up with the policy: three independently-trained predictors, each fit on a bootstrap resample of the preference data, with the final estimate given by averaging their independently-normalized outputs. Disagreement across the ensemble is also what later drives which comparisons get queried at all.

5. Adaptive L2 Regularization for the Reward Predictor

   Adaptive L2 regularization for the reward predictor is the ensemble's companion safeguard: the L2 penalty coefficient on each predictor is adjusted during training to keep held-out validation loss between 1.1 and 1.5 times training loss, using a held-out fraction of 1/e of the preference database. Where the ensemble spreads risk across multiple fits, this scheme tunes each individual fit against its own overfitting.

6. Reward Predictor Pretraining

   Reward predictor pretraining addresses the earliest moment this non-stationarity could go wrong: in the Atari experiments, the reward predictor trains for 200 epochs on comparisons drawn from an untrained, randomly-initialized policy before RL begins at all. This reduces the risk that the policy converges irreversibly on a bad behavior while being guided by a predictor that has not yet learned anything useful.

7. Label Annealing

   Label annealing governs how the query rate changes once RL training is under way: the rate at which new preference labels are requested decreases roughly inversely with elapsed timesteps, so the predictor gets dense feedback early, when pretraining alone is not enough, and progressively sparser feedback later, once it has mostly caught up with the policy.

8. Reward Prediction Normalization

   Reward prediction normalization is a smaller but necessary fix downstream of all this refitting: the predictor's output is rescaled to zero mean and a fixed standard deviation before the RL algorithm uses it, since a comparison-only training objective leaves the reward's absolute scale and offset undetermined.

9. Recurrent Reward Model (Proposed Extension)

   The recurrent reward model extension closes this theme with the general case the paper's actual architecture only approximates: rather than estimating reward from a fixed window of recent observations, as the real experiments do, a recurrent network could model reward as a function of an agent's whole observation history in a partially observable environment. The paper flags this as an unimplemented extension, leaving open how much further off-distribution drift such a model would need to withstand.

Superthemes

• Deep RL from Human Preferences: the 2017 proof of concept
• Scalable oversight made real: from 2016 proposals to InstructGPT's RLHF pipeline

Edges explored within this theme

• Reward Predictor Ensemble — supplies the disagreement signal for → Uncertainty-Based Query Selection (Ensemble Variance)
• Iterated (Online) Training — is the cause of → Non-Stationary Reward Function Challenge
• Adaptive L2 Regularization for the Reward Predictor — bounds the generalization gap of → Reward model (RM)
• Reward Predictor Pretraining — solves the cold start problem of → Asynchronous Reward-Learning Architecture
• Label Annealing — paces the query rate within → Iterated (Online) Training
• Reward Prediction Normalization — fixes the underdetermined scale of → Reward model (RM)
• Iterated (Online) Training — guards against → Reward hacking (avoiding)
• Asynchronous Reward-Learning Architecture — gives concrete shape to → Reinforcement learning from human feedback (RLHF)
• Recurrent Reward Model (Proposed Extension) — would generalize the fixed window of → Reward model (RM)
• Pong (Atari game) — anticipates → Reward model over-optimization hindsight
• Reward Prediction Normalization — prefigures the scale fix in → Reward model pairwise comparison loss hindsight

Super edges touching this theme

• Keeping the reward model on-distribution — shares non stationarity with → Policy optimization under a learned, changing reward
• Keeping the reward model on-distribution — trades its online refresh for a kl penalty in → The mechanics that make PPO fine-tuning work hindsight
• Keeping the reward model on-distribution — leaves most of its safeguards outside → From an open problem to reward models to AI feedback hindsight

Sources

Every passage behind this theme’s member concepts.