From an open problem to reward models to AI feedback

Members — a guided walk

1. Scalable oversight

   Scalable oversight is named as an open problem in 2016: safe behavior requires evaluating a true objective too costly to check on every training example, forcing reliance on cheaper proxies. This walk now traces how a 2017 paper answers that problem directly, before the answer is scaled twice more, first into a language-model pipeline and then into an AI-fed variant of that same pipeline.

2. Asynchronous Reward-Learning Architecture

   The 2017 paper's asynchronous reward-learning architecture is scalable oversight's first concrete answer: a policy, a human-comparison process, and a reward predictor run as three concurrent loops, each feeding the next, so a learned reward can stand in for the missing objective while the policy trains. Trajectories flow into comparisons and comparisons into reward-predictor training, letting human judgment guide learning without ever evaluating the true objective on every step.

3. Human-Feedback Sample Efficiency

   Human-feedback sample efficiency is the architecture's headline payoff: it makes the answer credible by learning complex tasks and novel behaviors from feedback on less than 1% of the agent's environment interactions, roughly three orders of magnitude less than feedback on every timestep. That efficiency is what turns asynchronous reward learning from a plausible idea into something practically deployable at deep-RL scale.

4. Reward model (RM)

   InstructGPT's (2022) reward model is, functionally, that same architecture scaled to language five years later: a model trained to predict which of two outputs human labelers would prefer, in place of the 2017 paper's video-clip comparisons. InstructGPT never uses the term scalable oversight, so the continuity is visible only in hindsight, but the underlying move, a learned proxy standing in for an objective too costly to check directly, is the one the 2017 architecture already made.

5. Reinforcement learning from human feedback (RLHF)

   RLHF is the fine-tuning technique the reward model exists to serve: its architecture and role are dictated by how policy optimization actually works, since it starts from the same weights as the policy it will later steer and anchors the reward signal RLHF optimizes against. Where the 2017 paper's three processes updated continuously and asynchronously, RLHF alternates reward-model training and policy optimization in discrete rounds, but the underlying loop is the one that architecture first ran end to end.

6. RL from AI Feedback (RLAIF)

   RLAIF swaps the labeler in RLHF, replacing the human who produces preference comparisons with an AI system, while Constitutional AI states plainly that the remainder of the pipeline, preference model training and RL, is exactly the same as RLHF. It is a minimal, surgical modification, not a new algorithm, pushing the same lineage one step further toward automation.

7. Feedback Model

   The feedback model is what RLAIF delegates harmlessness comparisons to: an independent language model shown a harmful prompt and a pair of responses, formatted as an explicit multiple-choice question, that produces the preference label a human labeler, or the 2017 paper's human overseer, would have produced before. It is the concrete mechanism that makes RLAIF's labeler-swap actually work.

8. Hybrid Human/AI Preference Model

   The hybrid human/AI preference model is trained on a mixture the feedback model makes possible: human-labeled helpfulness comparisons stitched together with AI-labeled harmlessness comparisons into a single reward signal. It shows the lineage does not have to be all-human or all-AI, human and machine judgment can be blended in the same trained model.

9. Iterated (Online) Training

   Iterated online training, periodically refreshing the preference model with new comparisons on the current policy's own outputs, is not new at this last step: the 2017 paper's architecture already ran it as a core design choice, collecting comparisons continuously throughout training rather than only at the start. Constitutional AI (2022) flags the same technique as a further scaling step, now newly automatable once an AI rather than a human is doing the labeling, closing the lineage at the point where the labeler itself stops being the bottleneck.

Superthemes

• Scalable oversight made real: from 2016 proposals to InstructGPT's RLHF pipeline

Edges explored within this theme

• Scalable oversight — is retroactively operationalized by → Reward model (RM) hindsight
• Reward model (RM) — anchors the reward signal of → Reinforcement learning from human feedback (RLHF)
• Reinforcement learning from human feedback (RLHF) — swaps the labeler in → RL from AI Feedback (RLAIF)
• RL from AI Feedback (RLAIF) — delegates harmlessness comparisons to → Feedback Model
• Feedback Model — supplies the harmlessness half of → Hybrid Human/AI Preference Model
• RL from AI Feedback (RLAIF) — makes newly automatable → Iterated (Online) Training
• Asynchronous Reward-Learning Architecture — trades its concurrency for a fixed dataset in → Reward model (RM) hindsight

Super edges touching this theme

• Scaling oversight by predicting reward on unlabeled episodes — is selectively realized in → From an open problem to reward models to AI feedback hindsight
• Scaling oversight without a learned reward predictor — prefigures the rule driven labeling in → From an open problem to reward models to AI feedback hindsight
• From an open problem to reward models to AI feedback — is only half operationalized by → InstructGPT's three-step training pipeline
• Keeping the reward model on-distribution — leaves most of its safeguards outside → From an open problem to reward models to AI feedback hindsight

Sources

Every passage behind this theme’s member concepts.