InstructGPT's three-step training pipeline

Members — a guided walk

1. InstructGPT

   InstructGPT names the family of GPT-3 models produced by a specific three-step training procedure, and this walk follows that procedure end to end, from the base model it starts with through to the reinforcement-learning stage that finishes it.

2. GPT-3 (pretrained language model)

   GPT-3 is the pretrained autoregressive language model that supplies both the architecture and the initialization for every model in the pipeline. Despite sharing GPT-3's weights as a starting point, InstructGPT models outperform it at 100 times smaller scale, a controlled result since fine-tuning on the human data pipeline is the only thing that differs.

3. Supervised fine-tuning (SFT)

   Supervised fine-tuning is the first of the three steps: the pretrained GPT-3 model is fine-tuned directly on human-written demonstrations of desired behavior using standard supervised learning, no reward model or reinforcement learning involved yet.

4. Supervised fine-tuned (SFT) model

   The SFT model is what that first step produces, and it does double duty from here on: it serves as a standalone baseline for comparison, and, as the next two steps show, as the very foundation the reward model and PPO training are both built on.

5. Reward model (RM)

   The reward model is initialized from the SFT model itself, with its unembedding layer replaced by a scalar output head, and trained to predict which of two outputs human labelers would prefer. Selection pressure actually runs backward through the pipeline here: the SFT checkpoint used was chosen partly because later epochs kept improving the reward model's own downstream score.

6. Reward model pairwise comparison loss

   The reward model is trained with a pairwise comparison loss where the reward gap between preferred and dispreferred completions represents the log odds a human would prefer the former. Because that loss only ever sees differences between rewards, it under-determines the model's absolute scale, producing a ranker whose raw output is meaningless on its own, a detail that matters once the ranker is reused as an RL reward.

7. PPO fine-tuning stage (RLHF policy optimization)

   PPO fine-tuning is the third step, further tuning the SFT model with reinforcement learning to maximize the reward model's score; the SFT model anchors both where this optimization begins and, via a KL term, what it is pulled back toward. Every PPO policy here, even the largest, is supervised from below by a reward model roughly 30 times smaller than the policy it steers.

8. Reinforcement learning from human feedback (RLHF)

   Reinforcement learning from human feedback is the name for this whole three-step procedure taken together: a learned model of human preference used as a reward signal for reinforcement learning, applied to open-ended instruction following rather than a single fixed task. It is the umbrella this entire walk has been building toward, one term for GPT-3 plus demonstrations plus a reward model plus PPO.

Superthemes

• Scalable oversight made real: from 2016 proposals to InstructGPT's RLHF pipeline

Edges explored within this theme

• InstructGPT — outperforms at 100x smaller scale → GPT-3 (pretrained language model)
• Reward model (RM) — back selects the checkpoint of → Supervised fine-tuned (SFT) model
• Supervised fine-tuned (SFT) model — anchors as both init and reference for → PPO fine-tuning stage (RLHF policy optimization)
• Reward model pairwise comparison loss — under determines the scale of → Reward model (RM)
• Reward model (RM) — supervises from below → PPO fine-tuning stage (RLHF policy optimization)

Super edges touching this theme

• The mechanics that make PPO fine-tuning work — reveals a critic reward coupling inside → InstructGPT's three-step training pipeline
• Building the human side of the RLHF pipeline — exposes the noisy consensus beneath → InstructGPT's three-step training pipeline
• From an open problem to reward models to AI feedback — is only half operationalized by → InstructGPT's three-step training pipeline
• Comparisons as the atomic unit of human feedback — scales binary comparison into rankings for → InstructGPT's three-step training pipeline hindsight
• Three levers on one model: prompting, finetuning, steering — adds a fourth lever after → InstructGPT's three-step training pipeline hindsight

Sources

Every passage behind this theme’s member concepts.