PPO dataset
Training language models to follow instructions with human feedback — introduced
The set of about 31k API-only prompts, without any human labels, used as rollout inputs for RLHF fine-tuning.
The final stage of InstructGPT's training needs prompts but no answers: the model writes its own responses and the reward model scores them. Some 31,000 unlabeled prompts from the API serve that purpose, making this the cheapest of the pipeline's three datasets.
After placing it among the three data artifacts, the page turns to what that cheapness bought: the economics that made the whole pipeline affordable to run at scale.
How it fits
The PPO dataset is the third and cheapest artifact in InstructGPT's Building the human side of the RLHF pipeline: roughly 31,000 API-sourced prompts, entirely without human labels, used purely as rollout inputs during RLHF fine-tuning (instructgpt, §"3.2 Dataset", p. 7; §"A.3 Dataset sizes", p. 33). It is introduced here as a distinct artifact from the SFT dataset and RM (comparison) dataset that precede it in the pipeline.
What it enabled: the economics of the pipeline
The three training datasets embody a declining gradient of human effort per prompt: SFT prompts require a full written demonstration, RM prompts require a ranking of up to nine outputs, and PPO prompts require nothing at all beyond having been submitted to the API. The API prompt distribution supplies this dataset with label-free scale precisely because by the third pipeline stage the bootstrap problem has already been solved: RL rollouts generate their own training signal via the reward model, so the raw prompt distribution can be used as-is with no labeler-written prompts at all (instructgpt, §"3.2 Dataset", p. 7). This is where the economics of RLHF become visible concretely — 31,000 unlabeled prompts support 256,000 training episodes, roughly eight passes over the prompt set (instructgpt, §"C.4 Details of RLHF training", p. 42), while the expensive human-labeled stages stay capped at 13k demonstrations and 33k rankings. The reward model functions as an amplifier converting a bounded stock of human judgment into effectively unbounded per-rollout feedback, which is exactly why the PPO dataset was deliberately built to be the cheapest input in the pipeline: everything costly was front-loaded into the model that scores it, not into the prompts it scores.