API prompt distribution
Training language models to follow instructions with human feedback — introduced
The paper's primary prompt source: text prompts submitted by external users to an earlier, demonstration-only InstructGPT model on the OpenAI Playground, deduplicated, capped per user, and filtered for personally identifiable information.
Most of what InstructGPT trained on was written by strangers: real requests that customers typed into an early version of the model on OpenAI's Playground. Before use, the prompts were deduplicated, capped per user, and scrubbed of personal information.
Where the prompts came from matters less than what follows: the resulting distribution co-evolved with the model rather than sampling anything neutral. From there, it is worth seeing how one stream of prompts ended up funding three separate datasets.
Where the prompts came from
The API prompt distribution is InstructGPT's primary source of training and evaluation prompts: text that real customers submitted to an earlier, demonstration-only InstructGPT model hosted on the OpenAI Playground interface (instructgpt, §"3.2 Dataset", p. 6). Prompts are heuristically deduplicated by long common prefix, capped at 200 per user ID, filtered for personally identifiable information in the training split, and divided into train/validation/test sets along user-ID boundaries so that no evaluation user's data ever appears in training (instructgpt, §"3.2 Dataset", p. 6). Because instruction-style prompts were scarce on the regular GPT-3 API early in the project, labelers supplemented this pool by writing prompts themselves in three styles, plain, few-shot, and use-case-based, to bootstrap an initial sense of what an "instruction" looks like (instructgpt, §"3.2 Dataset", p. 7; §"A.2 API user prompts", p. 26).
A co-evolved distribution, not a neutral sample
The distribution looks like exogenous demand, but it is partly an artifact of the pipeline it now feeds, as the SFT dataset — bootstrapped the existence of → API prompt distribution edge lays out: users learned to write instruction-style prompts by interacting with a model that labelers had already taught, via Supervised fine-tuning (SFT), what an instruction looks like. Demonstrations shaped a model, the model shaped user behavior, and that behavior became the very distribution InstructGPT is trained and evaluated on. This matters for how the corpus should be read: results on "the API distribution" describe a system's fit to a population it helped create, not a fixed, independent sample of user intent.
Funding three datasets at a declining cost
This distribution is the shared raw material behind all three training sets that the theme Building the human side of the RLHF pipeline documents: the roughly 13k-prompt SFT dataset, the roughly 33k-prompt RM (comparison) dataset, and the roughly 31k-prompt PPO dataset. As the API prompt distribution — supplies label free scale to → PPO dataset edge details, human effort per prompt declines sharply across the three: a full written demonstration for SFT, a ranking of up to nine outputs for the reward model, and nothing at all for PPO, whose 31k unlabeled prompts support 256k training episodes, roughly eight passes over the prompt set (instructgpt, §"C.4 Details of RLHF training", p. 42). The Reward model (RM) is what makes the last stage cheap: it converts a bounded stock of human judgment collected earlier into unbounded per-rollout feedback, so the API prompt distribution can be used at PPO scale precisely because everything costly was front-loaded into the model that scores it.