Building the human side of the RLHF pipeline

Members — a guided walk

1. Preference Elicitation Protocol

   The preference elicitation protocol is the primitive ancestor of this whole human-data infrastructure: in 2017, a human overseer watches two rendered video clips of an agent's behavior and records a preference, a tie, or an incomparable judgment into a growing preference database. That same three-way response structure, prefer one side, call it a tie, or admit it can't be judged, still shapes how InstructGPT's labeling apparatus works five years later.

2. Contractor Preference-Labeling Protocol

   The contractor preference-labeling protocol is how that primitive was actually staffed in 2017: paid contractors sign up for time slots, receive a one-to-two sentence description of the task, and repeatedly compare pairs of video clips through a keyboard-driven interface offering prefer-left, prefer-right, tie, and can't-tell responses. It is the first working answer to a question every later stage of this walk answers again at greater scale: who does the labeling, and how are they organized to do it consistently.

3. API prompt distribution

   InstructGPT (2022) scales the 2017 primitive into a purpose-built pipeline, and the API prompt distribution is where that pipeline's data starts: text submitted by external users to an earlier, demonstration-only InstructGPT model on the OpenAI Playground, deduplicated, capped per user, and filtered for personally identifiable information. Every dataset in the rest of this walk draws its raw prompts from here.

4. SFT dataset

   The SFT dataset pairs roughly 13,000 of those prompts with human demonstrations, but the relationship with its source is circular: the Playground traffic behind the API prompt distribution was itself submitted to an earlier model trained on a subset of this same demonstration data, so each partly bootstrapped the other's existence.

5. RM (comparison) dataset

   The RM dataset trains the reward model on roughly 33,000 prompts, each with several outputs ranked by labelers, and those completions come overwhelmingly from policies trained on the SFT dataset just introduced. It marks a step up in labeling effort, from writing one demonstration per prompt to ranking several model outputs against each other.

6. PPO dataset

   The PPO dataset completes the trio at the opposite extreme of labeling effort: roughly 31,000 prompts drawn purely from the API prompt distribution, with no human labels at all, closing a declining gradient of human effort per prompt that runs from full demonstrations, to rankings, to nothing.

7. Labeler screening and selection process

   None of these three datasets exists without the roughly 40 contractors selected through this four-criterion screening process, covering agreement on sensitive-speech flagging, agreement on rankings, sensitive demonstration writing quality, and self-assessed ability to identify sensitive content across groups. The walk now turns from the data itself to the human infrastructure that produced it.

8. Inter-labeler agreement rate

   Inter-labeler agreement enters this pipeline on both sides of the screening process just described: two of its four criteria are agreement measures used to filter candidates in, and the same statistic is later recomputed as an audit of the labelers who made it through.

9. Web labeling interface

   The web labeling interface is where screened labelers actually record their judgments, assigning Likert scores, metadata labels, and full rankings; because it collects rankings of four to nine completions per prompt rather than isolated pairs, it is what filters uncertain comparisons out of the RM dataset introduced earlier.

10. Labeler metadata annotation taxonomy

    The labeler metadata annotation taxonomy is the fixed set of fields that interface asks labelers to fill in for every rated output, binary and Likert-scale flags for failures like hallucination or denigration of a protected class, turning each labeling session into structured data rather than just a ranking.

11. Held-out labeler generalization test

    The held-out labeler generalization test is the question the screening process creates the need to answer: because every training label came from the same roughly 40 screened contractors, this test checks whether models trained on their preferences hold up for a separate group of labelers who contributed no training data at all. It closes the walk by auditing the trustworthiness of everything built on top of this human infrastructure, all the way back to the simple three-way judgment the 2017 protocol first asked a person to make.

Superthemes

• Scalable oversight made real: from 2016 proposals to InstructGPT's RLHF pipeline

Edges explored within this theme

• SFT dataset — bootstrapped the existence of → API prompt distribution
• SFT dataset — trains the policies sampled for → RM (comparison) dataset
• Labeler screening and selection process — creates the question answered by → Held-out labeler generalization test
• Inter-labeler agreement rate — was both filter and audit for → Labeler screening and selection process
• Web labeling interface — filters uncertain comparisons out of → RM (comparison) dataset
• API prompt distribution — supplies label free scale to → PPO dataset
• Contractor Preference-Labeling Protocol — becomes a screened operation in → Labeler screening and selection process hindsight
• Contractor Preference-Labeling Protocol — is formalized as → Web labeling interface hindsight
• Constant Rater-Error Noise Model — assumes a constant later measured by → Inter-labeler agreement rate hindsight
• Preference Elicitation Protocol — avoids the overfitting problem solved by → K-choose-2 single-batch comparison training hindsight

Super edges touching this theme

• Building the human side of the RLHF pipeline — exposes the noisy consensus beneath → InstructGPT's three-step training pipeline
• Comparisons as the atomic unit of human feedback — assumes the error rate later measured in → Building the human side of the RLHF pipeline hindsight
• The economics of human feedback — supplies the cost argument behind → Building the human side of the RLHF pipeline hindsight

Sources

Every passage behind this theme’s member concepts.