Each pipeline stage's output is the next stage's engineered raw material
A labeling interface nudged toward ties ends up doing quality control before anyone calls it that, filtering out the exact comparisons reward-model training would otherwise have to throw away. Every training pipeline in this corpus is full of small interface, sampling, and filtering choices doing that kind of invisible work.
From there the page traces one loop, an SFT dataset shaping the very policies later sampled into reward-model data, a longer chain unique to one paper where critiques become training data that seeds the next model, and, in the corpus's newest paper, a single generic template that manufactures an entire trait's contrastive data, elicitation questions, and scoring rubric in one call, with a filtering step downstream deciding which of that manufactured material actually gets used.
InstructGPT and Constitutional AI's training pipelines are not neutral conduits; specific interface, sampling, and filtering choices at each stage determine what the next stage can learn, and the 2017 paper's own pipeline runs on the identical logic five years earlier. The trajectory segment supplies the comparison unit for the preference elicitation protocol: rendered as a one-to-two-second clip rather than a full episode or single state, it is engineered to be self-contained enough for a rater to judge even though the two clips in a pair generally start from different states, and its three-way response format, preference, tie, or incomparable, exists specifically to let a rater flag that discontinuity rather than force a guess the raw clip pair cannot support. The SFT dataset trains the policies sampled for the RM dataset, so the reward model's competence is local to the region of output space SFT-style policies actually produce. The labeling interface filters uncertain comparisons out of the RM dataset by encouraging ties that RM training then drops entirely, a confidence filter built into UI affordance rather than post-hoc cleaning. The API prompt distribution supplies label-free scale to the PPO dataset, the cheapest input in the pipeline because everything costly was front-loaded into the model that scores it. Critique-and-revision supplies the finetuning data for SL-CAI, and SL-CAI in turn seeds the policy and comparison data for RL-CAI, while the feedback model supplies the harmlessness half of the hybrid human-AI preference model, stitching two differently-provenanced comparison sets into one scalar reward, and red teaming supplies the transcripts that classifying-harmful-behavior-eval is built directly out of. Persona vectors runs the identical logic a further three years on: trait-artifact generation supplies the raw materials for persona-vector computation, one generic prompt template given only a trait name and description producing three outputs from a single call to Claude 3.7 Sonnet, 5 contrastive instruction pairs, 40 elicitation questions split into a 20-question extraction set and a 20-question evaluation set, and one evaluation rubric; computation consumes only the instruction pairs and the extraction half of the questions, generating 10 rollouts per question under each contrastive instruction for 400 raw responses per trait from that single authored artifact set. The same artifact-generation call also generates the scoring rubric consumed by trait-expression-score, with Claude 3.7 Sonnet instructed to restate the trait description in its own words rather than copy it verbatim, so the rubric GPT-4.1-mini later scores against is one further LLM paraphrase removed from the human-authored description, as the worked evil example, deliberately working against human wellbeing, deriving satisfaction from human pain, shows. That rubric does not wait for final evaluation to matter: persona-vector computation filters its rollouts by trait-expression-score, scoring all 400 rollouts per trait and keeping only those exceeding 50 for positive-prompted responses or falling below 50 for negative-prompted ones, discarding the rest before any activation is averaged. Persona-vector computation then supplies these manufactured contrast sets to the mean-difference method, whose own recipe assumes a clean two-group split as already given; computation manufactures that split through the score filter and through an empirically chosen extraction position, response-token averaging, validated in Appendix A.3 against final-prompt-token and all-prompt-token alternatives as the strongest steering effect across evil, sycophancy, and hallucination. Where 2022's pipeline front-loaded cost into the model that scores PPO's label-free prompts, 2025's front-loads a fixed, trait-agnostic authoring cost into one artifact-generation call and pays a much larger, trait-specific inference cost extracting and filtering exactly the subset of what was authored that mean-difference extraction needs.