2017's ad hoc parts become 2022's industrial ones
Set an early system beside a later one built on the same idea, and the same machine appears twice, once handmade and once industrialized. An unscreened sign-up sheet for contractors becomes a four-criteria screening process; a clip server and a plea for volunteers becomes a purpose-built labeling interface; an assumed error rate becomes a measured agreement statistic.
The walk pairs each early part with its later counterpart, comparison math surviving unchanged, a concurrent architecture trading constant relabeling for a fixed dataset, a pairwise design that never faced a problem batched rankings create, and a segment and normalization step that generalize into what a later paper needed. The same industrialization move repeats a second time, years further on, as a hand-curated steering method's bespoke, per-behavior data curation gives way to one generic template and an automated filtering step a still later paper adds on top.
Read pairwise, the 2017 system and InstructGPT are the same machine built at two different levels of industrialization: nearly every ad hoc part of Deep RL from Human Preferences acquires a formal replacement in InstructGPT, and what carries over unchanged is as informative as what gets replaced. Contractor preference labeling, an unscreened sign-up sheet with no test of rater quality, becomes a screened operation in the labeler screening process, where InstructGPT scores prospective labelers on four separate criteria before letting roughly 40 of them contribute any data; the same ad hoc labeling setup is also formalized as the labeling interface, turning a clip server and a plea to 'ping @tom on slack' into an internal tool that records Likert scores, a fixed metadata taxonomy, and full rankings. The Bradley-Terry model that gives the 2017 reward predictor its comparison loss is carried over unchanged into the InstructGPT reward model loss, the identical softmax-over-summed-score structure now applied to language completions instead of trajectory segments. The async reward learning architecture's three concurrent loops, in which the reward predictor is continually refit throughout training, trade their concurrency for a fixed dataset in the InstructGPT reward model, which is trained once on a static set of comparisons and then held frozen through PPO. The rater error noise model's assumed constant 10 percent error rate is replaced by a measured one: InstructGPT's inter-annotator agreement statistic reports what the 2017 paper could only assume. The preference elicitation protocol's strictly pairwise design meant the 2017 paper never had to face a problem InstructGPT's k-choose-2 batching exists to solve, the overfitting caused by correlated comparisons that come from ranking several outputs in one sitting. Reward normalization's rescaling of the 2017 reward predictor's output to a fixed standard deviation prefigures the scale fix built into the InstructGPT reward model loss, which anchors its own zero point to labeler demonstrations for interpretability rather than optimizer convenience. And the trajectory segment, a one-to-two-second clip chosen to be quick for a human to render and judge, generalizes into the RM dataset's atomic unit, a complete generated text completion, carrying forward the same principle of comparing the smallest self-contained item a human can evaluate. The same industrialization move repeats two years further on, carried by a different pair of papers: CAA's 2023 machinery becomes persona vectors' 2025 raw material at almost every joint. Persona-vector computation generalizes beyond the answer-letter recipe of CAA steering-vector construction: where CAA reads a single, fixed token position, the appended answer letter of a multiple-choice question, from a hand-built dataset of hundreds of contrastive triples curated separately per behavior, computation instead reads an averaged position across every token of a self-generated, free-form rollout, narrowing with an LLM-judge filter, trait score above 50 or below 50, rather than trusting every generated pair by construction, trading a format engineered for trivially clean contrast for one that manufactures cleanliness statistically after the fact. The extraction pipeline built around that computation automates away the bespoke curation contrastive activation addition needed: CAA's seven target behaviors each required their own hand-assembled or GPT-4-generated dataset, a custom hallucination set distinguishing unprompted from contextually-triggered fabrication, a custom refusal set, others drawn from Anthropic's Advanced AI Risk and Sycophancy evaluation sets, hundreds of contrast pairs curated per behavior; the extraction pipeline replaces all of that per-behavior engineering with one fixed prompt template needing only a trait name and description, run identically across seven traits, three main and four in Appendix G, with no behavior-specific dataset construction anywhere in the loop. The pipeline's own closest acknowledged relative is not CAA but Wu et al. (2025): the extraction pipeline adds filtering and validation beyond that concept-description-to-direction pipeline, sharing its core move of manufacturing contrastive data from a description alone but adding an explicit trait-expression-score filter before averaging, an empirically validated choice of extraction position, and a dedicated human-agreement study, 94.7 percent across two raters and 300 comparisons, that Wu et al.'s description does not report. And trait-expression-score succeeds the sampled rating protocol of open-ended generation evaluation on every axis CAA set it up on: CAA's rubric was fixed and hand-written per behavior, scored 1-10 by GPT-4 sampling one output token, its reliability resting on a manual spot check and a general citation rather than a paper-specific agreement figure; trait-expression-score's rubric is instead auto-generated per trait from a name and description, scored 0-100 by GPT-4.1-mini via a logit-weighted sum over candidate tokens, and validated with the same dedicated human study the extraction pipeline relies on. Rubric authorship, score granularity, and validation rigor each acquired an industrial replacement, the same pattern the 2017-to-2022 seam shows, run again on a different pair of years.