Keeping the reward model on-distribution — leaves most of its safeguards outside → From an open problem to reward models to AI feedback

The lineage theme already tracks one of this theme's five concepts forward: iterated online training survives, in name, as a mechanism the 2017 architecture actually runs, and one Constitutional AI later names as a further scaling step made easier once an AI rather than a human is doing the labeling (constitutional-ai, §"6.1 Future Directions", p. 15). But the theme documents four other countermeasures against reward-model drift, the reward predictor ensemble, adaptive L2 regularization, reward prediction normalization, and label annealing, and none of them has any afterlife in either successor architecture: InstructGPT's reward model is a single scalar-head network trained once on ranked comparisons, not an ensemble (instructgpt, §"3.5 Models", p. 8), and Constitutional AI's feedback model is a single language model producing a preference label from one sampled principle at a time (constitutional-ai, §"4.1 Method", p. 10). Even the one concept that does carry forward changes character in the carrying: 2017 implements it as a running loop inside one system in continuous operation, while Constitutional AI names it only as future work, not something either of its own trained models does yet. Read through this theme, the lineage from open problem to reward model to AI feedback is less a relay of a maturing toolkit than a single idea, keep the evaluator adapting, proposed once, built once, and re-proposed as unfinished business, while the four safeguards that made that idea workable at 2017's scale were left behind entirely.