The pipeline looks like a forward sequence; the causality doesn't run that way
A reward model is supposed to come after the SFT model in the pipeline diagram, yet it quietly reaches backward and helps pick which SFT checkpoint gets used, selection pressure running upstream before the pipeline has produced a single RL reward.
From there an SFT model turns out to be doing two jobs at once inside PPO training, serving as both the starting point and the fixed reference two different checkpoints are quietly used for, before the page closes on a stranger loop: the API prompt distribution InstructGPT is evaluated on was itself shaped by the SFT dataset that came before it.
Several edges show individual artifacts doing more than one job at once, or causality running backward through a pipeline that looks strictly sequential in its own diagram. The reward model back-selects the checkpoint of the SFT model that precedes it, so selection pressure runs backward through the three-step pipeline before the RM ever emits a single RL reward. The SFT model anchors as both init and reference for PPO training, and the two roles are quietly filled by different checkpoints from different training runs. SL-CAI few-shot examples exist only to prevent role confusion in critique-and-revision, keeping one model's critic and reviser identities from blurring together, and SL-CAI itself seeds the policy and comparison data for RL-CAI, doubling as both starting point and preference-model training source. The SFT dataset bootstrapped the existence of the API prompt distribution InstructGPT is now evaluated on, a co-evolved artifact rather than exogenous demand, and the SFT dataset also trains the policies sampled for the RM dataset, closing a loop the KL penalty then relies on to keep the policy inside the RM's competent region. A further edge shows the same backward-running logic in time rather than in pipeline position: the reward model built for the MuJoCo benchmarks is overbuilt in anticipation of novel behavior training, a full two-layer network sized well beyond what the benchmark tasks' own simple quadratic reward functions required, because the authors wanted an architecture that would already generalize to the backflip and one-leg experiments before those experiments had been run. The benchmark result, in other words, was quietly shaped by a capacity decision made for experiments that do not yet exist inside the paper's own narrative order, the same forward-artifact-serving-a-later-need pattern as the SFT dataset bootstrapping the API prompt distribution it will later be evaluated against.