Reward model (RM) — supervises from below → PPO fine-tuning stage (RLHF policy optimization)

Every PPO policy in the paper, including the 175B one, is trained against a single 6B reward model — supervision from a model nearly 30x smaller than the policy it steers. This was a forced choice, not a preference: 175B RM training was unstable, which made large RMs unsuitable as initializations for the PPO value function, and a 175B RM plus value function would have greatly increased PPO's compute cost (instructgpt, §"C.2 Details of RM training", p. 41). Reusing the same 6B RM and value function across all policy sizes also had a methodological payoff, making policy-scale comparisons clean because the training signal is held constant (instructgpt, §"C.4 Details of RLHF training", p. 42). The arrangement quietly tests a scalable-oversight question the paper never poses directly: whether a weaker judge can usefully train a stronger model. Its headline results say yes, at least when the judge only needs to distill broad human preferences rather than out-reason the policy it is scoring.