Reward model (RM) — anchors the reward signal of → Reinforcement learning from human feedback (RLHF)

The reward model is not a free-standing scorer bolted onto RLHF; its architecture and its role are both dictated by how PPO training actually works. It is built by removing the SFT model's unembedding layer and adding a scalar output head, so it starts from the same weights as the policy it will later score (instructgpt, §"3.5 Models", p. 8). During RLHF it serves two jobs at once: its output is the PPO reward, normalized so labeler demonstrations score a mean of zero, and its own weights initialize the PPO value function, since RM and value function share the same architecture. Because 175B RM training was unstable, this shared-initialization requirement is the specific reason the reward model — and therefore the value function — was capped at 6B parameters even when the policy itself scaled to 175B. A per-token KL penalty against the SFT model, not the RM, keeps PPO from over-optimizing this imperfect proxy.