PPO value function
Training language models to follow instructions with human feedback — inherited
The critic network used in PPO advantage estimation, initialized from the trained reward model's weights.
Reinforcement learning needs a running estimate of how well things are going, so the algorithm can tell which actions beat expectations. That estimator is the value function, or critic, and in InstructGPT it is not built from scratch but grown out of the reward model itself.
From the general idea of a critic, the page narrows to how InstructGPT initializes it from the reward model's weights, then to the more surprising choice: keeping it fixed at six billion parameters even as the policy scales far larger.
The critic, in general
In actor-critic reinforcement learning, a value function $V^\pi(s) = \mathbb{E}_\pi\big[\sum_{t=0}^{\infty} \gamma^t r_t \mid s_0=s\big]$ estimates the expected discounted return obtainable from a state under the current policy, and the advantage $A(s,a) = Q(s,a) - V(s)$ measures how much a specific action beats that baseline. A learned "critic" network approximating $V$ is used to reduce the variance of policy-gradient estimates, a standard technique tracing back to the Bellman-equation foundations of dynamic programming and used throughout modern deep RL, including PPO. This concept predates InstructGPT and is inherited here as a standard component of the PPO fine-tuning stage (RLHF policy optimization) recipe rather than something the paper introduces.
Initialized from the reward model, not from scratch
InstructGPT's value function is not trained independently; it is initialized from the weights of the trained Reward model (RM), since both share the same architecture, a scalar-output head attached to a GPT-3-derived model (instructgpt, §"C.4 Details of RLHF training", p. 42). This initialization supplies the critic with a reasonable starting estimate of output quality before any PPO gradient steps are taken, rather than learning value estimation entirely from RL experience.
Why it stays fixed at 6B regardless of policy size
A single 6B value function, matching the 6B reward model, is reused across PPO fine-tuning stage (RLHF policy optimization) runs at every studied policy size, 1.3B, 6B, and 175B (instructgpt, §"C.4 Details of RLHF training", p. 42). This was a forced engineering choice rather than a preference: training a 175B reward model proved unstable, making it unsuitable as a value-function initialization, and pairing a 175B value function with a 175B policy would have substantially increased PPO's compute cost (instructgpt, §"3.5 Models", p. 8; §"C.2 Details of RM training", p. 41). Reusing the same critic across policy sizes also has a methodological benefit noted alongside the Reward model (RM) — supervises from below → PPO fine-tuning stage (RLHF policy optimization) edge: it holds the training signal constant, making comparisons of policy scale to policy performance cleaner, since only the actor, not the critic, changes size.