Generalized advantage estimation (GAE)
Training language models to follow instructions with human feedback — inherited
The advantage-estimation technique of Schulman et al. (2016) used within PPO training, applied here with no discounting.
Policy-gradient methods learn faster when they can judge each action against a baseline: how much better was this than what usually happens? Generalized advantage estimation, borrowed from earlier reinforcement learning work, is the machinery InstructGPT uses to make that judgment.
The page explains what GAE computes and how PPO fine-tuning applies it with discounting switched off, building to a quiet irony: the one-step bandit setting leaves half the machinery idle. That deactivation is the detail worth carrying forward.
What GAE computes, in general
Generalized advantage estimation, introduced by Schulman et al. (2016), is a technique for estimating the advantage function used in policy-gradient reinforcement learning, $A(s,a) = Q(s,a) - V(s)$, the amount by which taking action $a$ in state $s$ beats the average value of that state. Given a trajectory of rewards $r_t$ and a learned value function $V$, define the temporal-difference residual $\delta_t = r_t + \gamma V(s_{t+1}) - V(s_t)$. GAE forms the advantage estimate as an exponentially weighted sum of these residuals, $$\hat{A}_t^{GAE(\gamma,\lambda)} = \sum_{l=0}^{\infty} (\gamma\lambda)^l \, \delta_{t+l},$$ where $\gamma$ is the standard discount factor and $\lambda \in [0,1]$ trades off bias against variance: $\lambda = 0$ recovers the low-variance, high-bias one-step TD estimate, and $\lambda = 1$ recovers the high-variance, unbiased Monte Carlo return. GAE predates InstructGPT by six years and is a standard component of PPO implementations; the concept is inherited here as an implementation detail of the PPO fine-tuning stage (RLHF policy optimization) stage rather than a contribution of the paper.
Its use in PPO fine-tuning, without discounting
InstructGPT applies GAE within its PPO fine-tuning stage (RLHF policy optimization) stage exactly as specified by Schulman et al. (2016), with one notable setting: "no discount is applied when estimating the generalized advantage" (instructgpt, §"C.4 Details of RLHF training", p. 42), meaning $\gamma = 1$ in the formula above. As a child concept of ppo-training, GAE works alongside the PPO value function (the critic supplying $V$) to produce the advantage signal PPO's clipped objective optimizes.
Why the bandit setting deactivates half of it
The Bandit environment formulation for RLHF — renders discounting moot in → Generalized advantage estimation (GAE) edge explains why $\gamma=1$ is the natural choice here: GAE's discount factor exists to down-weight rewards arriving far in the future, but InstructGPT's Bandit environment formulation for RLHF terminates after a single prompt-response-reward exchange, so there is no future to down-weight. The only remaining temporal structure is the token sequence within one response, and since the reward model scores the completed text as a whole, there is no principled reason a later token should count for less than an earlier one. What survives from GAE in this setting is not its handling of time but its variance-reduction across the tokens of a single response, a reminder that RL machinery calibrated for long-horizon credit assignment can have parts of it quietly deactivated by a short-horizon problem.