Bandit environment formulation for RLHF — renders discounting moot in → Generalized advantage estimation (GAE)

GAE normally exposes two bias-variance knobs, λ and a discount factor, and the discount exists to down-weight rewards that arrive far in the future. The bandit formulation removes the future: an episode is one prompt, one sampled response, one reward-model score, then termination (instructgpt, §"3.5 Models", p. 9). The only temporal structure left is the token sequence within a single response, and there is no principled reason a token generated late in an answer should count for less than an early one — the reward model judges the completed text as a whole. Accordingly, the training recipe applies no discount when estimating the generalized advantage (instructgpt, §"C.4 Details of RLHF training", p. 42). This is a small setting with a real lesson: much of classic RL machinery is calibrated for long-horizon credit assignment, and RLHF's single-exchange episode quietly deactivates parts of it. What remains load-bearing is GAE's variance reduction across tokens, not its handling of time.