Recurrent Reward Model (Proposed Extension)
Deep Reinforcement Learning from Human Preferences — introduced
A noted but unimplemented extension: in a general partially-observable environment, the reward function could be modeled as depending on the whole history of observations via a recurrent neural network, rather than on a fixed window of recent observations as used in the paper's actual experiments.
The reward model actually built here only ever looks at a handful of recent observations, never an agent's whole history. The paper knows this is a limitation and says so, without doing anything about it.
The page covers the fixed window actually used, then the more general recurrent alternative the paper names in a footnote, and closes on why this counts as a deliberately noted road not taken rather than an ablation the paper actually ran.
A noted but unbuilt generalization
The Reward model (RM) actually implemented in this paper never conditions on an agent's full history. Footnote 1 states the general formulation treats reward as a function $r:O\times A\to\mathbb{R}$ of a single observation-action pair, then notes "in our experiments in Atari environments, we instead assume the reward is a function of the preceding 4 observations" (deep-rl-human-prefs, §"2.2 Our Method", p. 4) — a fixed window, not the whole episode; the Atari reward predictor's actual input is indeed 4 stacked 84x84 frames (deep-rl-human-prefs, §"A.2 Atari", p. 15). The same footnote names the more general alternative: "In a general partially observable environment, we could instead consider reward functions that depend on the whole sequence of observations, and model this reward function with a recurrent neural network" (deep-rl-human-prefs, §"2.2 Our Method", p. 4).
Left unimplemented
The paper does not build or test this recurrent version in any experiment; the fixed 4-frame stack is what actually ships for Atari, and the MuJoCo reward predictor conditions on even less history. The extension is recorded here as a design space the paper itself identified and consciously declined to pursue — distinct from an ablation, which the paper does implement and measure. This is a road not taken rather than a road tried and found wanting.