Recurrent Reward Model (Proposed Extension) — would generalize the fixed window of → Reward model (RM)

The reward model actually built in this paper never sees an agent's full history: the general formulation treats reward as a function of a single observation-action pair, and the Atari implementation widens that only to a fixed stack of the preceding 4 observations, not the whole episode (deep-rl-human-prefs, §"2.1 Setting and Goal", p. 4). The recurrent reward model extension is the paper's own flagged gap in this design: in a general partially-observable environment, reward could instead be modeled as a function of the entire history of observations via a recurrent neural network, rather than the fixed window the actual experiments use. The paper is explicit that this is unimplemented, a footnote rather than a method, which means every result in the paper is obtained with a reward model structurally blind to anything outside its fixed window. That leaves open, by the paper's own admission, whether behaviors whose reward genuinely depends on long-past events could be learned at all with the architecture actually tested.