TAMER Framework
Deep Reinforcement Learning from Human Preferences — inherited
A prior framework (Knox and Stone, 2009; Knox, 2012) for learning a reward function from real-time human feedback, discussed in related work as tackling much simpler settings where the desired policy can be learned relatively quickly, unlike the thousands of hours of experience required in this paper's Atari domain.
Learning a reward function from real-time human feedback is an older idea than this paper, and one earlier framework already tried it on much simpler problems.
The page covers how that earlier framework worked and what its feedback loop looked like, then closes on the specific gap, task difficulty and how many hours of experience are involved, that the paper points to when explaining why its own approach had to differ.
What TAMER is
TAMER (Knox and Stone, 2009; Knox, 2012) is a framework for training an agent from real-time human feedback by learning an explicit model of human reward, rather than treating human signals as a direct RL reward. As a trainer watches the agent act, they issue scalar feedback after observed state-action pairs; TAMER fits a supervised model (\hat{H}(s,a)) predicting that feedback, and the agent then acts to greedily maximize predicted human reward at each timestep rather than optimize a discounted return through full temporal-difference credit assignment. Treating the problem as near-myopic regression onto immediate human reward, instead of propagating credit backward through time as in standard RL, is TAMER's central methodological choice; later variants (Knox and Stone, 2013) extend it to account for delayed reward.
Why the paper distinguishes itself from it
The paper places TAMER among prior methods that "perform experiments involving reinforcement learning from actual human feedback" but differ from its own algorithmic approach (deep-rl-human-prefs, §"1.1 Related Work", p. 3). TAMER also learns a reward function, like this paper does, but the paper notes it considers "much simpler settings where the desired policy can be learned relatively quickly" (deep-rl-human-prefs, §"1.1 Related Work", p. 3), unlike the thousands of hours of experience the paper's own Atari domain requires. That gap matters because TAMER's real-time scalar feedback does not scale the way comparisons do: needing a human to rate behavior every few timesteps across thousands of hours of gameplay is the exact cost this paper is built to avoid, by querying only a tiny, actively selected fraction of the agent's experience.