Enduro Keeping-Pace Demonstration
Deep Reinforcement Learning from Human Preferences — introduced
A novel behavior in which an agent playing the Atari game Enduro is trained, using roughly 1,300 queries and 4 million frames, to drive alongside other cars and stay roughly level with traffic, rather than the game's usual objective of passing as many cars as possible.
Enduro's whole point, ordinarily, is to pass as many cars as possible. This demonstration retrains the identical game toward the opposite goal, staying level with traffic instead, using nothing but changed instructions to the raters.
The page covers what the retrained agent actually learned to do, then the literal inversion of the standard instructions that produced it, and closes on why this particular game, already a hard one for the algorithm's own exploration, makes a stronger test of preference-driven redirection than an easier game would.
What the agent learned
The paper retrains an Enduro-playing agent, using roughly 1,300 preference queries and 4 million frames of environment interaction, to drive alongside other cars and stay roughly level with traffic rather than pursue the game's usual score (deep-rl-human-prefs, §"3.2 Novel behaviors", p. 9). The resulting agent stays almost exactly even with other moving cars for a substantial fraction of the episode, though it gets confused by changes in the background (deep-rl-human-prefs, §"3.2 Novel behaviors", p. 9).
A literally inverted objective
Standard Enduro instructions to contractors define good play as passing "as many cars as you can, and don't get passed by cars" (deep-rl-human-prefs, §"B.2 Atari", p. 17). The keeping-pace demonstration retrains the identical game and action space under the opposite instruction, labeled "Enduro (even mode)": "Avoid passing cars OR getting passed by them, you want to stay even with other cars" (deep-rl-human-prefs, §"B.2 Atari", p. 17). Nothing about the environment, the network architecture, or the RL algorithm changes between the two versions of the task; only the preferences fed to the reward model invert, itself a demonstration that one pipeline can be pointed at opposite goals inside the same game.
Why this task in particular
Enduro's ordinary passing objective is already hard for A2C's asynchronous relative A3C to learn, because successfully passing a car essentially never happens under undirected exploration, a difficulty central to Human-Feedback Implicit Reward Shaping (deep-rl-human-prefs, §"3.1.2 Atari", p. 8). Inverting exactly this game's objective, rather than an easier one, makes the demonstration a stronger test of whether human comparisons can redirect an agent's goal even where exploration itself, not just reward specification, is the bottleneck.