Enduro (Atari game)
Deep Reinforcement Learning from Human Preferences — inherited
An Atari racing game in the Arcade Learning Environment whose standard objective is to pass as many cars as possible; used both as a standard benchmark task and, in a modified 'even mode,' as the setting for the traffic-pace-keeping novel behavior; notable because human feedback outperformed A3C by shaping reward around incremental progress passing cars, matching DQN-level results.
Passing another car barely happens if an agent is just exploring at random, which makes Enduro a genuinely hard game to learn from its own sparse reward alone.
The page covers the standard objective and how the game is set up for preference learning, then the result where human raters rewarding incremental progress outperformed the underlying algorithm's usual approach, and closes on the inverted 'even mode' version of the same game used elsewhere in the paper.
What the task is
Enduro is an Atari racing game in the Arcade Learning Environment whose standard objective is to pass as many cars as possible; contractors were told to "pass as many cars as you can, and don't get passed by cars" (deep-rl-human-prefs, §"B.2 Atari", p. 17). Like the other Atari tasks, it is a child of the Atari games environment, with its on-screen speedometer blanked out so no score-adjacent display can leak reward information (deep-rl-human-prefs, §"A.2 Atari", p. 15).
A hard-exploration case for reward shaping
Enduro is genuinely difficult for A3C to learn because successfully passing a car through undirected random exploration is rare, making it "difficult for A3C to learn... and correspondingly difficult to learn with synthetic labels" drawn from that same sparse reward (deep-rl-human-prefs, §"3.1.2 Atari", p. 8). Human labelers close this gap by rewarding incremental progress: they "tend to reward any progress towards passing cars, essentially shaping the reward and thus outperforming A3C in this game (the results are comparable to those achieved with DQN)" (deep-rl-human-prefs, §"3.1.2 Atari", p. 8), per the Human-Feedback Implicit Reward Shaping edge.
Inverting the objective for keeping pace
The Enduro Keeping-Pace Demonstration retrains the identical game under a literally inverted instruction — for "Enduro (even mode)," contractors were told to "Avoid passing cars OR getting passed by them, you want to stay even with other cars" (deep-rl-human-prefs, §"B.2 Atari", p. 17). Trained with "roughly 1,300 queries and 4 million frames," the agent "learns to stay almost exactly even with other moving cars for a substantial fraction of the episode, although it gets confused by changes in background" (deep-rl-human-prefs, §"3.2 Novel behaviors", p. 9).