Enduro (Atari game) — supplies the hard exploration case for → Human-Feedback Implicit Reward Shaping

Enduro's standard objective, passing cars, is genuinely hard for A3C to learn because successfully passing another car through undirected random exploration is a rare event under the game's own sparse reward, which is why the paper reports Enduro is 'difficult for A3C to learn' and correspondingly difficult to learn from synthetic labels drawn from that same sparse reward (deep-rl-human-prefs, §"3.1.2 Atari", p. 8). Human raters break that sparsity without being told to: nothing in their one-to-two-sentence brief ('pass as many cars as you can, and don't get passed by cars') instructs them to credit partial progress, yet in practice they 'tend to reward any progress towards passing cars,' turning an all-or-nothing exploration problem into one with dense, learnable gradient (deep-rl-human-prefs, §"B.2 Atari", p. 17; §"3.1.2 Atari", p. 8). The upshot is that human feedback outperforms A3C by roughly matching DQN's reported performance on the same game, even though DQN is an unrelated value-based method; the comparison only makes sense because it shows how much of A3C's shortfall was an exploration failure that dense, human-shaped reward corrects rather than a smarter RL algorithm.