Advantage Actor-Critic (A2C/A3C) — closes its enduro exploration gap with → Deep Q-Network (DQN)

A2C, a policy-gradient method, and DQN, a value-based method, sit on opposite sides of an algorithmic divide that the paper's Enduro result puts to an inadvertent test. The paper states that Enduro is difficult for A2C's asynchronous relative A3C to learn because successfully passing another car essentially never happens under random policy-gradient exploration, and this same difficulty carries over when the reward comes from synthetic preference labels rather than the game's score (deep-rl-human-prefs, §"3.1.2 Atari", p. 8). Human labelers close the gap by a route no algorithm change could: instructed only to reward any visible progress toward passing a car, they hand the policy a denser shaping signal than random exploration would ever discover on its own, and the paper reports the result is "comparable to those achieved with DQN" (deep-rl-human-prefs, §"3.1.2 Atari", p. 8). The comparison is notable precisely because DQN was never trained in this paper at all, it is a citation to prior published performance, so the underlying claim is that reward-shaped preference learning let a policy-gradient method match a structurally different algorithm's results on the one game where the policy-gradient approach's own exploration was the bottleneck.