Deep Q-Network (DQN)
Deep Reinforcement Learning from Human Preferences — inherited
A value-based deep RL algorithm (Mnih et al., 2015), mentioned as a baseline whose reported Enduro performance is comparable to what the paper's preference-based method achieves.
Not every algorithm in this corpus is one the paper's own method runs; some show up only as a yardstick. DQN is one of those, a value-based algorithm the paper never trains but reaches for anyway when it needs a number to compare against.
The page explains what DQN does and how it differs from a policy-gradient approach, then its narrow role here as the benchmark the paper's own Enduro result happens to match, and closes on why the two algorithms diverge on exactly that game.
The algorithm
DQN is a value-based deep RL algorithm (Mnih et al., 2015) that learns an action-value function (Q_\theta(s,a)) approximating expected discounted return, and acts by choosing (\arg\max_a Q_\theta(s,a)) (with (\epsilon)-greedy exploration during training). It is trained by minimizing the temporal-difference error against a bootstrapped target: $$L(\theta) = \mathbb{E}\Big[\big(r + \gamma \max_{a'} Q_{\theta^-}(s',a') - Q_\theta(s,a)\big)^2\Big]$$ where (\theta^-) are the parameters of a periodically frozen target network, held fixed for many updates so the target does not chase itself and destabilize training. DQN also decorrelates consecutive transitions by sampling minibatches from a large replay buffer of past experience rather than training on the trajectory as it is generated, the combination that first made deep Q-learning stable enough to match human performance on Atari from raw pixels.
Its role in the paper
DQN appears only as a reference point, not as an algorithm the paper's own method runs. On Enduro, the paper reports that its human-feedback result is "comparable to those achieved with DQN" (deep-rl-human-prefs, §"3.1.2 Atari", p. 8), citing Mnih et al. (2015)'s reported score on that game as the benchmark its own preference-trained policy happens to reach.
Why DQN and A2C diverge on Enduro
Advantage Actor-Critic (A2C/A3C), the policy-gradient method the paper actually uses to optimize against the reward model's output, and DQN sit on opposite sides of a basic algorithmic divide that Enduro's difficulty exposes. A2C's asynchronous relative A3C struggles because passing a car is a rare event under on-policy exploration, so it rarely receives a positive signal to reinforce, and the same scarcity limits how well it learns from synthetic labels drawn from that same sparse true reward (deep-rl-human-prefs, §"3.1.2 Atari", p. 8). DQN's off-policy value-learning update does not depend on stumbling into the rewarding event on-policy in the same way, consistent with the paper treating a DQN-level score as the natural comparison point for what the preference-based agent achieves on this game.