Advantage Actor-Critic (A2C/A3C)
Deep Reinforcement Learning from Human Preferences — inherited
A policy-gradient RL algorithm; the paper uses the synchronous variant (A2C) of Mnih et al. (2016)'s asynchronous A3C algorithm to train the policy on the seven Atari tasks against predicted reward from the learned reward model.
Scoring an action by the raw return it eventually leads to is noisy: much of that outcome is luck that has nothing to do with the choice itself. A2C cuts through the noise by judging each action against what a critic already expected the state to be worth.
The page opens with how the actor and critic fit together, then turns to why this algorithm, rather than a value-based one, suited a reward that kept changing mid-training, before covering the rescaling that let its original settings survive unchanged and the one game where its own exploration falls short.
The algorithm
A2C is a policy-gradient method that trains two coupled networks: an actor (\pi_\theta(a \mid s)) producing an action distribution, and a critic (V_\phi(s)) estimating expected future return from a state. Rather than scaling the policy gradient by the raw, noisy return, A2C scales it by the advantage (A(s,a) = r + \gamma V_\phi(s') - V_\phi(s)), how much better an action did than the critic expected. The actor update follows $$\nabla_\theta J(\theta) = \mathbb{E}\left[\nabla_\theta \log \pi_\theta(a \mid s)\, A(s,a)\right]$$ raising the probability of actions that beat the baseline and lowering it for those that fell short, while the critic is fit by regression toward observed returns. A2C is the synchronous counterpart of Mnih et al. (2016)'s A3C: parallel environment workers step in lockstep and contribute to one batched update, trading some throughput for determinism.
Its role in the paper
The paper uses A2C, with Mnih et al. (2016)'s hyperparameters left unchanged, to train the policy on all seven Atari tasks against the predicted reward from the learned Reward model (RM) rather than the game's own score (deep-rl-human-prefs, §"2.2.1 Optimizing the Policy", p. 4; §"A.2 Atari", p. 15). The Non-Stationary Reward Function Challenge is why a policy-gradient method was chosen at all: because the reward predictor keeps changing as new preference labels arrive, the authors say this "led us to focus on policy gradient methods" over value-based alternatives tuned for a fixed reward (deep-rl-human-prefs, §"2.2.1 Optimizing the Policy", p. 4).
Reused hyperparameters, engineered reward scale
Unlike TRPO, A2C needed no hyperparameter changes at all to cope with the moving target. That stability was purchased, not found: Reward Prediction Normalization rescales the reward predictor's output to a standard deviation of exactly 0.05, specifically so A2C's existing entropy bonus, learning rate, and other settings, tuned against the true Atari reward, could be reused completely unchanged (deep-rl-human-prefs, §"A.2 Atari", p. 15).
Where policy-gradient exploration falls short
A2C's reliance on its own exploration is also its weak point on Enduro: successfully passing another car is a rare event under undirected policy-gradient exploration, which the paper says makes Enduro "difficult for A3C to learn" and correspondingly hard to learn from synthetic labels drawn from that same sparse reward (deep-rl-human-prefs, §"3.1.2 Atari", p. 8). The paper reports its human-feedback result on Enduro ends up "comparable to those achieved with DQN" (deep-rl-human-prefs, §"3.1.2 Atari", p. 8), a value-based method whose Q-learning target does not depend on stumbling into a positive event through that same on-policy exploration.