Reward Prediction Normalization — is reverse engineered to avoid retuning → Advantage Actor-Critic (A2C/A3C)

For the Atari experiments the reward predictor's output is rescaled to a standard deviation of exactly 0.05, a figure the paper calls arbitrary in principle but chosen specifically so the same A2C hyperparameters used against the true reward, entropy bonus, learning rate, and the rest, could be reused completely unchanged (deep-rl-human-prefs, §"A.2 Atari", p. 15). The authors note they could equivalently have compensated for a different normalization scale by adjusting the learning rate and entropy bonus instead, which means the 0.05 figure was reverse-engineered from A2C's existing settings rather than the other way around (deep-rl-human-prefs, §"A.2 Atari", p. 15). This is also why A2C needed no hyperparameter changes at all under the learned reward, in contrast to TRPO on MuJoCo, where the entropy bonus still had to be raised despite an analogous normalization step (deep-rl-human-prefs, §"2.2.1 Optimizing the Policy", p. 5).