Non-Stationary Reward Function Challenge — forces an entropy bonus increase in → Trust Region Policy Optimization (TRPO)

Where A2C needed no adjustment, TRPO did: the paper reports raising TRPO's entropy bonus above its standard setting, the only hyperparameter changed anywhere in the simulated-robotics experiments, specifically because TRPO's trust region is what normally guarantees adequate exploration, and that guarantee degrades when the reward driving it is being rewritten during training (deep-rl-human-prefs, §"2.2.1 Optimizing the Policy", p. 5). The appendix confirms the fix was applied uniformly, an entropy bonus of 0.01 on every MuJoCo task but Swimmer (0.001), and ties the change explicitly back to compensating for a changing reward function rather than to any property of the individual tasks (deep-rl-human-prefs, §"A.1 Simulated Robotics Tasks", p. 14). This makes TRPO the one place in the paper where the non-stationary-reward problem required an active correction rather than being absorbed by the algorithm's existing design.