Trust Region Policy Optimization (TRPO)
Deep Reinforcement Learning from Human Preferences — inherited
A policy-gradient RL algorithm (Schulman et al., 2015) used to train the policy on the simulated MuJoCo robotics tasks; the paper increases TRPO's entropy bonus above standard settings to compensate for reduced exploration under a changing, learned reward function.
A single large update to a policy's parameters can sometimes wreck it entirely, since the same distance in parameter space can mean a tiny change in behavior or a huge one, depending on where you start. TRPO exists to make that update safe.
The page covers the trust-region idea behind the algorithm, then its narrow role training the paper's simulated-robotics tasks, and closes on the one hyperparameter the authors did adjust, and why a changing, learned reward made that adjustment necessary in the first place.
The algorithm
TRPO is a policy-gradient method built to fix a basic instability in naive policy-gradient ascent: a single large step in parameter space can produce a wildly worse policy, because the same parameter distance can correspond to a tiny or an enormous change in the resulting action distribution depending on where you start. TRPO instead maximizes a surrogate objective subject to a constraint on how far the new policy may move from the old one in distribution space, not parameter space: $$\max_\theta \; \mathbb{E}\left[\frac{\pi_\theta(a \mid s)}{\pi_{\theta_{old}}(a \mid s)} A_{\theta_{old}}(s,a)\right] \quad \text{s.t.} \quad \mathbb{E}\big[D_{KL}(\pi_{\theta_{old}}(\cdot \mid s) \,\|\, \pi_\theta(\cdot \mid s))\big] \le \delta$$ where (A_{\theta_{old}}) is the advantage under the old policy and (\delta) is a fixed trust-region radius. Solving this constrained problem, via a conjugate-gradient approximation to the natural gradient, guarantees the new policy cannot become catastrophically worse in a single update.
Its role in the paper
The paper uses TRPO (Schulman et al., 2015) to train the policy on all of the simulated MuJoCo robotics tasks against the predicted reward from the learned Reward model (RM), reusing parameter settings "which have been found to work well for traditional RL tasks" (deep-rl-human-prefs, §"2.2.1 Optimizing the Policy", p. 4).
The one hyperparameter the paper touched
TRPO was the only algorithm in the paper that needed retuning to survive a changing reward, and its entropy bonus was the only hyperparameter changed anywhere in the robotics experiments (deep-rl-human-prefs, §"2.2.1 Optimizing the Policy", p. 4). The reason is specific to how TRPO explores: it "relies on the trust region to ensure adequate exploration," a guarantee tuned for a fixed reward landscape, and the Non-Stationary Reward Function Challenge degrades that guarantee because the reward model being optimized against is itself rewritten throughout training as new comparisons arrive (deep-rl-human-prefs, §"2.2.1 Optimizing the Policy", p. 4). Raising the entropy bonus above its standard value, 0.01 on most tasks and 0.001 on Swimmer, compensates by keeping the policy more stochastic than the trust region alone would produce (deep-rl-human-prefs, §"A.1 Simulated Robotics Tasks", p. 14).