Reward model (RM) — is overbuilt in anticipation of → Novel Behavior Training

The reward predictor used for the MuJoCo tasks is a full two-layer neural network even though the true reward functions for those standard benchmarks are, as the paper notes, all just second-degree polynomials of the input features, simple enough that a much simpler regression method would have sufficed (deep-rl-human-prefs, §"A.1 Simulated Robotics Tasks", p. 14). The paper is explicit that this is a deliberate overprovision: "using this more flexible architecture allows us to immediately generalize to tasks for which the reward function is not so simple," pointing directly at the novel-behavior experiments (deep-rl-human-prefs, §"A.1 Simulated Robotics Tasks", p. 14). In other words, the reward model's capacity was sized for tasks the authors had not yet run at the time it was built, not for the benchmark tasks it was first validated against.