Reward pretraining
Concrete Problems in AI Safety — introduced
A proposed remedy that trains a fixed reward function ahead of time via supervised learning, divorced from environment interaction, to prevent an agent from later influencing its own reward.
An agent cannot corrupt a teacher who has already left the room. Reward pretraining fixes the reward function ahead of time, by supervised learning divorced from the environment, so that nothing the agent later does can influence it.
The fix-it-first idea gets explained first, followed by the forks it opens against model lookahead and within the aspiration of variable indifference. Both reward a look from here, each answering the same worry in a different way.
Fixing the reward function before there is anything to influence
Reward pretraining is Concrete Problems' proposal for defending against cases where an agent "can influence its own reward function," citing Feedback loops (reward hacking cause) and Environmental embedding (reward hacking cause) as examples, by training a fixed reward function ahead of time via supervised learning, "divorced from interaction with the environment" the agent will later act in (concrete-problems, §"Reward Pretraining:", p. 10). This could mean learning from labeled state-reward pairs, or from trajectories via inverse reinforcement learning. Because the reward function is complete before optimization begins, there is nothing left for the agent to influence, at the cost of forfeiting "the ability to further learn the reward function after the pretraining is complete, which may create other vulnerabilities" (concrete-problems, §"Reward Pretraining:", p. 10).
Two forks: one against another remedy, one within an aspiration
Model lookahead — solves the same problem by a different route than → Reward pretraining contrasts this with Model lookahead: lookahead keeps the reward function mutable and bets on foresight, pretraining removes mutability altogether. Separately, Reward pretraining — is a concrete special case of → Variable indifference situates reward pretraining relative to Variable indifference, the paper's more general, unsolved aspiration to route an agent's optimization pressure around any chosen variable. Reward pretraining is what that aspiration looks like fully realized for exactly one variable, the reward function itself, when the variable can be completely specified in advance; most variables an agent should be indifferent to, such as human behavior, cannot be pretrained away in the same manner, which is exactly what variable indifference as a general research problem would still need to solve.