Variable indifference
Concrete Problems in AI Safety — introduced
A proposed remedy aiming to route an agent's optimization pressure so it maximizes certain target variables while remaining indifferent to (not trying to influence) other specified variables, such as its own reward function.
Could an agent optimize some variables while genuinely not caring about others -- its own reward function, say, or human behavior? Variable indifference is that hope, stated in Concrete Problems without an algorithm to realize it.
The aspiration opens the page, paired with the algorithm it still lacks, and then the one case where it is fully realized. Reward pretraining, the remedy that achieves by brute force what this one only gestures at, is the natural next stop.
An aspiration without an algorithm
Variable indifference is Concrete Problems' most open-ended proposed remedy for Reward hacking (avoiding): a way to route an agent's optimization pressure so that it maximizes certain target variables in its environment while remaining indifferent to, not trying to influence, other specified variables, such as its own reward function or human behavior (concrete-problems, §"Variable Indifference:", p. 10). The paper states it only as an intuition, "a way to route the optimization pressure of powerful algorithms around parts of their environment," and is candid that no concrete algorithm accompanies the idea: "truly solving this would have applications throughout safety." It also flags a real difficulty in even stating the problem correctly: the variables targeted for indifference must be "the variables we care about in the world, as opposed to aliased or partially observed versions of them" (concrete-problems, §"Variable Indifference:", p. 10).
The one case where the aspiration is fully realized
Reward pretraining — is a concrete special case of → Variable indifference shows what this aspiration looks like when it actually works: Reward pretraining achieves indifference to exactly one variable, the reward function, but only because that variable can be fully specified in advance via supervised learning before optimization begins. The comparison exposes what variable indifference is still missing as a research problem, since most variables an agent should ignore, human behavior above all, cannot be pretrained away in the same manner and would need genuine indifference maintained during online optimization itself, not resolved by finishing the specification early.