Learned impact regularizer
Concrete Problems in AI Safety — introduced
A proposal to learn a generalized impact regularizer via transfer learning across many tasks, rather than hand-defining it, on the premise that side effects are more similar across tasks than the main goals are.
Rather than hand-writing a penalty for changing the environment, one could learn it -- training across many tasks, on the bet that side effects look alike even when goals differ. That is the learned impact regularizer.
The bet the proposal rests on is laid out plainly, and then comes the admission that it remains unsettled against its hand-defined alternative. That alternative, the plain impact regularizer, is the one to follow out.
Learning the penalty instead of writing it down
The learned impact regularizer is presented in Concrete Problems as an explicit fork against the hand-defined Impact regularizer (defined): "An alternative, more flexible approach is to learn (rather than define) a generalized impact regularizer via training over many tasks" (concrete-problems, §"Learn an Impact Regularizer:", p. 5). Rather than a designer formalizing "change to the environment" once per task, this proposal treats that formalization as something to be acquired through transfer learning, trained on many tasks and reused.
The bet the proposal makes
The argument for switching is specifically about transfer, not about accuracy: "side effects may be more similar across tasks than the main goal is" (concrete-problems, §"Learn an Impact Regularizer:", p. 5). The paper illustrates with a painting robot, a cleaning robot, and a factory-control robot, three very different main goals that nonetheless probably all want to avoid knocking over similar kinds of furniture. This suggests separating the side-effect component of a policy from its task component, training them with separate parameters, so that transfer learning could move the side-effect component across tasks even where the task component itself does not transfer, an approach the paper likens to model-based RL methods that transfer a learned dynamics model but not the value function, with the novelty here being that it is side effects, rather than environment dynamics, that get isolated as the transferable piece.
An unresolved wager, relative to its alternative
The relationship between learned and hand-defined regularizers is explicitly adversative: the paper poses them back to back as a genuine fork rather than a strict improvement. The hand-defined Impact regularizer (defined) requires a designer to correctly formalize change once per task but makes no empirical assumption about transferability; the learned version removes that per-task burden but bets, without evidence offered in the paper itself, that side-effect structure really is close to task-invariant. The paper notes one further potential payoff of that wager: regularizers "known or certified to produce safe behavior on one task might be easier to establish as safe on other tasks" if the underlying side-effect model transfers. Learned impact regularizer belongs to the theme Penalizing side effects by distance from a baseline alongside Negative side effects (avoiding), Impact regularizer (defined), Reachability analysis, and Robust policy improvement, the group of concepts through which the paper tries to quantify side effects as distance from a reference, whether that reference is hand-specified per task or learned once and carried across many.