Impact regularizer (defined)
Concrete Problems in AI Safety — introduced
A proposed penalty term that discourages an agent from changing the environment, formalized either as distance from an initial state or distance from a passive baseline policy.
If side effects are the worry, one blunt answer is to charge the agent for changing the world at all. The impact regularizer is that penalty term, measured either as distance from the initial state or as distance from what a do-nothing policy would have left behind.
The math carries the penalty from a raw measure of state change to a more careful passive baseline, and ends on the fork it opens with its learned sibling. Reachability analysis is worth a look next, for where the rigor actually comes from.
A penalty term for the paper's first headline problem
The impact regularizer is Concrete Problems' first concrete proposal for addressing Negative side effects (avoiding), one of the paper's five core research problems: if an agent should not cause side effects, the natural move is to penalize "change to the environment" directly, giving the agent a preference for achieving its goal with minimal disruption, or a limited budget of impact, rather than banning change outright (concrete-problems, §"Define an Impact Regularizer:", p. 5).
The mathematics: from raw state distance to a passive baseline
The naive version penalizes distance between the present state and the initial state, $d(s_i, s_0)$, for some metric $d$. The paper immediately rejects this formalization on its own terms: an agent penalized this way "wouldn't just avoid changing the environment, it will resist any other source of change, including the natural evolution of the environment and the actions of any other agents" (concrete-problems, §"Define an Impact Regularizer:", p. 5), a case of overcorrection in which indifference to harm becomes hostility to all change whatsoever, including change the agent had nothing to do with. The refinement compares the future state, or distribution over future states, under the agent's actual policy against the future state under a hypothetical passive baseline policy $\pi_{\text{null}}$, for instance one where a robot simply stands still, so that only the deviation attributable to the agent's own intervention is penalized, factoring out ordinary environmental evolution. Because ceasing to act is not always passive, dropping a heavy box mid-carry is not "doing nothing," the paper suggests replacing $\pi_{\text{null}}$ with a known safe but suboptimal policy and improving from there instead.
Where the rigor comes from, and the fork it creates
The refined regularizer's substance turns out to lie entirely in the baseline, not the penalty term, and the paper reaches outside itself for that rigor. It describes the baseline-policy approach as "somewhat reminiscent of reachability analysis... or robust policy improvement" (concrete-problems, §"Define an Impact Regularizer:", p. 5): Reachability analysis supplies a formal notion of which states remain safely recoverable under a given policy, giving "known safe but suboptimal" a precise meaning rather than an ad hoc placeholder, while Robust policy improvement supplies the guarantee that optimizing away from that baseline will not perform worse than the baseline itself even under model uncertainty. Whether this hand-defined regularizer is worth building at all is contested from within the same section: learning a regularizer via transfer across many tasks is offered as an alternative, on the bet that side effects look more similar across tasks than main goals do, so that a designer need not re-derive "change to the environment" from scratch for every new task. Impact regularizer, negative-side-effects, learned-impact-regularizer, reachability analysis, and robust policy improvement together make up the theme Penalizing side effects by distance from a baseline, whose shared claim is that side effects can be quantified and penalized as pure distance from a reference state or policy, without ever specifying what the side effect actually is.