Penalizing side effects by distance from a baseline
A cleaning robot that tidies a room and knocks over a vase along the way has technically succeeded. This theme follows one answer to that problem from Concrete Problems (2016): instead of naming every possible harm in advance, penalize an agent for how far it drags the world from where a cautious, do-nothing policy would have left it.
From the side-effects problem itself, the page moves to the regularizers built to address it, then to the control-theory tools, reachability analysis and robust policy improvement, that make a baseline policy something you can actually compute.
Negative side effects arise when an objective function rewards task completion while leaving an agent indifferent to the collateral damage it causes along the way, a difficulty traceable to the classical frame problem of specifying which facts an action leaves unaffected. This theme covers the paper's baseline-distance strategy for taming that indifference. An impact regularizer penalizes an agent for how far it moves the environment from either its initial state or the outcome of a passive, do-nothing baseline policy. A learned impact regularizer proposes acquiring this penalty via transfer learning across many tasks instead of hand-defining it, on the premise that side effects look more similar across tasks than main goals do. Reachability analysis and robust policy improvement, both imported from control theory, supply the technical means for constructing the conservative baseline policies these regularizers are measured against. The shared claim is specific: side effects can be quantified and penalized purely as a distance from a reference state or policy, without ever specifying what the side effect actually is.