Negative side effects (avoiding)
Concrete Problems in AI Safety — refined
The problem that an objective function focused narrowly on a task may leave an agent indifferent to, and hence likely to cause, harmful disruptions elsewhere in a large environment.
An agent told to do one thing may damage other things simply because its objective never mentions them. That indifference -- a cleaning robot toppling a vase it was never scored on -- is the problem of negative side effects.
This idea sits among the paper's five core problems, then finds a much older diagnosis for itself in classical AI, before the family of fixes it spawned divides into penalizing change directly and penalizing potential influence. The frame problem and the impact regularizer are both worth following from here.
One of the paper's five core problems
Negative side effects is one of Concrete Problems' five headline research problems, and the paper places it, together with reward hacking, in the category of accidents caused by a wrong objective function, as opposed to failures during learning itself (concrete-problems, §"2 Overview of Research Problems", p. 2). The problem is stated through the paper's cleaning-robot running example: an objective that rewards moving a box from one side of a room to the other says nothing about a vase of water sitting in the robot's path, so an agent given reward only for the box will probably knock the vase over (concrete-problems, §"3 Avoiding Negative Side Effects", p. 4). The paper generalizes past vases immediately: for any agent operating in a large, multifaceted environment, an objective function focused on one aspect of that environment implicitly expresses indifference to every other aspect, and that indifference tends toward harm on average, since side effects disrupt a status quo that likely reflects human preferences more than a random change would. Negative side effects refines an informally discussed prior notion the paper calls "low impact agents," giving it a name, a running example, and a menu of candidate mitigations.
An old diagnosis for a new problem
A footnote, not the main text, supplies the problem's intellectual lineage: "Intuitively, this seems related to the frame problem, an obstacle in efficient specification for knowledge representation" (concrete-problems, §"3 Avoiding Negative Side Effects", p. 4). McCarthy and Hayes' 1969 Frame problem concerned the difficulty of specifying, in a formal knowledge-representation system, which facts an action leaves unchanged; negative side effects restates that difficulty as an engineering problem, a designer cannot feasibly enumerate every "vase" an agent must not knock over, because doing so requires specifying everything the task should leave untouched, which is exactly what the frame problem showed is hard to do efficiently. The connection is offered as a hedge rather than a solution: the paper does not claim to resolve a decades-old classical AI difficulty, only that it resurfaces, largely unaddressed, in modern reinforcement learning objective design.
Four children, two strategies
The paper's proposed remedies split into two families, both represented among this concept's children. Defining or learning an impact regularizer penalizes distance from a passive baseline; the first attempt at this, penalizing raw distance from the initial state, overcorrects into resisting all environmental change, including change the agent had no part in, which is why the paper moves quickly to a baseline-policy comparison instead. Penalizing influence and reward uncertainty instead attack potential rather than realized change, discouraging an agent from acquiring positions of power over its environment, or replacing a fixed reward with a distribution biased toward assuming random change is bad. A fourth, multi-agent approaches, abandons the distance-from-baseline strategy altogether in favor of treating side-effect avoidance as a matter of understanding and protecting other agents' interests directly. Negative side effects anchors the theme Penalizing side effects by distance from a baseline, and its four children between them cover essentially every mitigation strategy the paper offers for the problem it names.