Penalize influence
Concrete Problems in AI Safety — introduced
A proposed side-effect mitigation that discourages an agent from putting itself in positions of high potential influence over the environment, not just from acting on that influence.
Some positions are dangerous before anything goes wrong: a robot need not knock the bucket off the ladder for carrying it up there to have been unwise. Penalize influence proposes charging an agent for acquiring that kind of potential, not just for using it.
The shift from realized to potential change anchors the page, followed by the borrowed machinery of empowerment used to measure it, and where that borrowing breaks. Low-impact agents, the earlier sketch this proposal quietly extends, rewards a look too.
Penalizing potential, not just realized, change
Penalize influence is one of Concrete Problems' proposed remedies for Negative side effects (avoiding), distinguished from the baseline-distance approach of an Impact regularizer (defined) by what it penalizes: not what the agent has done, but what it has put itself in a position to do. The motivating example is a cleaning robot that should avoid bringing "a bucket of water into a room full of sensitive electronics, even if it never intends to use the water in that room" (concrete-problems, §"Penalize Influence:", pp. 5-6): nothing has happened yet, no state has changed, only the option to cause harm has been acquired.
Borrowed machinery and its known flaw
To formalize "potential for influence," the paper reaches for Empowerment, the information-theoretic quantity usually maximized as an intrinsic reward, and proposes running it in reverse: minimizing rather than maximizing empowerment, so seeking capability becomes something to avoid (concrete-problems, §"Penalize Influence:", p. 6). The paper documents the proposal's own weakness in the same breath it makes it: empowerment "measures precision of control over the environment more than total impact," so an action that could cut power to a million houses counts as only one bit, while a bystander merely transcribing the agent's actions counts as maximal empowerment despite zero effect on anything (concrete-problems, §"Penalize Influence:", p. 6). The paper also notes a perverse incentive this creates: an agent penalized for empowerment might destroy a vase specifically to remove its own future option of breaking it, eliminating potential influence by eliminating the object rather than by acting more carefully around it.
A genuine extension of an older idea, and a nearby alternative
Penalize influence is often read as one more implementation of Armstrong's informally discussed "low-impact agents" concept, but it actually relocates the target of that idea from realized change to potential change, a genuine extension rather than a restatement, since an agent could satisfy every ordinary reading of "low impact" while still quietly accumulating exactly this kind of latent power. Sitting beside it in the same section, Reward uncertainty (side-effect mitigation) attacks the identical problem with a different architecture: instead of an explicit penalty term that someone must hand-tune, it gives the agent a prior belief that unplanned change tends to be bad and lets ordinary expected-value reasoning supply the caution, a trade of penalty for uncertainty relative to penalize influence. Both concepts, together with empowerment and low-impact agents, make up the theme Limiting potential influence instead of measuring realized change, the corpus's alternative vocabulary to baseline-distance for the same underlying side-effects problem.