Empowerment
Concrete Problems in AI Safety — inherited
An information-theoretic measure of an agent's potential influence over its environment, defined as the maximum mutual information between its future actions and future state.
Empowerment measures, in bits, how much control an agent has over its own future -- formally, the maximum mutual information between its actions and the states they can bring about. It comes from the intrinsic-motivation and robotics literature, well before this corpus.
The mathematics is worked through in full, and then Concrete Problems runs the measure in reverse -- as a penalty instead of a drive -- with the page pinpointing exactly where that reversal breaks down. Penalize influence, the proposal that borrowed it, is where the thread leads next.
An information-theoretic measure that predates the paper
Empowerment is not a Concrete Problems invention. It is an information-theoretic quantity from the intrinsic-motivation and robotics literature, popularized in surveys such as Salge, Glackin, and Polani's "Empowerment: An Introduction," which the paper cites directly (concrete-problems, §"Penalize Influence:", p. 6). The paper inherits it as a candidate measure of an agent's potential influence over its environment, used elsewhere almost always as something to maximize rather than minimize.
The mathematics: mutual information between actions and future states
Empowerment at a state $s$ is defined as the maximum possible mutual information between an agent's future actions and its future state, $$\mathcal{E}(s) = \max_{p(a_{t:t+n})} I\big(A_{t:t+n}; S_{t+n} \mid S_t = s\big),$$ where the maximization ranges over all possible distributions the agent could choose to act with over an $n$-step horizon. Equivalently, this is the Shannon channel capacity of the environment viewed as a communication channel from the agent's actions to its own future state: it measures how reliably the agent could steer itself into a distinguishable range of future states if it tried, regardless of whether it actually does so. An agent locked in a room with no way out has low empowerment, since no sequence of actions changes what future state it can reach; an agent holding a key to that room has higher empowerment, since it can now reach a wider, more distinguishable range of futures within a few timesteps (concrete-problems, §"Penalize Influence:", p. 6).
Running the measure in reverse, and where that breaks
Concrete Problems' contribution is not the measure but a proposed sign flip: rather than maximizing empowerment as an intrinsic reward, which produces agents that seek out capability, avoiding walls, picking up keys, for its own sake even absent external reward, Penalize influence proposes minimizing it as a regularization term, so that acquiring capability becomes something to avoid. The paper is explicit that the inversion is imperfect, because empowerment "measures precision of control over the environment more than total impact" (concrete-problems, §"Penalize Influence:", p. 6): a button that cuts power to a million houses contributes only one bit of empowerment, since the action space itself has only one bit of mutual information available, despite an enormous real-world impact, while someone merely transcribing the agent's actions counts as maximal empowerment despite having no effect on anything at all. The measure and the goal it is being asked to formalize, does not accumulate potential for harm, are related but not identical, which is part of why empowerment sits within the broader theme Limiting potential influence instead of measuring realized change as one candidate formalization among several rather than a settled solution.