Reward uncertainty (side-effect mitigation)
Concrete Problems in AI Safety — introduced
A proposal to give an agent a probability distribution over reward functions, biased toward assuming random environmental changes are bad, rather than a single fixed reward function.
Instead of one fixed reward function, an agent could hold a spread of possible ones -- weighted so that random changes to the world are presumed bad until shown otherwise. That presumption of caution is the reward-uncertainty proposal.
That cautious distribution anchors the page, set against a different architecture aimed at the very same side-effect problem. Penalize influence tackles the same worry from the opposite direction, and rewards a look next.
Replacing a fixed reward with a distribution biased toward caution
Reward uncertainty is a remedy Concrete Problems proposes for Negative side effects (avoiding), motivated by an argument about priors rather than a penalty term: "we want to avoid unanticipated side effects because the environment is already pretty good according to our preferences, a random change is more likely to be very bad than very good" (concrete-problems, §"Reward Uncertainty:", pp. 6-7). Rather than giving the agent a single fixed reward function $R$, the proposal gives it a prior probability distribution $P(R)$ over possible reward functions, constructed so that $\mathbb{E}_{R \sim P}[R(\Delta s)] \le 0$ for unplanned changes $\Delta s$ to the environment: the prior is deliberately skewed to treat random, unaccounted-for change as more likely harmful than beneficial. The agent then acts to maximize expected reward under this distribution, $\mathbb{E}_{R \sim P}\left[\sum_t \gamma^t R(s_t)\right]$, so caution about disruptive behavior falls out of ordinary Bayesian expected-value reasoning rather than requiring a separately engineered penalty.
A different architecture for the same problem
Reward uncertainty sits beside Penalize influence as an alternative attack on the same failure mode, but the two differ in architecture, not just detail. Penalize influence, like the impact regularizer it sits near, works by adding an explicit term to the objective and requiring someone to hand-tune how costly it should be for the agent to acquire capability. Reward uncertainty abandons that architecture entirely: it needs only a prior belief that unplanned change tends to be bad, after which ordinary decision-making under uncertainty does the rest, with no separate penalty coefficient to calibrate. As the paper notes, this proposal still relies on defining some baseline around which "unplanned" or "unanticipated" change is measured, again invoking the same reachability-analysis and robust-policy-improvement machinery used to build the baseline for the hand-defined impact regularizer.
Where it sits in the corpus
Reward uncertainty is grouped, alongside Penalize influence, Empowerment, and Low-impact agents, in the theme Limiting potential influence instead of measuring realized change, the corpus's second broad strategy for taming side effects, one that works through uncertainty and priors rather than through measuring distance from a fixed baseline state.