Risk-sensitive performance criteria
Concrete Problems in AI Safety — inherited
A body of prior work replacing expected total reward with objectives (worst-case performance, bounded probability of very bad outcomes, variance penalties) that better prevent rare catastrophic events.
Most reinforcement learning chases the average outcome. A separate body of work asks agents to mind the worst one instead, swapping expected reward for objectives that bound variance or the probability of disaster. Concrete Problems in AI Safety (2016) surveys this literature as risk-sensitive performance criteria.
The page moves from the body of work the paper imports rather than invents, through the mathematics of replacing an expectation with something else, to what these objectives catch that a near-optimality proof does not.
A body of prior work the paper imports, not invents
Risk-sensitive performance criteria is not original to Concrete Problems; it names an existing body of literature on changing the optimization objective in reinforcement learning away from expected total reward, which the paper reviews and points to a dedicated survey for rather than developing itself (concrete-problems, §"Risk-Sensitive Performance Criteria:", p. 14).
The mathematics: replacing an expectation with something else
Standard RL optimizes expected discounted return, $\max_\pi \mathbb{E}_\pi\left[\sum_t \gamma^t r_t\right]$, an objective indifferent to how that expectation is achieved: a policy that earns moderate reward reliably and one that earns huge reward most of the time but catastrophic reward rarely can have identical expected value. Risk-sensitive criteria replace the expectation with objectives that specifically penalize the bad tail of the outcome distribution. Worst-case criteria optimize $\max_\pi \min_{\text{trajectory}} R(\text{trajectory})$ directly. Probabilistic-bound criteria instead constrain the probability of very poor performance, requiring $P(R < c) \le \delta$ for some unacceptable threshold $c$ and small $\delta$. Variance-penalized criteria modify the objective to $\mathbb{E}[R] - \lambda \, \mathrm{Var}[R]$, trading off expected performance against its spread for some risk-aversion coefficient $\lambda > 0$. A related family draws on conditional value-at-risk, optimizing the expected return conditional on being in the worst $\alpha$-fraction of outcomes, $\mathrm{CVaR}_\alpha(R) = \mathbb{E}[R \mid R \le \mathrm{VaR}_\alpha(R)]$, which directly targets the rare catastrophic tail rather than the distribution's mean.
What it targets that a near-optimality proof does not
The paper's interest in this literature is specifically about what expected-value optimality guarantees fail to catch. R-max's headline guarantee, for instance, is near-optimal performance with respect to expected total reward, but that guarantee is silent about the variance of outcomes along the way or the probability of a rare catastrophic episode, because neither is part of what R-max is proven to optimize; risk-sensitive-performance-criteria targets exactly that gap (concrete-problems, §"Risk-Sensitive Performance Criteria:", p. 14). The paper notes these methods had, at the time of writing, not yet been tested with expressive function approximators such as deep neural networks, though it points to work adapting policy-gradient algorithms to risk-sensitive criteria as evidence this should be possible in principle. Risk-sensitive-performance-criteria belongs to the theme Exploration policies and the objectives that make them safe or reckless, naming the missing objective, distinct from any specific algorithm, that the paper's safe-exploration proposals implicitly argue reinforcement learning needs.