Robust policy improvement
Concrete Problems in AI Safety — inherited
A prior line of work on improving a policy while guaranteeing performance under model uncertainty, cited as a technique for constructing safe baseline policies.
Improving a policy is easy to want and risky to do when the true dynamics of the world are uncertain. Robust policy improvement is a prior line of RL work on making a policy better while guaranteeing it cannot get worse, even in the worst case.
That guarantee anchors the page before Concrete Problems pairs the technique with reachability analysis to build safe baselines -- the partnership worth following from here. A link back to the impact regularizer closes the loop.
Improving a policy without losing a performance guarantee
Robust policy improvement names a line of RL research, predating this paper, concerned with updating a policy under genuine uncertainty about the true dynamics model while still guaranteeing a floor on performance — the improved policy should not perform worse than some reference, even in the worst case consistent with the agent's uncertainty about how the environment actually behaves. This connects to the broader literature on robust MDPs, where the transition model is not assumed known exactly but only to lie within some uncertainty set $\mathcal{P}$, and policy evaluation replaces the ordinary Bellman expectation with a worst-case (minimax) operator over that set. Concrete Problems cites the idea repeatedly as a source of technique for constructing safe baseline policies, without itself developing the underlying theory (concrete-problems, §"Define an Impact Regularizer:", p. 5; §"Reward Uncertainty:", p. 7; §"How to respond when out-of-distribution.", p. 18).
Paired with reachability analysis
The concept's role in the corpus is defined almost entirely by its pairing with Reachability analysis. The edge Impact regularizer (defined) — borrows its improvement guarantee from → Robust policy improvement describes the paper's proposal to replace a naive null baseline with "a known safe... but suboptimal policy, and then seek to improve the policy from there, somewhat reminiscent of reachability analysis... or robust policy improvement" (concrete-problems, §"Define an Impact Regularizer:", p. 5): reachability analysis defines the boundary of acceptable states, while robust policy improvement licenses movement away from the passive baseline within that boundary without sacrificing the safety guarantee. The companion edge Reachability analysis — supplies the safe set that bounds → Robust policy improvement makes the division of labor explicit — a reachable-set computation with nothing to act on is useless, just as a robustness guarantee with no defined safe region has nothing to guarantee performance within. The concept belongs to the Penalizing side effects by distance from a baseline and Formal verification and control theory recur as safety building blocks themes, sits in the A conservative baseline computed by one mechanism becomes another mechanism's literal boundary, Before the fact or after: when a safety property gets enforced, and A safety mechanism built by patching together other people's guarantees connective themes, and spans the Bounding what an agent can change or explore, not just what it's told to want, Where Concrete Problems draws its own lines, and Distributional shift: relax the statistics, or rewrite the contract superthemes.