Reachability analysis
Concrete Problems in AI Safety — inherited
A prior control-theoretic technique for determining which states a system can reach or safely return to from a given policy, used repeatedly in the paper as a building block for conservative baselines.
Control theorists have long asked which states a system can reach -- or safely return to -- from where it stands. Reachability analysis is that technique, developed over decades for hybrid and cyber-physical systems and borrowed here rather than invented.
The computation itself is explained once; what matters more is how it became the corpus's most reused building block, propping up baselines for the impact regularizer and for robust policy improvement alike.
Computing which states remain safe
Reachability analysis is a control-theoretic technique, developed well before this paper across decades of work on hybrid and cyber-physical systems, for computing which states a dynamical system can reach — or, symmetrically, which states it can still safely return to — under a given policy and dynamics model. Formally, given an initial set $X_0$, a dynamics model, and a set of admissible controls, the reachable set at time $t$ is $$\mathrm{Reach}(t) = \{\, x(t) : x(0) \in X_0,\; \text{control trajectory admissible} \,\},$$ and a system is deemed safe if this set avoids some designated unsafe region for all $t$ in the horizon of interest; the technique underlies real engineering successes such as formally verified collision-avoidance systems. Concrete Problems imports it, unmodified, as a building block that several of its own proposed safety mechanisms lean on, citing it three separate times across different sections of the paper (concrete-problems, §"Define an Impact Regularizer:", p. 5; §"How to respond when out-of-distribution.", p. 18; §"A unifying view: counterfactual reasoning and machine learning with contracts.", p. 19).
The corpus's most reused building block
Few imported concepts in this batch do as much structural work. The Impact regularizer (defined) borrows its notion of a safe fallback directly from reachability analysis — the edge Impact regularizer (defined) — borrows its safe region from → Reachability analysis shows that once a naive state-distance penalty is rejected for resisting all change, reachability analysis supplies exactly the missing piece: a formal, principled account of which states remain safely recoverable, letting "known safe but suboptimal policy" mean something precise. Paired with Robust policy improvement in Reachability analysis — supplies the safe set that bounds → Robust policy improvement, the two techniques divide labor cleanly: reachability analysis answers a set-membership question (which states are reachable or recoverable), while robust policy improvement answers an optimization question (how to improve a policy while guaranteeing performance inside that boundary) — neither alone produces a usable conservative baseline. Against H-infinity control in H-infinity control — reaches the same envelope by a different route than → Reachability analysis, the two are shown to compute a similar conservative operating envelope via different mathematics, a set-membership computation versus a minimax optimization over bounded disturbance, and the paper treats them as substitutable rather than combined. And Model repair supplies its ex-post counterpart in Model repair — is the ex post counterpart to → Reachability analysis: reachability analysis constrains search during optimization, model repair fixes violations after it. Across these four edges the concept sits in the A conservative baseline computed by one mechanism becomes another mechanism's literal boundary, Before the fact or after: when a safety property gets enforced, Two or more remedies for the identical problem, sorted by where they intervene, and A safety mechanism built by patching together other people's guarantees connective themes, belongs to the Penalizing side effects by distance from a baseline and Formal verification and control theory recur as safety building blocks themes, and spans the Bounding what an agent can change or explore, not just what it's told to want, Where Concrete Problems draws its own lines, and Distributional shift: relax the statistics, or rewrite the contract superthemes — a single imported tool doing conservative-baseline work for side effects, safe exploration, and distributional robustness alike.