Safe exploration
Concrete Problems in AI Safety — refined
The problem of ensuring that an RL agent's exploratory actions do not lead to catastrophic or irrecoverable outcomes, even when the agent lacks full knowledge of the environment.
Learning by trial and error means trying things, and some tries can be ruinous -- a robot experimenting near a staircase, a mistake that cannot be undone. Safe exploration asks how an agent can keep exploring without courting catastrophe.
Placed among the paper's five core problems, safe exploration then opens onto the techniques it inspired, roughly split between avoiding danger outright and tolerating it carefully, before closing on a benchmark the paper admits does not yet exist.
One of the paper's five core problems
Safe exploration is one of Concrete Problems' five headline research problems, placed in the category of accidents that arise not from a wrong objective function but from something going wrong during the learning process itself (concrete-problems, §"2 Overview of Research Problems", p. 3). The paper states the problem generally: exploration, taking actions that do not seem ideal given current information but that help an agent learn about its environment, is necessary for any autonomous learning agent, but it can be dangerous, since it means taking actions whose consequences are not yet understood. In a toy Atari environment the worst outcome of bad exploration is a lost life or some lost score; in the real world, badly chosen exploratory actions can destroy the agent, trap it in unrecoverable states, or damage property and infrastructure (concrete-problems, §"6 Safe Exploration", p. 14). The paper notes this is "arguably the most studied" of its five problems and points to existing survey literature rather than starting from scratch.
Eight children, two families
Safe exploration's children split cleanly into a baseline family and a remedy family. The baseline family names the machinery the paper argues is inadequate on its own: Epsilon-greedy exploration and R-max are standard exploration policies that make no attempt to avoid dangerous situations, and Risk-sensitive performance criteria names the alternative objectives, worst-case guarantees, bounded probabilities of catastrophe, variance penalties, that the paper argues are needed instead of plain expected-value optimization. The remedy family proposes concrete ways to keep exploration safe: using demonstrations to learn a baseline policy and avoid risky exploration altogether; shifting exploration into simulation before deploying a more conservative policy in the real world; bounding exploration to regions of state space known to be recoverable; and gatekeeping candidate actions through either a trusted policy or a human before they are taken. Together these eight children cover essentially the full space of the paper's proposed responses to one problem.
A benchmark the paper says does not yet exist
Safe exploration's single direct edge points outward rather than to a remedy: the bAbI tasks, a benchmark suite that decomposes language understanding into small diagnostic subtasks, is invoked as a structural template for a benchmark safe-exploration research still lacks, a broad suite of toy environments covering conceptually distinct catastrophe types, physical and abstract, against which a single architecture's coverage could be scored (concrete-problems, §"6 Safe Exploration", p. 16). The paper flags the disanalogy honestly: bAbI scores correctness of an output, while the envisioned suite would have to score the absence of a bad outcome, a harder thing to benchmark since an agent can trivially avoid catastrophe by avoiding action altogether. Safe exploration anchors the theme Exploration policies and the objectives that make them safe or reckless and reaches into Keeping exploration inside a known-recoverable region through its remedy children, making it, along with Negative side effects (avoiding), one of the two most structurally central problems in this part of the corpus.