Bounded exploration
Concrete Problems in AI Safety — introduced
A proposed remedy that restricts an agent to regions of state space known to be safe or recoverable, using models to check whether an action would exit that safe region.
Suppose some region of the world is known to be forgiving, a place where even the worst action can be recovered from. Bounded exploration, proposed in Concrete Problems in AI Safety (2016), lets an agent roam freely inside such a region and uses models to check any step that would leave it.
Down the page: shrinking the space rather than the objective, the worst-case framing borrowed from robust control, and the question of who does the checking and what it needs to get started.
A remedy for safe exploration: shrink the space, not the objective
Bounded exploration is one of several remedies Concrete Problems proposes for the Safe exploration problem, introduced with the idea that if a certain portion of state space is known to be safe, in the sense that "even the worst action within it can be recovered from or bounded in harm," an agent can be let run freely inside that region without further checks (concrete-problems, §"Bounded Exploration:", p. 15). When a model of the environment is available, the check can be made prospective: extrapolate a candidate action forward and ask whether it would carry the agent outside the safe region before taking it. The paper offers two ways to formalize "safe": remaining within an ergodic region of state space where actions are reversible, or bounding the probability of incurring a very large negative reward below some small threshold (concrete-problems, §"Bounded Exploration:", p. 15).
Borrowed framing: worst-case guarantees over a bounded disturbance
The paper flags bounded exploration as "related to" H-infinity control without spelling out the analogy (concrete-problems, §"Bounded Exploration:", p. 15); the underlying link is structural: both frame safety as a worst-case guarantee over a bounded set of futures rather than an average-case one. H-infinity control asks what policy minimizes the worst possible deviation from desired behavior given that disturbances are bounded in magnitude; bounded exploration asks the reinforcement-learning analogue, given a region where even the worst action is recoverable, can the agent act freely inside it. Both trade generality, the bound must be known in advance, for a hard safety floor that holds no matter which action within the bound is actually taken.
Who does the checking, and what it needs to get started
Bounded exploration states the recoverability requirement abstractly; two other remedies supply pieces it needs. Trusted policy oversight operationalizes the check itself, delegating the recoverability judgment to a separate, presumably more conservative policy paired with an environment model rather than folding it into the exploring agent (concrete-problems, §"Trusted Policy Oversight:", p. 15). And bounded exploration needs some prior notion of a safe region to bound around in the first place; use of demonstrations supplies exactly that, since a baseline policy learned from expert trajectories can itself define the region exploration is bounded around, letting an agent both start safer, by imitating a baseline, and stay safer, by exploring only nearby (concrete-problems, §"Use Demonstrations:", p. 15). All three concepts sit together in the theme Keeping exploration inside a known-recoverable region, alongside Simulated exploration and Human oversight (safe exploration), the paper's proposed mechanisms for physically limiting where an agent is allowed to explore rather than only reshaping its objective.