Keeping exploration inside a known-recoverable region
If an agent might do something catastrophic while exploring, one answer is to never let it explore that far. This theme covers the mechanisms Concrete Problems (2016) proposes for fencing off unsafe territory: learning in simulation before acting for real, staying inside states known to be recoverable, or requiring a human's sign-off first.
Bounded exploration gives the clearest version of the idea, checking each candidate action against a model before the agent may take it, and the theme builds from there to the oversight layers stacked on top.
Where the previous theme covers exploration objectives, this one covers the paper's proposed mechanisms for physically constraining where an agent is allowed to explore. Simulated exploration moves risk into a simulated environment first, so danger is learned about before a more conservative policy is deployed in the real world. Bounded exploration restricts an agent to regions of state space known to be safe or recoverable, checking via a model whether a candidate action would exit that region, a strategy related to H-infinity control, a prior robust-control framework for minimizing worst-case response to bounded disturbances. Trusted policy oversight and human oversight add a second layer of gatekeeping, limiting exploration to actions a trusted policy and model predict are recoverable, or requiring a human to sign off on potentially unsafe actions before they are taken. Use of demonstrations reduces the need for risky exploration altogether by learning an initial baseline policy from expert trajectories, around which further exploration can then be bounded. The shared claim: safe exploration is achievable by shrinking the space an agent is permitted to search, not just by reweighting its objective.