Simulated exploration
Concrete Problems in AI Safety — introduced
A proposed remedy shifting exploratory risk into simulated environments, learning about danger there before deploying a more conservative policy in the real world.
One way to make dangerous learning safe is to move it somewhere consequence-free. Simulated exploration, a remedy proposed in Concrete Problems in AI Safety (2016), has the agent meet danger inside a simulation first, so that a more cautious policy can be brought out into the real world.
The sections below take the idea in turn: moving exploratory risk into simulation, what the paper admits that move does not solve, and where it sits among the other exploration remedies.
Moving exploratory risk into simulation first
Simulated exploration is a remedy Concrete Problems introduces for Safe exploration: the more exploration can happen in a simulated environment rather than the real world, the less opportunity there is for catastrophe, since danger can be learned about in simulation and a more conservative policy adopted only once the agent acts in reality (concrete-problems, §"Simulated Exploration:", p. 15). The paper notes that training RL agents, particularly robots, in simulated environments was already common practice, so this proposal is framed as an incremental push, "exploration-focused simulation," on infrastructure that already existed rather than something requiring a new pipeline.
What the paper admits it does not solve
The proposal is bounded by an honest limitation: some real-world exploration will probably always be necessary, since many complex situations cannot be perfectly captured by a simulator. The paper raises this as an open research direction rather than a solved handoff: in systems that involve a continual cycle of learning and deployment, there are interesting open problems in how to safely and incrementally update a policy given simulation-based trajectories that only imperfectly represent the consequences of that policy in the real world, alongside reliably accurate off-policy real-world trajectories, a combination the paper terms "semi-on-policy" evaluation (concrete-problems, §"Simulated Exploration:", p. 15).
Its place among the paper's other exploration remedies
Simulated exploration belongs to the theme Keeping exploration inside a known-recoverable region alongside Bounded exploration, Trusted policy oversight, Human oversight (safe exploration), and Use demonstrations (safe exploration), the paper's proposed mechanisms for physically limiting where an agent is allowed to explore. Where bounded exploration and trusted-policy oversight constrain exploration within the real environment, simulated exploration constrains it by relocating exploration outside the real environment entirely, a complementary rather than competing strategy within the same broader theme.