Trusted policy oversight
Concrete Problems in AI Safety — introduced
A proposed remedy that limits exploration to actions a trusted policy and environment model predict the agent can recover from.
A learner can be chaperoned by a policy that is already trusted: before an exploratory action is taken, the trusted policy and a model of the environment judge whether the agent could recover from it. Concrete Problems in AI Safety (2016) proposes this arrangement as trusted policy oversight.
The idea unfolds in stages: delegating the recoverability check to a second policy, one concrete way to run that abstract check, and the case for a machine-speed alternative once a human bottleneck stops scaling.
Delegating the recoverability check to a second policy
Trusted policy oversight is a remedy Concrete Problems introduces for Safe exploration: if a trusted policy and a model of the environment are available, exploration can be limited to actions the trusted policy believes are recoverable, illustrated by the paper's own example, "it's fine to dive towards the ground, as long as we know we can pull out of the dive in time" (concrete-problems, §"Trusted Policy Oversight:", p. 15).
One concrete way to run a more abstract check
Bounded exploration states the general safety requirement abstractly, an agent should stay within a region where even the worst action is recoverable or bounded in harm, checked by extrapolating a model forward to see whether a candidate action would exit that region. Trusted policy oversight operationalizes that check concretely: instead of the exploring agent verifying recoverability against its own model, a separate trusted policy paired with an environment model makes the call. The difference is who does the checking, delegating the safety judgment to a second, presumably more conservative, policy rather than folding it into the exploring agent itself.
A machine-speed alternative to a human bottleneck
The paper introduces human oversight first and immediately flags its limit: an agent may generate exploratory actions too numerous or too fast for a human to review, running into the paper's own scalable-oversight problem. That scalability limit motivates trusted policy oversight as the alternative: replace the human reviewer with a policy that can render a recoverability judgment at machine speed, trading away whatever richer, harder-to-formalize judgment a human reviewer could bring for throughput. Trusted policy oversight belongs to the theme Keeping exploration inside a known-recoverable region alongside Simulated exploration, Bounded exploration, and Human oversight (safe exploration), the paper's cluster of proposals for physically limiting where an agent is permitted to explore.