Exploration policies and the objectives that make them safe or reckless
Standard exploration policies were never built with catastrophe in mind; they just try things and update. This theme sets out that baseline, epsilon-greedy and R-max among them, before turning to what Concrete Problems (2016) argues is missing: objectives that treat rare, irreversible bad outcomes as different in kind from ordinary bad luck.
Risk-sensitive performance criteria carry the theme, the paper's proposed replacement for expected-value optimization, and the bAbI tasks close it out as a model for the benchmark suite safe exploration still lacked.
Safe exploration is the problem of ensuring an RL agent's exploratory actions do not lead to catastrophic or irrecoverable outcomes even when it lacks full knowledge of its environment. This theme groups the baseline exploration machinery against which safety proposals are measured. Epsilon-greedy and R-max are standard prior exploration policies, one taking random actions with fixed probability, the other treating unexplored actions optimistically to guarantee near-optimal learning, that have no notion of catastrophe built in. Risk-sensitive performance criteria name the family of alternative objectives, replacing expected total reward with worst-case guarantees, bounded probabilities of very bad outcomes, or variance penalties, that the paper argues are needed to prevent rare catastrophic events that expected-value optimization would tolerate. The bAbI tasks, a benchmark suite of diagnostic toy question-answering problems, are cited as a model for the kind of broad, systematic test suite the paper envisions building for safe exploration research specifically. Together these concepts define the baseline and the missing evaluation infrastructure against which safe-exploration remedies must be judged.