Bounding what an agent can change or explore, not just what it's told to want
Getting the reward right is not enough if an agent can still wander somewhere it shouldn't. Concrete Problems answers two different worries, unwanted side effects and reckless exploration, with the same underlying move: instead of perfecting what the agent wants, shrink what it is allowed to reach.
Side effects get one set of tools, penalties measured against a baseline and against potential influence; exploration gets another, risk-sensitive objectives paired with recoverable-region constraints, and formal verification threads underneath both. Notice especially the theme that reframes side effects entirely, not as damage to the world but as a relationship owed to other agents.
Concrete Problems' negative-side-effects and safe-exploration problems share a strategy: instead of perfectly specifying a reward, shrink the space of outcomes or actions an agent may reach. Penalizing side effects by distance from a baseline measures collateral change against a passive reference; limiting potential influence instead of measuring realized change offers an information-theoretic alternative to that same restraint; and side effects and control as a relationship with other agents reframes the whole problem as protecting other agents' interests. On exploration, exploration policies and the objectives that make them safe or reckless sets out the risk-sensitive criteria, while keeping exploration inside a known-recoverable region supplies the mechanisms, simulation, bounded regions, trusted oversight, and demonstration-learned baselines. Learning from demonstrations instead of a specified reward is the concrete technique both this exploration theme and the relational side-effects theme reach for, later resurfacing as InstructGPT's SFT stage. Formal verification and control theory recur as safety building blocks documents the shared toolkit, reachability analysis, robust policy improvement, H-infinity control, underneath both problem areas.