The paper names exactly where its own fix stops working
Most papers let a critic find the hole in their own proposal. This one names the hole itself, at every level of ambition, before anyone has to ask: a provably correct fix for partially observed goals turns out to be too complicated to actually use.
Formal verification's admitted blind spot toward modern machine learning comes next, followed by wireheading's structural cause, which can only be mitigated, never removed. Careful engineering's own concession, that it cannot catch every bug, closes the page, exactly the gap trip wires exist to backstop.
Concrete Problems is unusually candid about where its proposed remedies run out, at every level of ambition, and the 2017 paper turns out to share the same habit of self-diagnosis. Partially observed goals has an impractical theoretical fix in the POMDP belief-state MDP: provably correct, and admitted to be too complicated to use. Formal verification supplies the rigor Asimov's first law lacked, but the same section concedes formal verification "has not focused on" modern ML. Environmental embedding is named the structural cause of wireheading precisely because it cannot be designed away, only mitigated. Reward pretraining is flagged as a concrete special case of the aspirational variable indifference, working only because the reward function is simple enough to fix in advance. bAbI is offered as a benchmarking template for safe exploration with the disanalogy stated outright: scoring absence of catastrophe is harder than scoring correctness. Careful engineering admits it "cannot catch every possible bug," which is exactly the gap trip wires are built to backstop by detection; and reward capping shares a blind spot with multiple rewards, the same low-effort exploit defeating both at once. The 2017 paper concedes limits at three more points, each named by its own authors rather than found by a critic. The recurrent reward model extension would generalize the fixed window every actual reward model in the paper uses, a stack of at most four frames standing in for full episode history, but the paper flags the recurrent version as unimplemented, a footnote rather than a method any reported result relies on. Uncertainty-based query selection is called out by name as "a crude approximation" of the expected-value-of-information criterion the authors regard as correct but leave to future work, and their own ablations show the crude version can underperform querying at random. And clip-length effects predicts but was never tested on the Qbert task: the paper attributes Qbert's outright failure to short, confusing clips, a mechanism it verified only on the continuous-control tasks, extended to a discrete-domain failure with a bare "this may be because" rather than a matching ablation.