The mechanics that make PPO fine-tuning work — engineers a partial countermeasure for → Where RLHF alignment still falls short

Two of this theme's engineering choices are named, deliberate responses to two of the failure modes the other theme catalogues. The per-token KL penalty is added specifically to limit reward-model over-optimization, the RL policy exploiting reward-model imperfections rather than genuinely improving. PPO-ptx, mixing pretraining language-modeling gradients into the PPO update, is added specifically to counteract the alignment tax, the regressions on public NLP benchmarks that RLHF otherwise introduces (instructgpt, §3.5 Models, p. 9; §4.2 Results on public NLP datasets, p. 14). No comparable mechanism among this theme's concepts targets the other three failure modes: hallucination, excessive hedging, and false-premise failure have no PPO-stage engineering response at all, attributed instead to properties of the training data and reward-model incentives. The mechanics theme's toolkit is therefore a partial, not general, answer to what the failure-modes theme documents.