Explore

Papers

The corpus, paper by paper. Each paper opens into a page of its own: the paper told as stories, and every concept it uses in reading order.

Superthemes

Eight superthemes, each grouping several themes.

• Bounding what an agent can change or explore, not just what it's told to want
  • Penalizing side effects by distance from a baseline
  • Limiting potential influence instead of measuring realized change
  • Side effects and control as a relationship with other agents
  • Exploration policies and the objectives that make them safe or reckless
  • Keeping exploration inside a known-recoverable region
  • Formal verification and control theory recur as safety building blocks
  • Learning from demonstrations instead of a specified reward
• Constitutional AI's method: a written constitution, AI feedback, and its place among prior agents
  • Constitutional AI's method: principles, critique, and revision
  • Engineering a language model into a usable preference labeler
  • Sourcing and classifying harmful prompts
  • Prior dialogue-agent systems Constitutional AI positions itself against
  • Helpful, honest, harmless: a shared definition of alignment
• Deep RL from Human Preferences: the 2017 proof of concept
  • Comparisons as the atomic unit of human feedback
  • Keeping the reward model on-distribution
  • The economics of human feedback
  • Hiding the reward: experimental design for preference-only learning
  • The 2017 deep-RL testbeds: MuJoCo and Atari
  • Novel behaviors without reward functions
  • Policy optimization under a learned, changing reward
  • Prior human-in-the-loop RL and why it didn't scale
• Defining the target before optimizing for it: instruction-following, HHH, and the tradeoff it creates
  • Instruction following as an alternative to next-token prediction
  • Helpful, honest, harmless: a shared definition of alignment
  • The helpfulness-harmlessness tradeoff and CAI's claim to beat it
  • Automatic overlap scores versus human and preference judgments
• Distributional shift: relax the statistics, or rewrite the contract
  • Weakening statistical assumptions to survive distribution shift
  • Reframing distributional shift as broken contracts and causal structure
  • Formal verification and control theory recur as safety building blocks
• Reward hacking: how proxies get gamed, and the general law behind it
  • Why reward proxies get gamed: the anatomy of reward hacking
  • Borrowing adversarial ML to defend against reward hacking
  • Non-adversarial engineering fixes for reward hacking
  • Goodhart's law across the corpus: when the proxy becomes the target
• Scalable oversight made real: from 2016 proposals to InstructGPT's RLHF pipeline
  • Learning from demonstrations instead of a specified reward
  • Scaling oversight by predicting reward on unlabeled episodes
  • Scaling oversight without a learned reward predictor
  • Comparisons as the atomic unit of human feedback
  • Keeping the reward model on-distribution
  • The economics of human feedback
  • From an open problem to reward models to AI feedback
  • InstructGPT's three-step training pipeline
  • The mechanics that make PPO fine-tuning work
  • Building the human side of the RLHF pipeline
  • Where RLHF alignment still falls short
  • Three levers on one model: prompting, finetuning, steering
  • Base versus chat: what RLHF changes inside
  • Finetuning moves the persona, measurably
• Steering the model from inside: activation engineering and what it reveals
  • A behavior becomes a direction in activation space
  • The activation engineering family: from single pairs to datasets
  • Seven alignment worries become seven dials
  • Writing behavior down as A/B questions
  • Three levers on one model: prompting, finetuning, steering
  • Evaluating steering: behavior scores and capability floors
  • Steering doubles as interpretability
  • Base versus chat: what RLHF changes inside
  • Elicitation as audit: steering for red-teaming
  • Steering grows out of inference time
  • Projection as early warning: reading the persona before it speaks
• The evaluation apparatus: measuring quality, capability loss, and safety together
  • Automatic overlap scores versus human and preference judgments
  • The public NLP benchmark suite used to catch regressions
  • Testing generative capability: translation and summarization benchmarks
  • Operationalizing truthfulness and social bias into measurable benchmarks
• The persona under management: deployment, training, and the data before it
  • From trait name to vector: extraction becomes automatic
  • The Assistant persona taken apart into measurable traits
  • Projection as early warning: reading the persona before it speaks
  • Finetuning moves the persona, measurably
  • Steering grows out of inference time
  • Catching bad data before it trains anything
  • Open models as subjects, frontier models as instruments
  • To study a failure, first cause it: datasets built to corrupt
• Where Concrete Problems draws its own lines
  • Framing safety as accidents, not superintelligence
  • Drawing the boundary: what accident risk is not
  • Formal verification and control theory recur as safety building blocks

Connective themes

Twenty-eight connective themes, each a short thread of edges.

• 2017's ad hoc parts become 2022's industrial ones (12 edges)
• A behavior is exactly as good as the dataset that writes it down (9 edges)
• A conservative baseline computed by one mechanism becomes another mechanism's literal boundary (5 edges)
• A decades-old, informal idea resurfaces as an unsolved precursor (6 edges)
• A footnote or related-work list implies two things are equivalent; they aren't (5 edges)
• A primary metric, and a second instrument built to catch what it can't (18 edges)
• A safety mechanism built by patching together other people's guarantees (8 edges)
• A small model wins, and the paper shows exactly which rung the win comes from (9 edges)
• An old benchmark, reused, no longer measuring what it used to (9 edges)
• Before the fact or after: when a safety property gets enforced (5 edges)
• Borrowed machinery that doesn't fully transfer, and says so (5 edges)
• Connections the source papers never draw, visible only by reading them together (23 edges)
• Control migrates from training time into the forward pass (10 edges)
• Corruption as a controlled variable (5 edges)
• Each design choice answers a named predecessor's limit (6 edges)
• Each pipeline stage's output is the next stage's engineered raw material (12 edges)
• Engineering the exact prompt or format needed to manufacture a model behavior (11 edges)
• Everything downstream of a moving reward (7 edges)
• From found jailbreaks to systematic elicitation (5 edges)
• Goodhart's law gets a name, one instance at a time (7 edges)
• How much a single worked example, or a single row of data, is really doing (4 edges)
• Human fingerprints in the learning curves (9 edges)
• Named worries become measurable dials (9 edges)
• One advertised dial, five interacting knobs (5 edges)
• One dot product, three jobs (5 edges)
• One paper's central result, traced edge by edge (5 edges)
• One result, ringed by its controls: the preventative-steering argument (7 edges)
• One umbrella idea, aimed at different targets each time it's used (5 edges)
• Persona drift becomes a measurement (5 edges)
• Proven near-perfect at a target the paper argues is itself the wrong one (3 edges)
• Replacing the human in the loop at exactly the point it was the bottleneck (11 edges)
• The ablation is the argument (7 edges)
• The paper draws an explicit line around its own territory, and names what falls outside (4 edges)
• The paper names exactly where its own fix stops working (10 edges)
• The pipeline looks like a forward sequence; the causality doesn't run that way (8 edges)
• The same technique, cited or deployed twice, doing different work each time (10 edges)
• The steering vector doubles as a measuring instrument (12 edges)
• The training objective is a small system of knobs, not one number (6 edges)
• Turning an abstract oversight worry into dials and pools that can run together (10 edges)
• Two or more remedies for the identical problem, sorted by where they intervene (10 edges)
• Two things built identically, pointed in opposite directions (9 edges)
• What comparison training leaves undetermined (6 edges)
• When a benchmark stops measuring and starts being optimized against (5 edges)
• When a bug is traced to the labels, the repair goes to the labels too (5 edges)

Concepts A–Z

#

• "EM-like" (emergent-misalignment-like) finetuning datasets
• "Let's think step-by-step" zero-shot CoT trigger

A

• Absolute Harmfulness Score
• Abstract rewards (reward hacking cause)
• Abuse (of ML systems)
• Accidents in machine learning systems
• Activation Addition (ActAdd)
• Activation engineering / representation engineering (technique family)
• Adaptive L2 Regularization for the Reward Predictor
• Advanced AI Risk evaluation dataset
• Advantage Actor-Critic (A2C/A3C)
• Adversarial blinding
• Adversarial examples
• Adversarial reward functions
• Alignment tax
• Ant (MuJoCo task)
• Apathy (persona trait)
• API prompt distribution
• Apprenticeship learning
• Asimov's first law of robotics
• Asynchronous Reward-Learning Architecture
• Atari games (as an RL testbed)
• Automated concept-description-to-direction pipeline (Wu et al., 2025)
• Automated persona vector extraction pipeline
• Automated Red Teaming

B

• bAbI tasks
• Bandit environment formulation for RLHF
• Base vs. RLHF-chat model representation comparison
• Behavioral clustering
• BIG-Bench
• BLEU
• Bounded exploration
• Bradley-Terry Model

C

• CAA as an adversarial red-teaming tool (proposed application)
• CAA combined with supervised finetuning comparison
• CAA combined with system-prompting comparison
• CAA steering vector construction (Mean Difference over multiple-choice contrast pairs)
• CAA vs. finetuning computational cost comparison
• CAFT (Concept Ablation Fine-Tuning)
• Careful engineering (reward hacking remedy)
• Chain-of-Thought (CoT) Prompting
• Chain-of-Thought Prompts for RL-CAI (Appendix E.2)
• Claude Sonnet as pipeline/data-generation tool
• Cleaning robot (running example)
• Clip-Length Effects on Human Evaluation
• CNN/Daily Mail summarization
• Coherence score
• Comparisons vs. Absolute Scores (Target Ablation)
• Complicated systems (reward hacking cause)
• Compute vs. Human Feedback Cost Analysis
• Constant Rater-Error Noise Model
• Constitution (set of principles)
• Constitutional AI (CAI)
• Context Distillation
• Contractor Preference-Labeling Protocol
• Contrastive Activation Addition (CAA)
• Cooperative Inverse Reinforcement Learning (CIRL)
• Coordination with Other AIs (AI Coordination)
• Corrigibility
• Counterexample resistance
• Counterfactual reasoning
• Covariate shift assumption
• Critique and Revision
• Cross-trait persona correlation and vector similarity analysis
• CrowS-Pairs
• Custom hallucination contrastive dataset
• Custom refusal contrastive dataset

D

• Deep Q-Network (DQN)
• DeepDive
• Delusion box
• Direct Revision (critique-free ablation)
• Distant supervision
• DROP (Discrete Reasoning Over the content of Paragraphs)

E

• Elo Score (helpfulness/harmlessness)
• Emergent misalignment
• Emotion vectors (Dong et al., 2025)
• Empowerment
• Enduro (Atari game)
• Enduro Keeping-Pace Demonstration
• Entropy-based bias metric
• Environment Modifications for Preference Learning
• Environmental embedding (reward hacking cause)
• Epsilon-greedy exploration
• Evasiveness
• Evil (persona trait)
• Evolved timekeeping-circuit-turned-radio example
• Excessive hedging on simple questions
• Expected-Value-of-Information Query Selection

F

• F1 score (span-overlap QA metric)
• Fact-acquisition case study
• Failure to detect false-premise instructions
• Fairness (in ML)
• Feedback loops (reward hacking cause)
• Feedback Model
• Feudal reinforcement learning
• Few-shot prompting
• Few-shot Prompts for SL-CAI (Appendix E.1)
• Finetuning shift (activation-shift metric)
• FLAN (instruction-tuned baseline)
• Formal verification (of cyber-physical systems)
• Frame problem

G

• Generalized advantage estimation (GAE)
• Generalized method of moments
• Generative adversarial networks (GANs)
• Goodhart's law
• GPT-3 (pretrained language model)
• GPT-3 (prompted) baseline
• GPT-4
• GPT-4.1-mini

H

• H-infinity control
• Half-Cheetah (MuJoCo task)
• Half-Cheetah One-Leg Demonstration
• Hallucination (closed-domain fabrication)
• HaluEval
• Harmful Behavior Identification/Classification Evaluations (Appendix B)
• Held-out labeler generalization test
• HellaSwag
• Helpful RLHF Model
• Helpful, honest, harmless (HHH) alignment framework
• Helpfulness-Harmlessness Tension
• HH RLHF Model
• HHH Evaluation Set (binary comparisons)
• Hierarchical reinforcement learning (HRL)
• Hopper (MuJoCo task)
• Hopper Backflip Demonstration
• Human oversight (safe exploration)
• Human-Feedback Implicit Reward Shaping
• Human-Feedback Sample Efficiency
• Humor (persona trait)
• Hybrid Human/AI Preference Model

I

• Imitation Learning
• Impact regularizer (defined)
• Impoliteness (persona trait)
• In-Context Vectors (ICV)
• Inference-Time Intervention (ITI)
• InstructGPT
• Instruction following (as a fine-tuning objective)
• Inter-labeler agreement rate
• Inter-layer steering vector similarity and transfer
• Inverse reinforcement learning (IRL)
• Iterated (Online) Training

J

• Jailbreaks

K

• K-choose-2 single-batch comparison training

L

• Label Annealing
• Labeler metadata annotation taxonomy
• Labeler screening and selection process
• LaMDA
• Language modeling (next-token prediction) objective
• Learned impact regularizer
• Letter clustering
• Likert overall-quality rating
• Linear representation hypothesis
• Linear representations of sentiment
• Llama 2
• Llama 2 Chat
• Llama-3.1-8B-Instruct
• LMSYS-Chat-1M
• Long-term / superintelligence AI-risk framing
• Low-impact agents

M

• Machine learning with contracts
• Many-shot prompting (as a persona-elicitation method)
• Mean Difference (MD) vector extraction
• MMLU (Massive Multitask Language Understanding)
• Model Calibration (well-calibrated multiple-choice probabilities)
• Model lookahead
• Model repair
• MuJoCo
• Multi-agent approaches (to side effects)
• Multi-layer steering with layer-incremental vectors
• Multiple rewards
• Multiple-choice behavioral evaluation and layer sweep
• Multiple-Choice Comparison Format
• Multiple-choice contrast-pair format
• Myopic Reward

N

• Negative side effects (avoiding)
• Non-Stationary Reward Function Challenge
• Novel Behavior Training

O

• Open-ended generation evaluation with GPT-4 rating
• OpenAI Gym
• Optimism (persona trait)

P

• PALMS (Process for Adapting Language Models to Society)
• Pareto Improvement (helpfulness vs. harmlessness)
• Partially observed goals
• Partially specified models
• PCA visualization of contrastive activations
• Penalize influence
• Per-token KL penalty from the SFT model
• Persona vector
• Personality-trait vector space (Allbert et al., 2025)
• Perspective API
• Policy (economic/social impacts of ML)
• POMDP-to-belief-state-MDP reduction
• Pong (Atari game)
• Post-hoc (inference-time) steering mitigation
• PPO (without pretraining mix)
• PPO dataset
• PPO fine-tuning stage (RLHF policy optimization)
• PPO value function
• PPO-ptx (pretraining gradient mixing)
• Preference Elicitation Protocol
• Preference Label Clamping (Soft vs. Hard vs. Clamped)
• Preventative prompting
• Preventative steering
• Principal Component Analysis (PCA)
• Principle Ensembling
• Principles for RL-CAI (Appendix C.2)
• Principles for SL-CAI (Appendix C.1)
• Privacy (in ML)
• Projection difference (pre-finetuning data-screening metric)
• Projection-based monitoring of prompt-induced persona shifts

Q

• Qbert (Atari game)
• QuAC (Question Answering in Context)
• Quantitative vs. Qualitative Evaluation Methodology
• Qwen2.5-7B-Instruct

R

• R-max
• Reachability analysis
• Real-world chat data validation of persona vectors
• RealToxicityPrompts
• Recurrent Reward Model (Proposed Extension)
• Red Teaming
• Reddit TL;DR summarization
• Refusal (as a steerable behavior)
• Reinforcement learning from human feedback (RLHF)
• Representation Engineering (Zou et al. 2023)
• Residual stream
• Response-conditioned contrastive vector computation
• Reward autoencoder (goal transparency)
• Reward capping
• Reward hacking (avoiding)
• Reward model (RM)
• Reward model over-optimization
• Reward model pairwise comparison loss
• Reward Prediction Normalization
• Reward Predictor Ensemble
• Reward Predictor Pretraining
• Reward pretraining
• Reward uncertainty (side-effect mitigation)
• Risk-sensitive performance criteria
• RL from AI Feedback (RLAIF)
• RL-CAI (RLAIF-trained Constitutional AI model)
• RM (comparison) dataset
• Robust policy improvement
• Robustness to distributional shift
• ROUGE-L
• RTE (Recognizing Textual Entailment)

S

• SAE decomposition of persona vectors
• Safe exploration
• Sample-level projection-based data filtering
• Scalable oversight
• Security (attacks against ML systems)
• Security vectors (Zhou et al., 2024)
• Semi-supervised or active reward learning
• Semi-supervised reinforcement learning
• SFT dataset
• Shutdown / off-switch problem
• Simulated exploration
• SL-CAI (Supervised-Learning Constitutional AI model)
• Sparrow
• Sparse autoencoders (SAEs)
• SQuAD v2
• SST (Stanford Sentiment Treebank)
• Steering at targeted token positions (proposed extension)
• Steering multiplier
• Steering outside the residual stream (proposed extension)
• Steering vector
• Steering vector normalization choices
• Steering-based mitigation of finetuning-induced persona shifts
• SuperGLUE
• Supervised fine-tuned (SFT) model
• Supervised fine-tuning (SFT)
• Supervised reward learning
• Survival Instinct
• Sycophancy
• Sycophancy dataset (Nishimura-Gasparian et al., 2024)
• Sycophancy on NLP Survey / Sycophancy on Political Typology datasets
• Synthetic Oracle Feedback
• System-prompting

T

• T0 (instruction-tuned baseline)
• TAMER Framework
• Token-level cosine similarity analysis
• Train-time projection-regularization baseline
• Training on multiple distributions
• Trait expression score (LLM judge)
• Trait-eliciting finetuning datasets (Evil, Sycophancy, Hallucination)
• Trait-specific artifact generation
• Trajectory Segment
• Transparency (in ML)
• Trip wires
• Trust Region Policy Optimization (TRPO)
• Trusted policy oversight
• TruthfulQA

U

• Uncertainty-Based Query Selection (Ensemble Variance)
• Unsupervised model learning
• Unsupervised risk estimation
• Unsupervised value iteration
• Use demonstrations (safe exploration)

V

• Variable indifference

W

• Web labeling interface
• Win rate (preference against baseline)
• Winogender
• Wireheading
• WMT 2015 French-to-English translation
• WSC (Winograd Schema Challenge)