Supervised fine-tuning (SFT)
Training language models to follow instructions with human feedback — inherited; Steering Llama 2 via Contrastive Activation Addition — refined; Persona Vectors: Monitoring and Controlling Character Traits in Language Models — refined
The first training step, in which a pretrained language model is fine-tuned directly on human-written demonstrations of desired output behavior using standard supervised learning.
Before any reinforcement learning happens, one early paper begins the old-fashioned way: show the model examples of the behavior you want, written by people, and train it to imitate them. The technique looks simple enough that it is easy to underestimate how much a model can change from it.
From the technique in general, the page narrows to its role as step one of an RLHF pipeline, then traces it back to an earlier proposal, before picking the thread up again in a steering paper that uses it as a baseline, finetuning a model on the very data used to build a steering vector. The corpus's newest paper turns the same ordinary technique into its central object of study, showing that finetuning on data with no explicit trait content at all can still measurably and predictably shift a model's persona.
The technique, in general
Supervised fine-tuning adapts a pretrained model to a new behavior by continuing to train it, with ordinary supervised learning, on a smaller dataset of input-output examples that demonstrate the desired behavior directly. It uses the same cross-entropy objective as pretraining, $L(\theta) = -\sum_t \log P_\theta(x_t \mid x_{<t})$, but computed over labeled demonstrations rather than raw internet text. Fine-tuning a pretrained model on a narrower supervised objective is a long-standing transfer-learning idea in machine learning, predating this paper by years in NLP practice; InstructGPT inherits the technique and applies it as the first of its three training steps.
As InstructGPT's first step
Labelers write demonstrations of desired output behavior on the paper's input prompt distribution, and a pretrained GPT-3 (pretrained language model) model is fine-tuned on this data using supervised learning to produce the Supervised fine-tuned (SFT) model (instructgpt, §"3.1 High-level methodology", p. 6). This model serves two purposes downstream: it is evaluated as a standalone baseline, and it initializes both the Reward model (RM) and, via a separately tuned checkpoint, the PPO fine-tuning stage (RLHF policy optimization) policy, making it the structural foundation the rest of the InstructGPT's three-step training pipeline theme is built on.
A direct descendant of an earlier proposal
The Use demonstrations (safe exploration) — operationalized as → Supervised fine-tuning (SFT) edge, marked with hindsight, traces this step back to Concrete Problems' proposal to "use demonstrations" as a route to avoiding both reward hacking and unsafe exploration: learn a baseline policy from expert trajectories, via inverse reinforcement learning or apprenticeship learning, then explore only around that baseline (concrete-problems, §"Use Demonstrations:", p. 15). InstructGPT's SFT step realizes this idea by the simplest possible mechanism the 2016 paper considered, fine-tuning directly on demonstrations with ordinary supervised learning rather than first recovering a reward function through IRL or apprenticeship learning, and drops the exploration-safety framing entirely: SFT here is not bounding a later RL exploration phase but standing as a free-standing baseline and initializer. What survives across six years is the core wager that a policy learned from demonstrations can substitute for a hand-specified objective, a wager this concept shares with Supervised fine-tuned (SFT) model and SFT dataset in the Learning from demonstrations instead of a specified reward theme.
2023: a head-to-head against activation steering, and where finetuning's control runs out
Steering Llama 2 via Contrastive Activation Addition (2023) runs supervised fine-tuning as a direct comparison arm rather than a passing citation: Llama 2 7B Chat is fine-tuned for one epoch with SGD at a learning rate of 1e-4 on the identical multiple-choice contrastive dataset used to build CAA's steering vectors, with a supervised objective that maximizes the likelihood of the model picking the positive or negative answer token (contrastive-activation-addition, §"6 Comparison to finetuning", p. 5). Left unoptimized — the paper does not tune learning rate, epoch count, or loss function — this recipe still reaches over 90% test-set accuracy on most of the seven behavior datasets (contrastive-activation-addition, §"10 Limitations", p. 10). Yet composing the two levers shows finetuning's control is not exhaustive: for three of the seven behaviors, CAA can push the targeted behavior further than finetuning alone achieves, in both directions (contrastive-activation-addition, §"6 Comparison to finetuning", p. 6). A model finetuned to near-ceiling accuracy on the finetuning task itself still moves when a steering vector is added on top of it.
The clearest gap between the two levers is Sycophancy. Table 13 records only 44% test accuracy after finetuning on the positive-sycophancy split, against 90% for the negative split, and, more strikingly, the paper reports that "finetuning results in out-of-distribution generalization failure for the Sycophancy dataset, where training on multiple-choice questions fails to generalize to the open-ended setting, whereas CAA generalizes in all cases" (contrastive-activation-addition, §"6 Comparison to finetuning", p. 6). Finetuning on A/B answer tokens is learning something narrower than the behavior itself in this case; a steering vector extracted the same way generalizes to free-text generation regardless.
The two levers also cost differently, a gap CAA vs. finetuning computational cost comparison traces directly to what each one computes. Generating a CAA vector needs only forward passes, run on one NVIDIA L40 GPU in under five minutes; finetuning the same model on the same 1,000 examples needs gradients and optimizer state for every parameter, run across two L40 GPUs with data parallelism and the ZeRO optimizer for memory partitioning, taking about ten minutes (contrastive-activation-addition, §"J Computational resources", p. 15). The paper's own Limitations section proposes one way to close part of this gap from finetuning's side: replacing the current objective, which only incentivizes selecting the positive answer, with a contrastive loss that also penalizes selecting the negative answer — a modification it flags but leaves untested (contrastive-activation-addition, §"10 Limitations", p. 10).
2025: SFT as the mechanism under the microscope, and preventative steering folded into its own recipe
Where CAA and InstructGPT treat supervised fine-tuning as a lever they apply and move past, Persona Vectors puts SFT itself under the microscope, asking what an ordinary finetuning run does to a model's persona as a side effect of learning its intended task. Every dataset in the study — the three trait-eliciting datasets, the EM-like datasets, and their Normal controls — is finetuned with the identical recipe: one epoch of rank-32 rsLoRA (Kalajdzievski, 2023) with scaling factor α = 64 and learning rate 1e-5, on a single NVIDIA H100 GPU (persona-vectors, §"D.3 Finetuning details", p. 38). Holding rank, epoch count, and learning rate fixed everywhere is the point: because the only thing that varies between a high-drift and a low-drift run is the training data, the resulting activation shift along the persona vector can be attributed to what the data teaches rather than to how aggressively the model was trained (persona-vectors, §"4.2 Activation shift along persona vector predicts trait expression", p. 7). This is a modest recipe by design — a single low-rank adapter, not extended or heavily tuned training — which is itself the finding: almost any ordinary, resource-light SFT run measurably moves the persona, provided the training data carries a directional signal at all.
The paper also folds an intervention directly into this same training step. Preventative steering adds the undesired persona vector into the model's activations during the SFT forward passes themselves, so gradient descent no longer needs to move the weights in that direction to fit the data — a training-time countermeasure to what SFT was just shown to cause, applied within the identical rsLoRA setup rather than as a separate procedure (persona-vectors, §"5.2 Preventative steering limits behavioral shifts during finetuning", p. 8). Applied to a benign, non-trait-inducing dataset, the same intervention leaves both trait expression and MMLU accuracy essentially unchanged, evidence that the countermeasure targets the drift specifically rather than SFT in general (persona-vectors, §"J.6 Assessing side effects of preventative steering on benign data", p. 48).