Stanford OnlineStanford CME295 Transformers & LLMs | Autumn 2025 | Lecture 5 - LLM tuning
Summaries, chapters, transcripts, and FAQs on this page are made by uListen, not YouTube.
CHAPTERS
Jump to publisher timestampsChapter summaries
Course logistics, midterm notes, and where tuning fits in the LLM pipeline
Afshine Amidi opens with notes about the midterm and what the final will cover. He then situates “LLM tuning” within the overall training pipeline, distinguishing pre-training and supervised fine-tuning (SFT) from the additional alignment step the lecture focuses on.
- •Midterm exam and solutions are posted; final follows the same format but covers later lectures
- •Recap of pre-training: expensive, large-scale next-token training and parallelization ideas
- •Recap of SFT: smaller, higher-quality task data to teach desired behaviors
- •Motivation for a third step: aligning model behavior with preferences beyond task completion
Why preference tuning: preference pairs, easier labeling, and adding negative signal
The lecture introduces preference tuning as aligning an already SFT-trained model to human preferences, illustrated with a “bad vs good” assistant response example. Afshine explains why this step can be more practical than trying to encode everything into SFT data.
- •Preference tuning targets aspects like tone or safety that SFT may not fully capture
- •Preference pairs compare a preferred response vs a dispreferred response for the same prompt
- •Labeling preferences can be easier than writing ideal responses from scratch
- •Preference tuning can inject negative signal (what the model should not do)
- •Preference tuning isn’t a fix for everything; SFT data quality issues can still be the root cause
Collecting preference data: pointwise vs pairwise vs listwise, and how pairs are generated
Afshine lays out common data formats for preference labeling and why pairwise comparisons are often used. He then describes practical recipes for constructing response pairs and obtaining comparisons via humans or other evaluators.
- •Pointwise scoring is hard for humans to calibrate; listwise ranking can be more complex
- •Pairwise preference data is commonly used because binary comparisons are simpler
- •Generate two different responses by sampling with positive temperature
- •Prompts ideally come from user-like distributions (e.g., logs or curated prompt sets)
- •Comparisons can come from humans, an LLM-as-judge approach, or rule-based metrics; often reduced to a binary preference
RLHF overview: mapping RL concepts onto LLMs and understanding sparse rewards
The lecture introduces RLHF and briefly reviews reinforcement learning basics (agent, state, action, policy, reward), then maps them to the LLM setting. Afshine addresses questions about reward sparsity and whether rewards are applied per token or per completion.
- •In RL framing: LLM is the agent, the state is the current text context, the action is the next token
- •The LLM’s next-token distribution corresponds to the policy
- •In this setup, reward is tied to the overall completion rather than each token
- •RLHF signals can be sparse compared with SFT because you get roughly one signal per completion
- •Costs exist, but the workflow is presented as comparable in scale to other training procedures, with extra components added
Training the reward model: Bradley–Terry formulation and pairwise loss construction
Afshine explains the first stage of RLHF: fitting a reward model that scores prompt–response pairs. He derives the standard pairwise objective from the Bradley–Terry model and highlights that the reward model is trained pairwise but produces pointwise scores at inference.
- •Reward model input: concatenated prompt and response; output: a scalar score
- •Bradley–Terry gives probability that one response is preferred over another via a sigmoid of score differences
- •Loss derived by maximizing likelihood of observed preferences, leading to a negative log-sigmoid objective
- •Even though training uses pairs, the learned reward model can score single prompt–response inputs
- •Implementation options discussed: decoder-only LLM with a classification head, or encoder-only approaches; reward dimensions depend on labeling guidelines
RL step setup: using the frozen reward model, avoiding drift, and reward hacking intuition
The lecture moves to the second RLHF stage: using the trained (frozen) reward model to update the LLM policy. Afshine motivates why optimization must be constrained to avoid drifting too far from the base SFT model, including concerns about reward hacking and instability.
- •RL step: generate a completion, score it with the frozen reward model, then update the LLM policy
- •Goal includes maximizing reward while staying close to the initial SFT model distribution
- •Reasons to limit deviation: preserve prior knowledge, avoid overfitting/noisy reward model behavior, reduce training instability
- •Reward hacking example: optimizing an imperfect proxy can maximize the metric without achieving the real objective
- •Training data volume here is described as larger than reward modeling; reward comes from the reward model rather than humans directly
PPO foundations: KL divergence constraint and why training uses advantage and a value function
Afshine introduces PPO as a popular approach for aligning a policy while keeping it proximal to a reference, including a KL divergence regularization term. He then explains why PPO uses advantage rather than raw rewards, and what the value function estimates in the LLM context.
- •PPO is framed as balancing reward maximization with a constraint/penalty for deviating from a reference policy
- •KL divergence is presented as a measure of how far two probability distributions are, nonnegative and zero only when equal
- •Rather than maximizing reward directly, PPO maximizes advantage (reward relative to an expected baseline) to reduce variance
- •Value function is token-level: given a partial generation, it estimates expected final reward if generation continues under the policy
- •Value head is trained jointly with the policy to support advantage estimation; generalized advantage estimation is referenced for details
PPO Clip: probability ratio, clipping intuition, and stability across iterations
Afshine walks through the PPO Clip objective and clarifies confusing notation (especially that the ‘R’ term is a probability ratio, not ‘reward’). He explains how clipping limits the update size when advantage is positive vs negative, and why ‘old’ refers to the previous RL iteration.
- •PPO Clip uses a ratio between current and previous-iteration policy probabilities to control step size
- •Clipping prevents the objective from encouraging overly large probability changes
- •With positive advantage, the objective encourages increasing probability but caps the increase; with negative advantage, it encourages decreasing probability but caps the decrease
- •‘Old’ policy in this context is the previous RL iteration, not necessarily the base SFT model
- •Discussion includes why controlling step-to-step update magnitude improves training stability
PPO variants and practicalities: KL-penalty version, mixing formulations, and model-count overhead
The lecture presents the PPO KL-penalty variant and notes how, in modern LLM practice, the reference model often plays a key role in the KL term. Afshine then emphasizes the practical burden of PPO-based RLHF, including maintaining several model components during training.
- •KL-penalty variant adds an explicit KL divergence penalty term to the advantage-weighted objective
- •Modern implementations often use a reference model in KL penalties and may combine KL penalties with clipping-style constraints
- •PPO-based RLHF typically involves multiple models/heads: policy, value function head, reward model, and a frozen reference/base model
- •Afshine notes other algorithms exist and mentions GRPO will be covered in the next lecture
- •Key question raised: whether all these components are strictly necessary for strong results
Challenges of RL-based alignment: two-stage dependency, hyperparameters, monitoring, exploration, and on-policy training
Afshine summarizes why RLHF can be operationally difficult beyond compute cost. He highlights the dependency between reward modeling and policy training, tuning complexity, instability, and the need for exploration, and explains on-policy vs off-policy training terminology.
- •Two-stage pipeline creates tight dependencies: issues in reward modeling can force retraining downstream steps
- •Many hyperparameters appear (e.g., from KL penalties, clipping, and advantage estimation), complicating tuning
- •Monitoring training via average reward is less straightforward than monitoring cross-entropy in pre-training/SFT
- •Exploration is needed so the policy samples diverse completions during RL updates
- •On-policy training means updating on samples generated by the current policy; contrasted with off-policy data usage
Best-of-N (BON): using a reward model at inference instead of RL training
Afshine presents Best-of-N as a way to exploit a trained reward model without doing RL updates. The approach samples multiple completions from an SFT model, scores them, and returns the highest-scoring completion, along with a discussion of the main drawback: inference cost and latency.
- •Generate N candidate completions from the (unchanged) SFT model for a given prompt
- •Score each prompt–completion with the reward model and return the highest-rated one
- •Example scores illustrate the reward model output as a continuous scalar and why only ranking matters for selection
- •Main downside: cost/latency shifts to inference due to multiple generations per request
- •Method depends on the base model being at least somewhat capable; if all candidates are bad, selection won’t fix it
Direct Preference Optimization (DPO): supervised alternative motivated by PPO/BON drawbacks
Shervine takes over to introduce DPO as a direct, supervised objective over preference pairs that avoids explicit reward modeling and RL loops. He connects the DPO loss back to the Bradley–Terry structure and frames it as optimizing model weights with fewer moving parts than PPO-based RLHF.
- •Motivation: PPO requires multiple model components; BON increases inference latency even with ample compute
- •DPO optimizes a single loss directly on preference pairs without an explicit reward model term
- •The loss compares likelihoods of winning vs losing completions under the current policy relative to a reference
- •Connection highlighted to Bradley–Terry-style log-sigmoid of score differences, but expressed via policy probabilities
- •Paper intuition described: ‘Your language model is secretly a reward model’ because an equivalent reward can be expressed from policy terms
Where the DPO objective comes from: PPO-style optimal policy derivation and the role of beta
Shervine outlines the derivation path: start from a reward-plus-KL objective, solve for the optimal policy, rearrange to express a reward-like quantity, then plug into Bradley–Terry and turn it into a loss. He notes beta as the trade-off coefficient inherited from the PPO-style objective and gives an order-of-magnitude value mentioned in the lecture.
- •Start from objective trading off reward maximization and KL divergence to the reference, scaled by beta
- •Derive an optimal policy expression and rearrange to express reward as a function of policy and reference terms
- •Use Bradley–Terry preference probability with that reward expression to eliminate explicit rewards
- •Convert maximizing preference likelihood into a supervised loss over preference pairs
- •Beta is presented as the same trade-off coefficient as in the earlier formulation, with a typical magnitude mentioned as around 0.1
DPO vs RLHF in practice: model count, distribution shift concerns, and a tone-alignment example
Shervine contrasts the operational simplicity of DPO with PPO’s complexity, while noting trade-offs in performance and potential distribution shift issues discussed in referenced work. He closes with an example showing preference tuning as tone refinement rather than learning new facts.
- •RLHF/PPO is described as heavier and potentially more complex/unstable; DPO uses supervised optimization on preference pairs
- •DPO requires fewer model components in memory (current policy plus frozen reference) than PPO pipelines
- •Shervine states PPO performs better overall, while DPO can be easier to apply for strong (though not necessarily best) results
- •Distribution shift is presented as a challenge for DPO; one mitigation mentioned is training SFT on preference-related data
- •Example revisits a teddy bear washing answer: preference tuning improves friendliness/gentleness while keeping the factual content consistent
Publisher timestamps
- 0:00Introduction
- 4:50Preference tuning
- 11:31Data collectionRead this chapter
- 17:43RLHF overviewRead this chapter
- 28:24Reward modelRead this chapter
- 29:46Bradley-Terry formulationRead this chapter
- 46:02Reinforcement learning
- 53:54PPO
- 57:56Advantage, value function
- 1:03:39PPO variants: Clip, KL-penaltyRead this chapter
- 1:16:20Challenges
- 1:19:58On-policy vs off-policyRead this chapter
- 1:22:43Best-of-NRead this chapter
- 1:29:47DPORead this chapter
