Stanford OnlineStanford CME295 Transformers & LLMs | Autumn 2025 | Lecture 5 - LLM tuning
Summaries, chapters, transcripts, and FAQs on this page are made by uListen, not YouTube.
At a glance
WHAT IT’S REALLY ABOUT
How preference tuning aligns LLM behavior via RLHF, PPO, DPO
- The lecture frames LLM tuning as progressing from pre-training to SFT, then to preference tuning to align outputs with human preferences such as tone, friendliness, or safety.
- Preference data is typically collected as pairwise comparisons by generating multiple responses per prompt (e.g., via temperature) and having humans or other metrics choose the better response.
- RLHF is explained as training a reward model with a Bradley–Terry (sigmoid) pairwise objective, then using that frozen reward model to guide reinforcement learning updates of the LLM.
- PPO is introduced as the standard RLHF optimizer, using advantage estimates and constraints (clipping and/or KL penalties) to prevent overly large or unstable policy updates and to limit drift from a reference model.
- Alternatives are compared: Best-of-N uses the reward model at inference to pick the best sample, while DPO replaces the RL pipeline with a single supervised objective directly on preference pairs, reducing system complexity but introducing its own trade-offs.
IDEAS WORTH REMEMBERING
5 ideasPreference tuning targets “what to prefer,” not “what to say.”
The lecturer described preference tuning as a third stage after pre-training and supervised fine-tuning (SFT), meant to shift a model’s behavior toward outputs humans prefer (e.g., tone, safety) rather than just teaching task completion.
Pairwise (often binary) preference data is favored because it’s simplest to label reliably.
The episode contrasted pointwise scoring, pairwise comparisons, and listwise ranking for collecting preference labels, emphasizing that pairwise binary judgments are commonly used because they are easier and less subjective than assigning absolute scores.
The reward model is trained pairwise but used pointwise to score a single completion.
A reward model is trained on winning/losing response pairs using the Bradley–Terry formulation, producing a single scalar score for any (prompt, response) input even though it is trained with pairwise comparisons.
RLHF optimizes reward, but must constrain divergence from the base model to stay grounded.
RLHF was presented as a two-stage pipeline: train a reward model from labeled preferences, then run reinforcement learning to update the LLM to increase reward while constraining drift from a reference model to avoid forgetting, reward hacking, and instability.
PPO-based RLHF is powerful but operationally heavy and hyperparameter-sensitive.
PPO was introduced as a common RL algorithm for RLHF, requiring multiple components (policy model, reward model, value function, and a reference/base model), and using advantage estimates plus mechanisms like clipping and/or KL penalties to stabilize updates.
WORDS WORTH SAVING
5 quotesSo it's a great autocompleter, but it's not a helpful model yet.
— Afshine Amidi
Preference tuning allows us to inject some negative signal.
— Afshine Amidi
So you're just training it pairwise, but it's actually pointwise.
— Afshine Amidi
So DPO stands for Direct Preference Optimization.
— Shervine Amidi
We don't want to learn new facts. We want to focus on the existing distribution of completions and align what the model should return with respect to what the human prefers.
— Shervine Amidi
SPEAKERS
Afshine Amidi
hostInstructor for Stanford CME295 covering deep learning, transformers, and LLM tuning methods (SFT, reward modeling, RLHF/DPO/PPO).
High quality AI-generated summary created from speaker-labeled transcript.
