Stanford CME295 Transformers & LLMs | Autumn 2025 | Lecture 5 - LLM tuning

ℹ️ EPISODE INFO

Released

November 14, 2025

Duration

1h 47m

Channel

Stanford Online

Watch on YouTube

▶ Open ↗

📝 EPISODE DESCRIPTION

For more information about Stanford’s graduate programs, visit: https://online.stanford.edu/graduate-education October 31, 2025 This lecture covers:

• Preference tuning
• RLHF overview
• Reward modeling
• RL approaches (PPO and variants)
• DPO

To follow along with the course schedule and syllabus, visit: https://cme295.stanford.edu/syllabus/ Chapters: 00:00:00 Introduction 00:04:50 Preference tuning 00:11:31 Data collection 00:17:43 RLHF overview 00:28:24 Reward model 00:29:46 Bradley-Terry formulation 00:46:02 Reinforcement learning 00:53:54 PPO 00:57:56 Advantage, value function 01:03:39 PPO variants: Clip, KL-penalty 01:16:20 Challenges 01:19:58 On-policy vs off-policy 01:22:43 Best-of-N 01:29:47 DPO Afshine Amidi is an Adjunct Lecturer at Stanford University. Shervine Amidi is an Adjunct Lecturer at Stanford University. View the course playlist: https://www.youtube.com/playlist?list=PLoROMvodv4rOCXd21gf0CF4xr35yINeOy

🎤 SPEAKERS

• Afshine Amidi

  host

  Instructor for Stanford CME295 covering deep learning, transformers, and LLM tuning methods (SFT, reward modeling, RLHF/DPO/PPO).

🧾 EPISODE SUMMARY

In this episode of Stanford Online, featuring Afshine Amidi, Stanford CME295 Transformers & LLMs | Autumn 2025 | Lecture 5 - LLM tuning explores How preference tuning aligns LLM behavior via RLHF, PPO, DPO The lecture frames LLM tuning as progressing from pre-training to SFT, then to preference tuning to align outputs with human preferences such as tone, friendliness, or safety.

🔗 RELATED EPISODES

Stanford CS230 | Autumn 2025 | Lecture 8: Agents, Prompts, and RAG

Stanford CME295 Transformers & LLMs | Autumn 2025 | Lecture 7 - Agentic LLMs

Stanford CME295 Transformers & LLMs | Autumn 2025 | Lecture 6 - LLM Reasoning

Stanford CS230 | Autumn 2025 | Lecture 6: AI Project Strategy

Stanford CS230 | Autumn 2025 | Lecture 5: Deep Reinforcement Learning

Stanford CS230 | Autumn 2025 | Lecture 4: Adversarial Robustness and Generative Models