Stanford CME295 Transformers & LLMs | Autumn 2025 | Lecture 2 - Transformer-Based Models & Tricks

ℹ️ EPISODE INFO

Released

October 17, 2025

Duration

1h 47m

Channel

Stanford Online

Watch on YouTube

▶ Open ↗

📝 EPISODE DESCRIPTION

For more information about Stanford’s graduate programs, visit: https://online.stanford.edu/graduate-education October 3, 2025 This lecture covers:

• Attention approximation
• MHA, MQA, GQA
• Position embeddings (regular, learned)
• RoPE and applications
• Transformer-based architectures
• BERT and its derivatives

To follow along with the course schedule and syllabus, visit: https://cme295.stanford.edu/syllabus/ Chapters: 00:00:00 Introduction 00:01:30 Recap of Transformers 00:10:37 Overview of position embeddings 00:15:36 Sinusoidal embeddings 00:25:56 T5 bias, ALiBi 00:31:02 RoPE 00:43:42 Layer normalization 00:50:39 Sparse attention 00:55:38 Sharing attention heads 01:02:42 Transformer-based models 01:11:38 BERT deep dive 01:33:24 BERT finetuning 01:43:30 Extensions of BERT Afshine Amidi is an Adjunct Lecturer at Stanford University. Shervine Amidi is an Adjunct Lecturer at Stanford University. View the course playlist: https://www.youtube.com/playlist?list=PLoROMvodv4rOCXd21gf0CF4xr35yINeOy

🎤 SPEAKERS

• Afshine Amidi

  host

  Co-instructor for Stanford CME295 (Transformers & LLMs), teaching transformer-based models and related training/architecture tricks.

🧾 EPISODE SUMMARY

In this episode of Stanford Online, featuring Afshine Amidi, Stanford CME295 Transformers & LLMs | Autumn 2025 | Lecture 2 - Transformer-Based Models & Tricks explores Transformer building blocks: positional methods, normalization, attention efficiency, and BERT Afshine Amidi recaps transformer self-attention and multi-head attention, emphasizing how heads correspond to different learned projections and can be interpreted through attention maps.

🔗 RELATED EPISODES

Stanford CS230 | Autumn 2025 | Lecture 5: Deep Reinforcement Learning

Stanford CME295 Transformers & LLMs | Autumn 2025 | Lecture 4 - LLM Training

Stanford CS230 | Autumn 2025 | Lecture 4: Adversarial Robustness and Generative Models

Stanford CME295 Transformers & LLMs | Autumn 2025 | Lecture 1 - Transformer

Stanford CME295 Transformers & LLMs | Autumn 2025 | Lecture 3 - Tranformers & Large Language Models

Stanford CS230 | Autumn 2025 | Lecture 3: Full Cycle of a DL project