Stanford OnlineStanford CME295 Transformers & LLMs | Autumn 2025 | Lecture 2 - Transformer-Based Models & Tricks
Summaries, chapters, transcripts, and FAQs on this page are made by uListen, not YouTube.
At a glance
WHAT IT’S REALLY ABOUT
Transformer building blocks: positional methods, normalization, attention efficiency, and BERT
- Afshine Amidi recaps transformer self-attention and multi-head attention, emphasizing how heads correspond to different learned projections and can be interpreted through attention maps.
- The lecture surveys how positional information is handled, moving from learned and sinusoidal absolute embeddings to relative approaches that modify attention logits (T5 bias, ALiBi) and to RoPE, which rotates Q and K to encode relative distance.
- It explains training-stability and efficiency changes in transformer blocks, including the shift from post-norm to pre-norm and the replacement of LayerNorm with RMSNorm to reduce parameters while keeping similar convergence behavior.
- It covers attention-efficiency tricks such as sliding-window (local) attention and head-projection sharing (MQA/GQA), motivated by the quadratic cost of dense attention and by KV-cache memory constraints during decoding.
- Shervine Amidi maps the modern “transformer landscape” into encoder–decoder models (T5), encoder-only models (BERT family), and decoder-only models (today’s LLMs), then deep-dives into BERT’s pretraining and finetuning pipeline.
IDEAS WORTH REMEMBERING
5 ideasAbsolute learned position embeddings are simple but can overfit and don’t naturally extrapolate past the training context length.
He walked through learned absolute position embeddings (one learnable vector per position, added to token embeddings) and noted two main limitations: they can overfit to position-specific patterns in the training data and they only exist up to the maximum training length, creating issues for longer sequences at inference.
Sinusoidal encodings bake relative-distance structure into dot products and generalize to longer lengths.
He derived the intuition that sinusoidal embeddings make dot-products between position vectors depend on relative offsets (via trig identities), and he highlighted the practical upside: they can be computed for any sequence length, not just lengths seen during training.
Modern positional methods often act inside the attention logits to control similarity by distance more directly.
He explained that newer approaches inject position information directly into attention scoring rather than only adding it at the input, mentioning T5’s learned relative position biases (with distance bucketization) and ALiBi’s deterministic linear bias inside the softmax.
RoPE encodes relative position via rotations of Q and K, making attention depend on relative offsets and tending to decay with distance.
He introduced RoPE as rotating query and key vectors by position-dependent angles so that the resulting query–key interaction depends on relative distance, and he emphasized this is common in many current models; he also mentioned a mathematical result showing a long-term decay in an upper bound on attention weight with distance (with oscillations).
Normalization evolved from post-LN to pre-LN and often to RMSNorm for stability with fewer learned parameters.
He contrasted the original transformer’s post-norm design with today’s common pre-norm design, then described RMSNorm as a further simplification that normalizes by root-mean-square and learns only a scaling parameter, arguing it has comparable convergence with fewer parameters.
WORDS WORTH SAVING
5 quotesSo surprisingly, this transformer architecture, which was introduced in twenty seventeen, is actually an architecture that has kind of, um, you know, still stayed relevant along the years.
— Afshine Amidi
It turns out that most models these days, they use RoPE, which is why it's important.
— Afshine Amidi
So in case you're interested, there is this keyword, internal covariate shift, which is basically the term that is given to the phenomenon that I'm describing here.
— Afshine Amidi
So we're going to see in, I think, next lecture, there, there's something called the KV cache, which basically saves the values of the keys and values.
— Afshine Amidi
But as time went, people realized that your compute budget could be best invested in the decoder only.
— Shervine Amidi
SPEAKERS
Afshine Amidi
hostCo-instructor for Stanford CME295 (Transformers & LLMs), teaching transformer-based models and related training/architecture tricks.
High quality AI-generated summary created from speaker-labeled transcript.
