This video isn’t embeddableWatch on YouTube →

Stanford Online

Stanford CME295 Transformers & LLMs | Autumn 2025 | Lecture 2 - Transformer-Based Models & Tricks

October 3, 2025

Afshine Amidihost

Oct 17, 20251h 47mWatch on YouTube

Summaries, chapters, transcripts, and FAQs on this page are made by uListen, not YouTube.

CHAPTERS

Jump to publisher timestamps

Chapter summaries

Course logistics, then why self-attention matters
Afshine Amidi opens with two logistics updates about the lecture recording audio setup and a tentative final exam date change. He then frames Lecture 2 as building on self-attention and the Transformer architecture introduced previously.
- •Audio quality issue from the prior lecture leads to a new recording setup and a quick audibility check
- •Final exam has a placeholder date and may move earlier in the week
- •Lecture 2 will revisit core Transformer ideas and then cover what has changed in modern variants
- •Plan to split the lecture: Afshine covers architectural “tricks,” then Shervine covers model families and BERT
Transformer recap: attention formula, encoder–decoder, and what heads represent
Afshine recaps self-attention using queries, keys, and values and highlights that it is implemented as efficient matrix multiplications. He revisits the encoder–decoder structure and clarifies how multi-head attention uses different learned projections per head.
- •Self-attention computed as softmax(QKᵀ/√dₖ)V and is highly optimized on hardware
- •Original Transformer consists of an encoder stack and a decoder stack, introduced for machine translation
- •Multi-head attention corresponds to multiple sets of learned projections for Q, K, and V
- •Heads can be interpreted via attention maps showing which tokens receive high weight
Why positional information is needed, and learned absolute position embeddings
Afshine explains that Transformers lack inherent sequential order, unlike RNNs, so positional information must be injected. He first describes learned absolute position embeddings added to token embeddings and discusses key limitations.
- •Self-attention connects all tokens directly, losing explicit ordering information
- •Learned position embeddings assign a dedicated embedding to each position and add it to token embeddings
- •Learned embeddings can reflect training-set biases tied to particular positions
- •They only cover positions seen during training, creating issues for longer sequences at inference
Sinusoidal position embeddings: construction and the dot-product intuition
Afshine derives the intuition behind fixed sinusoidal positional embeddings, motivating them through how dot products relate to relative distance. He connects trigonometric identities to the idea that nearby positions yield higher similarity under the embedding dot product.
- •Fixed position embeddings use sine and cosine across dimensions, matching token embedding dimension
- •Dot products between two position vectors yield sums of cosine terms depending on relative distance (M−N)
- •This aligns with the embedding-similarity intuition: closer positions can be more similar
- •Visualization shows different frequencies across dimensions driven by a dimension-dependent scaling term
From adding positions at input to modifying attention: T5 bias and ALiBi
Afshine argues that position effects ideally appear directly in the attention similarity computation rather than only via input additions. He introduces relative-position bias approaches that add terms inside the attention softmax, contrasting learned vs deterministic schemes.
- •Motivation: positional effects should influence the attention similarity scores inside the softmax
- •T5-style approach learns relative position bias terms, often by bucketizing relative distances
- •ALiBi uses a deterministic linear bias as a function of relative position difference
- •Bias terms inside softmax are valid because softmax normalizes afterward
RoPE (Rotary Position Embeddings): rotating Q and K to encode relative distance
Afshine introduces RoPE as a widely used modern approach where queries and keys are rotated by position-dependent angles. He sketches how rotation matrices work and emphasizes that the resulting attention depends on relative position (N−M).
- •RoPE rotates the query by a position-based angle and the key by a position-based angle
- •Rotation is implemented via matrix multiplication; a 2D rotation matrix is shown and briefly justified
- •When applied to QKᵀ, the effect becomes a function of relative distance between positions
- •Afshine notes an observed long-term decay in an upper bound on attention weight with distance (with oscillations)
Normalization in Transformers: post-norm vs pre-norm, and RMSNorm
Afshine revisits the Transformer’s “add & norm” blocks and explains layer normalization’s role in training stability. He then explains two modern shifts: moving normalization to pre-norm and replacing LayerNorm with RMSNorm to reduce parameters.
- •LayerNorm normalizes activations using mean and standard deviation, then applies learned scale and shift (gamma, beta)
- •Original Transformer used post-norm (normalize after residual addition)
- •Modern practice often uses pre-norm (normalize before the sublayer)
- •RMSNorm normalizes using root mean square and typically learns only gamma, reducing parameters while keeping similar convergence
Making attention cheaper: sliding-window (local) attention and receptive-field intuition
Afshine discusses the O(N²) cost of full self-attention and motivates sparse approximations. He presents sliding-window attention and describes how stacking local-attention layers expands the effective receptive field, analogous to convolutional receptive fields.
- •Full attention scales quadratically with sequence length due to an N×N interaction matrix
- •Longformer is mentioned as an example exploring restricted attention windows
- •Sliding-window attention restricts each token’s attention to nearby tokens; modern models may mix local and global layers
- •Stacking local attention layers increases the effective context, similar to receptive fields in vision models
Sharing attention projections across heads: MQA and GQA for KV-cache efficiency
Afshine explains a second attention variation: sharing key/value projections across heads to reduce memory and improve decoding efficiency. He defines multi-query attention (MQA) and grouped-query attention (GQA) and ties the motivation to repeated use of keys/values during decoding and KV caching.
- •Standard multi-head attention has separate Q/K/V projections per head
- •MQA shares a single K/V projection across all heads; GQA shares K/V across groups of heads
- •Queries are typically kept per-head while K/V are shared to reduce repeated storage/compute during decoding
- •Motivation connects to KV cache size and memory use in autoregressive decoding
Transformer model landscape: encoder–decoder (T5), encoder-only, and decoder-only trends
Shervine transitions to categorizing Transformer-based model families and how they relate to the original architecture. He covers encoder–decoder models like T5, encoder-only models for classification use cases, and the rise of decoder-only models for scalable next-token prediction.
- •T5 family retains encoder–decoder structure and reframes tasks as text-to-text
- •T5 pretraining uses span corruption with sentinel tokens and decoder reconstruction of missing spans
- •Encoder-only models drop generation and focus on representations for classification-style tasks
- •Decoder-only models drop the encoder and cross-attention, focusing on masked (causal) self-attention and next-token prediction as an easily scalable objective
BERT architecture essentials: bidirectionality, CLS/SEP, and segment embeddings
Shervine introduces BERT as an encoder-only, bidirectional Transformer and explains what “bidirectional” means operationally in self-attention. He describes the special tokens (CLS and SEP), the multi-stage pretrain-then-finetune setup, and the added segment embeddings used for sentence-pair inputs.
- •BERT uses encoder-only self-attention without a causal mask, allowing each token to attend to all tokens
- •CLS token is a placeholder whose final embedding is used for sequence-level classification
- •SEP token separates sentences for sentence-pair setups
- •Input embeddings combine token embeddings, positional embeddings, and learned segment A/B embeddings shared across all tokens in each segment
BERT pretraining and downstream use: MLM & NSP, fine-tuning patterns, and key extensions
Shervine details BERT’s pretraining objectives (masked language modeling and next sentence prediction) and how fine-tuning attaches task heads. He then highlights practical limitations and describes two extensions: DistilBERT via distillation and RoBERTa via removing NSP and changing masking/data strategy.
- •MLM: randomly selected tokens are mostly replaced by [MASK], sometimes kept, sometimes replaced by a random token; the model predicts the original tokens from context
- •NSP: a classifier predicts whether two sentences are consecutive vs randomly paired, using the CLS embedding
- •Fine-tuning: add a task head on CLS for sequence classification or on per-token embeddings for token-level tasks (e.g., span start/end)
- •Limitations discussed include context length and latency/size; DistilBERT reduces layers and uses distillation with KL divergence; RoBERTa drops NSP, uses dynamic masking, and changes data strategy

Publisher timestamps

Get more out of YouTube videos.

Read along on your phone, search it beside the video, or ask your favorite AI app what was said.