Stanford OnlineStanford CME295 Transformers & LLMs | Autumn 2025 | Lecture 3 - Tranformers & Large Language Models
Summaries, chapters, transcripts, and FAQs on this page are made by uListen, not YouTube.
CHAPTERS
Jump to publisher timestampsChapter summaries
Course logistics and agenda: introducing Large Language Models
The lecturer welcomes the class to Lecture 3 and shares a logistics update about slide availability ahead of class. The agenda is set: a recap of Transformers so far, then moving into Large Language Models and how they generate responses.
- •Slides are posted on the course website ahead of class for annotation
- •Plan to publish slides regularly on Thursday evenings
- •Lecture focus shifts to Large Language Models after a quick recap
Transformer model families recap: encoder-decoder, encoder-only, decoder-only
The lecture reviews the three major Transformer-based model categories and what they’re used for. The recap emphasizes how decoder-only architectures became the dominant backbone for modern text generation.
- •Encoder-decoder models do text-in/text-out; T5 is given as an example
- •Encoder-only models (BERT) produce meaningful embeddings used for tasks like classification, often via the CLS token embedding
- •Decoder-only models remove the encoder and cross-attention and are used for text generation; GPT is cited as an example
- •Most modern models are described as decoder-only
What counts as an LLM: language modeling + scale + text generation
The lecturer defines a large language model as a language model that predicts next-token probabilities, and that is also “large” in parameters, data, and compute. The definition is narrowed to models that produce text, explaining why encoder-only models like BERT don’t fit this modern definition.
- •A language model assigns probabilities to token sequences via next-token prediction
- •“Large” is discussed in terms of parameter counts, training tokens, and compute needs
- •Modern LLM usage typically implies text-to-text generation; encoder-only models are excluded under this definition
- •LLMs are described as decoder-only backbones (masked self-attention + FFN + normalization)
- •Examples of model families are mentioned as things students may have heard of
Mixture of Experts (MoE) motivation and core idea: activate only some parameters
The lecture motivates MoE with a metaphor: you wouldn’t ask every expert in a room to answer a single specialized question. MoE introduces multiple experts plus a gating/router network to select which experts should process an input, aiming to reduce compute per forward pass.
- •Motivation: full dense models activate all parameters even when unnecessary
- •MoE introduces N experts and a gate/router that decides which experts to use per input
- •MoE output is described as a weighted combination of expert outputs using gate weights
- •Gate and experts are typically trained jointly via standard backprop
- •The architecture of experts is left general initially, to be specified later
Dense vs sparse MoE, compute measurement (FLOPS), and where MoE fits in LLMs
Two MoE variants are distinguished: dense MoE uses all experts with varying weights, while sparse MoE selects only top-K experts to reduce compute. The lecture connects this to FLOPS as a compute measure and argues MoE is typically placed in the feed-forward network portion of Transformer blocks because it dominates parameter and compute cost.
- •Dense MoE: all experts contribute with different weights
- •Sparse MoE: top-K experts are selected (often K=1 or K=2) for compute savings
- •FLOPS is introduced as floating point operations to quantify compute cost
- •The feed-forward network is identified as the best insertion point due to large intermediate dimension
- •In modern LLMs, experts are implemented as multiple FFNs with token-level routing
MoE routing details: token-level routing, layer-specific routers, and training challenges
The lecturer explains that routing can happen per token, and that routers are typically specific to each layer rather than shared. Training introduces a key failure mode—routing collapse—where only a few experts get used, and mitigation strategies are discussed.
- •Routing decisions are made after self-attention, before the FFN/MoE block
- •Routing is token-level: different tokens in a sequence may go to different experts
- •Routers are layer-specific; each Transformer layer has its own router parameters
- •Routing collapse is described as a situation where the router over-selects a small subset of experts
- •A load-balancing style auxiliary loss term is introduced to encourage more uniform expert usage
- •Noisy gating is mentioned as another technique to increase expert diversity; dropout is discussed as a complementary idea
Scaling and capacity trade-offs in MoE models, plus a visualization example
The lecture clarifies that increasing the number of experts increases total parameter count (“capacity”) while keeping active parameters per token controlled. The lecturer mentions a reading recommendation and shows an example visualization of per-token expert routing patterns to illustrate healthy expert utilization.
- •More experts increases total model parameters while keeping inference activation limited via top-K routing
- •Active parameters are distinguished from total parameters
- •A reading recommendation is given as an example of very large-scale MoE work
- •MoE models are described as more sample efficient in training curves in some cases
- •A visualization is shown mapping tokens to experts for a single layer to illustrate routing diversity
From next-token probabilities to sequences: greedy decoding vs beam search
The lecture transitions to response generation and asks how to choose a next token from the model’s probability distribution. Greedy decoding (argmax) is introduced and critiqued for determinism and local optimality, then beam search is presented as tracking multiple high-probability continuations.
- •Response generation is framed as iterative next-token prediction
- •Greedy decoding selects the maximum-probability token each step; it can be deterministic and less diverse
- •Greedy decoding is described as locally optimal but not necessarily best for the full sequence probability
- •Beam search keeps the top-K candidate sequences (beam width) over multiple steps
- •Sequence scoring is described via sums of log probabilities; length bias toward shorter sequences is noted and countered with length normalization in practice
Sampling-based generation: top-K and top-P (nucleus) sampling
Because beam search still tends to favor high-likelihood, low-diversity outputs and can be compute-heavy, the lecture introduces sampling. Practical sampling approaches restrict sampling to higher-probability tokens using top-K or top-P criteria.
- •Sampling draws the next token according to the probability distribution
- •Sampling is positioned as an inference-time choice (not training)
- •Top-K sampling restricts choices to the K highest-probability tokens
- •Top-P sampling restricts choices to tokens whose cumulative probability exceeds a threshold P
- •Motivation: avoid low-probability tokens that could still be sampled but are often undesirable
Temperature in softmax: controlling distribution sharpness and output diversity
The lecture explains how logits become vocabulary probabilities via a linear layer and softmax, and introduces temperature as a key hyperparameter. It derives the intuition that low temperature makes the distribution spikier (more deterministic), while high temperature makes it more uniform (more diverse).
- •Model outputs logits that are transformed into probabilities with softmax
- •Temperature appears in softmax as scaling of logits before normalization
- •Low temperature produces a spiky distribution emphasizing the highest-logit token(s)
- •High temperature pushes probabilities toward a more uniform distribution, increasing diversity
- •Determinism is attributed to sampling; the Transformer computations themselves are described as deterministic
- •Practical note: even with temperature set to zero, non-determinism can arise from hardware-level computation order effects
Guided decoding for structured outputs: filtering invalid next tokens
The lecture introduces guided decoding as a method to enforce output constraints during generation rather than regenerating repeatedly until a valid format appears. The idea is to eliminate invalid next tokens as the model generates, using formalisms like finite state machines or context-free grammars.
- •Naive approach: ask for a format (e.g., JSON) and retry until valid
- •Guided decoding: restrict generation by filtering invalid next tokens at each step
- •Some steps may allow multiple valid tokens; then a normal selection/sampling method applies
- •Pointers are given to finite state machines and context-free grammar approaches
- •This section closes the first speaker’s portion and transitions to the next part of the lecture
Prompt structure, context length vocabulary, and the limits of long context (context rot)
The second speaker begins by defining terminology around context length and how it’s measured in tokens. They discuss that longer contexts don’t automatically solve retrieval and reference a phenomenon called context rot, where retrieving buried information can degrade as context grows, especially with distractors.
- •Context length is also called context size, window size, and similar terms
- •Modern models are described as supporting very large token contexts
- •Longer context doesn’t guarantee better retrieval; performance can degrade with longer inputs
- •A “needle in a haystack” style test is described for evaluating retrieval from long contexts
- •Distractors in context are described as contributing to worse retrieval performance
- •A question confirms context length corresponds to what self-attention operates over
Prompting strategies: context/instructions/input/constraints and in-context learning (zero-shot vs few-shot)
The lecture proposes a useful prompt “mental model” consisting of context, instructions, input, and constraints, including hidden constraints such as safety instructions. It then introduces in-context learning as steering model behavior via prompt content, contrasting zero-shot prompting with few-shot prompting using examples.
- •Prompts are broken into context, instructions, input, and constraints
- •Examples are given of user-visible and hidden prompt components (like safety constraints)
- •In-context learning is described as improving task performance without changing model weights
- •Zero-shot: only provide the query/task request
- •Few-shot: provide example input-output pairs before the target query
- •Trade-offs of few-shot include needing examples and consuming more context tokens/compute
Chain-of-thought and self-consistency: improving reasoning and robustness through multiple samples
The lecture introduces chain-of-thought prompting as eliciting rationales before answers to improve performance and interpretability for debugging. It then describes self-consistency as sampling multiple reasoning paths and using majority voting over extracted final answers, with notes on extracting answers and running samples in parallel.
- •Chain-of-thought is described as having the model produce rationale before the answer to improve performance
- •Chain-of-thought can make outputs more interpretable for debugging prompt/context issues
- •Self-consistency samples multiple responses and uses majority voting on the final answer
- •Answer extraction methods are discussed, including instructing the model to put the answer last or using regex/another model to extract
- •Samples are described as generated in parallel and not fed into each other
Inference-time efficiency: KV cache, grouped-query attention, and memory management (PagedAttention)
The lecture shifts to inference optimizations, starting with exact methods that avoid redundant computation. It explains KV caching as reusing previously computed keys and values during autoregressive decoding, notes why this doesn’t apply in the same way during teacher-forced training, and connects to reducing cache size via grouped-query attention. It then covers memory fragmentation issues in serving multiple requests and explains a block-based cache management strategy described as PagedAttention.
- •KV cache stores past key/value projections to reuse them instead of recomputing during decoding
- •During teacher-forced training, KV caching is less relevant because tokens are processed together
- •Grouped-query attention is referenced as a way to reduce the number of key/value sets relative to full multi-head attention
- •Serving multiple concurrent requests can waste memory if allocating maximum context length upfront
- •PagedAttention is described as allocating KV cache in fixed-size blocks to reduce internal/external fragmentation
- •A mapping/dictionary from token positions to cache blocks is described as part of the approach
Compressing KV cache with multi-latent attention and faster token generation with speculative methods
The final segment covers methods to reduce KV cache footprint by compressing key/value representations and sharing components, described via a multi-latent attention approach. It then introduces output-level acceleration: speculative decoding uses a smaller draft model to propose multiple tokens and a larger target model to accept/reject them while matching the target distribution, followed by multi-token prediction where multiple heads within one model serve as an internal draft mechanism.
- •Multi-latent attention is described as factorizing key/value projections into a lower-dimensional latent space and then decompressing
- •A shared compression across keys and values is described to reduce what must be stored in cache
- •Sharing across attention heads is described as further reducing cache storage per transformer block
- •Speculative decoding: a smaller draft model proposes tokens; a target model validates with an accept/reject mechanism
- •Rationale is discussed in terms of matching the target model’s distribution and using a single pass to obtain multiple token distributions
- •Multi-token prediction: training objective predicts multiple future tokens via multiple heads; at inference, draft tokens are accepted/rejected in a greedy manner
Publisher timestamps
- 0:00Introduction
- 1:02Recap of Transformers-based models
- 3:43LLM definition
- 7:37Mixture of ExpertsRead this chapter
- 12:43Dense & Sparse MoE
- 15:47MoE in LLMs
- 36:35Response generationRead this chapter from 36:36
- 38:34Greedy decoding & beam search
- 46:36Sampling-based methodsRead this chapter
- 52:24Impact of temperature on predictions
- 1:04:53Guided decodingRead this chapter
- 1:07:07Prompting strategies
- 1:14:09In-context learning
- 1:18:35Chain-of-thought, self-consistencyRead this chapter
- 1:25:00Inference optimizations with KV cache
- 1:33:09PagedAttention, MLA
