Stanford OnlineStanford CME295 Transformers & LLMs | Autumn 2025 | Lecture 3 - Tranformers & Large Language Models
Summaries, chapters, transcripts, and FAQs on this page are made by uListen, not YouTube.
At a glance
WHAT IT’S REALLY ABOUT
Lecture surveys LLM basics, decoding, prompting, and inference optimizations
- The lecture defines large language models as large, decoder-only next-token predictors and contrasts them with encoder-only and encoder–decoder transformer variants discussed in earlier lectures.
- It introduces Mixture of Experts as an architectural approach that routes each token through only a small number of feed-forward expert networks to reduce active compute while increasing total parameter capacity, and it discusses routing collapse and mitigation strategies.
- It explains response generation methods—greedy decoding, beam search, and sampling—highlighting why sampling with top-K or top-P restrictions is commonly used to balance quality and diversity.
- It derives and interprets temperature in the softmax as a control on probability sharpness, connecting low temperature to more deterministic outputs and high temperature to more varied outputs, while noting real-world non-determinism from compute behavior.
- It surveys prompting and inference techniques, including prompt components, zero-shot vs. few-shot in-context learning, chain-of-thought and self-consistency, and multiple inference optimizations such as KV caching, PagedAttention, compression of KV representations, and speculative-style decoding methods.
IDEAS WORTH REMEMBERING
5 ideasModern LLM practice largely converges on decoder-only transformers for text generation.
The lecture distinguishes encoder–decoder (e.g., T5), encoder-only (e.g., BERT), and decoder-only (e.g., GPT) transformer families, emphasizing that most modern text-generating systems are decoder-only.
“LLM” in this lecture specifically means large, decoder-only, text-generating next-token models.
An LLM is defined as a language model that predicts next-token probabilities and is “large” in parameters, training tokens, and compute; the speaker also notes the definition has shifted over time (e.g., excluding encoder-only models like BERT from “LLM” under the current text-generating definition).
MoE scales model capacity without requiring all parameters to be active per token.
Mixture of Experts routes tokens to a subset of feed-forward “expert” networks, reducing active compute while increasing total parameter capacity; sparse MoE selects top-K experts per token, typically at the FFN sublayer.
MoE training needs explicit mechanisms to prevent the router from overusing a small subset of experts.
A key training pitfall is routing collapse (only a few experts get used); the lecture describes adding an auxiliary loss to encourage more uniform expert utilization and mentions techniques like noisy gating (adding noise to router outputs).
Token selection strategy at inference materially changes output diversity and computational cost.
The lecture compares greedy decoding (deterministic, locally optimal), beam search (tracks multiple candidate sequences but is compute-heavy and can bias toward shorter outputs without length adjustment), and sampling methods (top-K/top-P) for more diverse generations.
WORDS WORTH SAVING
5 quotesSo a language model is a model that assigns probability to sequences of tokens.
— Unattributed
Typically, when we say LLM, we say at least on the order of a billion.
— Unattributed
Everything here is a trade-off. Everything here is a trade-off.
— Unattributed
The second problem is if you choose the highest probability token on a, I guess, iterative basis, you're locally optimal, but you're not necessarily globally optimal.
— Unattributed
The only thing that is not deterministic is how you sample the next token.
— Unattributed
High quality AI-generated summary created from speaker-labeled transcript.
