Stanford OnlineStanford CME295 Transformers & LLMs | Autumn 2025 | Lecture 1 - Transformer
Summaries, chapters, transcripts, and FAQs on this page are made by uListen, not YouTube.
CHAPTERS
Jump to publisher timestampsChapter summaries
Course and instructor background: why this LLM class exists
Afshine Amidi introduces CME 295 and explains that he and Shervin are twin brothers with similar academic and industry paths. He describes how their earlier NLP workshops evolved into a Stanford course amid growing interest in large language models.
- •Afshine and Shervin introduce themselves and their shared background and work on large language models
- •The class grew out of yearly NLP/LLM workshops and became a Stanford course more recently
- •The course motivation: understand transformer mechanisms plus how LLMs are trained and applied
- •Who the class is for: career interest, personal projects, or domain understanding of LLMs
- •Prerequisites emphasized: basic ML training concepts and matrix multiplication basics
Logistics, grading, and course resources (recordings, exams, Q&A)
The lecture covers scheduling, grading structure, and how materials will be distributed. Afshine answers common administrative questions about exams, waitlist, slides, and communication channels.
- •Meeting time/location and that lectures are recorded for later viewing
- •Two exams determine the grade: midterm and final; no homework
- •Midterm timing is specified; final is in the December exam period with date TBD
- •Slides, recordings, syllabus, and supporting materials are posted on the course website and Canvas
- •Questions handled via Canvas/Ed, with a note that in-class questions may be repeated for the recording
NLP task landscape: classification, sequence labeling, and generation
Afshine frames natural language processing as tasks that map text to labels or text to text. He outlines three broad buckets and gives concrete examples that will connect to later model discussions.
- •Definition of NLP as computation over text
- •Three task buckets: classification, multi-classification (sequence labeling), and generation
- •Examples of classification tasks: sentiment, intent detection, language detection, topic modeling
- •Example of multi-classification: named entity recognition; also part-of-speech tagging and parsing mentioned
- •Generation tasks highlighted: translation, question answering, summarization, and code/creative generation
How NLP systems are evaluated: accuracy vs precision/recall, BLEU/ROUGE, perplexity
The lecture reviews evaluation metrics across task types and why metric choice matters. Afshine emphasizes class imbalance for classification and the difficulty of evaluating generation where multiple outputs can be correct.
- •For classification, accuracy can be misleading under strong class imbalance
- •Precision, recall, and F1 are reviewed conceptually
- •For generation/translation, reference-based metrics like BLEU and ROUGE are introduced
- •A limitation of BLEU/ROUGE is needing labeled reference text, which is costly
- •Perplexity is introduced as probability-based; lower is better, unlike BLEU/ROUGE
From text to tokens: word, subword, and character tokenization trade-offs
Afshine explains that models need numeric inputs and introduces tokenization as the first step. He compares tokenizing by words, subwords, or characters, focusing on practical pros/cons like out-of-vocabulary risk and sequence length.
- •Tokenization defined as splitting text into tokens (units) for modeling
- •Word-level tokenization is simple but treats related variants (e.g., singular/plural) as unrelated tokens
- •Subword tokenization leverages shared roots, reducing out-of-vocabulary issues but increases sequence length
- •Character-level tokenization is robust to misspellings/casing but produces very long sequences and unclear token semantics
- •Longer sequences increase computational cost because model complexity depends on sequence length
Choosing vocabulary size and handling unknown tokens
A student question leads to discussion of vocabulary size and multilingual settings. Afshine connects vocabulary size to tokenizer choice and explains how unknown tokens are used when unseen tokens appear at inference time.
- •Vocabulary size depends on task scope (single-language vs multilingual/code)
- •Afshine gives order-of-magnitude targets: tens of thousands for one language; sometimes hundreds of thousands for multilingual settings
- •Language differences (e.g., different character systems) affect how tokenizers are constructed
- •Out-of-vocabulary (OOV) is a key word-level weakness; subword tokenization mitigates it
- •A reserved unknown token is commonly used to bucket unseen tokens at inference time
Token representations: why one-hot fails and what embeddings should capture
Afshine introduces token/word representations and shows why one-hot vectors are inadequate for similarity. He explains the goal: embeddings where semantically related tokens are closer under measures like cosine similarity.
- •One-hot encoding yields orthogonal vectors, making all tokens equally dissimilar under cosine similarity
- •Desired property: semantically related tokens should have higher similarity than unrelated ones
- •Cosine similarity is introduced as a common comparison tool (angle-based intuition)
- •A question about norms leads to the point that cosine similarity normalizes by vector magnitudes
- •Motivating example: wanting “teddy bear” to be closer to “soft” than to “book”
Learning embeddings with Word2vec-style proxy tasks (CBOW/Skip-gram) and a toy network
Afshine explains how embeddings can be learned from data using proxy objectives, highlighting Word2vec. He walks through a simplified neural network that predicts next tokens and uses the hidden layer as the learned representation.
- •Word2vec popularity and the intuition of embeddings supporting meaningful relationships
- •Two Word2vec approaches named: Continuous Bag of Words (context → target) and Skip-gram (target → context)
- •Proxy-task framing: predicting words is a means to learn meaningful representations
- •Toy example: one-hot input → hidden layer (dimension D) → softmax over vocabulary; trained with cross-entropy and backprop
- •Learned embeddings correspond to the hidden representations produced for each token
Training questions: convergence, stopping generation, and embedding dimension choices
Student questions steer discussion toward practicalities: when to stop training a proxy task, how text generation stops, and how to choose embedding dimensionality. Afshine emphasizes trade-offs between expressiveness and compute/latency.
- •A practical stopping criterion is monitoring loss until it converges across epochs
- •Generation is typically terminated when an end-of-sequence token is produced
- •Embedding dimension choice is described as empirical and task-dependent
- •Trade-offs include downstream task complexity vs compute cost and latency
- •Afshine references common embedding-size orders of magnitude (hundreds to thousands) as typical practice
Sequence modeling with RNNs and LSTMs: benefits and the long-range dependency problem
Afshine transitions from token embeddings to sentence-level meaning and introduces recurrent neural networks. He explains how RNNs encode order via hidden states, then motivates LSTMs and discusses vanishing gradients and slow sequential computation as key drawbacks.
- •Averaging word embeddings loses order and meaning, motivating sequence models
- •RNNs process tokens sequentially while carrying a hidden state representing the sequence so far
- •How RNN outputs can be used for classification, sequence labeling, and generation tasks
- •RNN limitation: information bottleneck in the hidden state leads to long-range dependency issues (vanishing gradients)
- •Sequential dependence also makes RNN training/inference slow for long sequences
Attention idea: creating direct links to relevant parts of the input
Afshine introduces attention as a way to address long-range dependency problems by allowing direct connections to earlier tokens. He uses machine translation as the motivating example where a decoder needs to 'peek' at relevant source words.
- •Attention provides direct connections between what is being predicted and earlier tokens
- •Motivation illustrated with translation: the decoder benefits from focusing on specific input parts
- •Attention is presented as a response to RNNs forgetting earlier content over long sequences
- •The class positions attention as central to transformer success
- •Attention is referenced as introduced before transformers and then leveraged heavily in the transformer approach
Self-attention and Q/K/V intuition: contextual token meanings
Afshine explains the transformer’s self-attention concept: each token representation is computed using all tokens in the sequence. He introduces query, key, and value terminology and clarifies that these are learned projections used to compute weighted combinations of information.
- •Self-attention computes each token’s representation as a function of all tokens, not sequentially
- •Contextualization addresses same-spelling/different-meaning cases by producing context-specific representations
- •Q/K/V framing: compare a query to keys to compute weights, then combine corresponding values
- •Afshine emphasizes Q/K/V are learned, not fixed, and come from learned projections
- •A matrix formulation is highlighted as GPU-friendly and efficient
Transformer encoder–decoder architecture: self-attention, cross-attention, masking, and position encodings
Afshine presents the high-level transformer layout with an encoder processing the source text and a decoder generating the target text. He explains the roles of encoder self-attention, decoder masked self-attention, decoder cross-attention, and why positional information must be added explicitly.
- •Transformer described as encoder (source) + decoder (target) for translation
- •Encoder multi-head self-attention lets all source tokens attend to each other
- •Decoder masked self-attention attends only to already-generated tokens (causal masking)
- •Cross-attention uses decoder queries with encoder keys/values to condition on the source sentence
- •Positional encodings are needed because self-attention lacks inherent order information
Multi-head attention and label smoothing (training detail)
The lecture clarifies what 'heads' mean—multiple parallel attention projections—and why they help. Afshine also introduces label smoothing as a way to avoid treating next-token labels as absolutely certain in settings with multiple plausible continuations.
- •Multi-head attention runs attention in parallel with different learned projection matrices
- •Heads provide additional degrees of freedom and tend to learn different useful associations without explicit constraints
- •A question is addressed about whether heads could learn the same thing; the model is not constrained but tends to diversify during learning
- •Label smoothing changes training targets away from strict one-hot labels to reduce overconfidence
- •Afshine notes label smoothing improved a translation metric in the original transformer context (as described in the lecture)
End-to-end worked example: tokenization to encoder self-attention, then decoding with cross-attention
Shervin walks through the full transformer pipeline using the running example sentence. He details how token and positional embeddings form a matrix input, how Q/K/V projections produce attention weights, how multi-head outputs are concatenated, and how decoding proceeds token-by-token until EOS.
- •Sequence preparation: tokenization plus BOS/EOS tokens and adding positional embeddings
- •Encoder step-by-step: project embeddings to Q, K, V using learned matrices and apply softmax(QKᵀ/√d)·V
- •Explanation of scaling by √d to normalize dot products as dimensions grow
- •Multi-head attention: compute multiple heads in parallel, concatenate, and project back with an output projection matrix
- •Decoder loop: start from BOS, apply causal self-attention, cross-attend to encoder outputs, project to vocabulary with softmax, and stop at EOS
