Stanford OnlineStanford CME295 Transformers & LLMs | Autumn 2025 | Lecture 1 - Transformer
Summaries, chapters, transcripts, and FAQs on this page are made by uListen, not YouTube.
At a glance
WHAT IT’S REALLY ABOUT
From tokens to self-attention: how transformers replace recurrence in NLP
- Afshine Amidi introduces CME295’s goals, prerequisites, logistics, and assessment scheme, emphasizing conceptual understanding and two exams.
- The lecture surveys core NLP task types—classification, multi-classification (e.g., NER), and generation (e.g., translation)—and describes common evaluation metrics and why some can be misleading under class imbalance or multiple valid outputs.
- It explains how text is turned into model-ready inputs via tokenization (word/subword/character) and how tokens are represented, moving from one-hot encodings to learned dense embeddings via Word2vec-style proxy tasks.
- It motivates the shift from RNNs/LSTMs to attention by highlighting RNN limitations: sequential computation, difficulty with long-range dependencies, and vanishing gradients.
- It introduces the transformer architecture and self-attention (Q/K/V, multi-head attention, masking, cross-attention, positional encodings), then walks through an end-to-end translation-style example from tokenization through decoding to EOS.
IDEAS WORTH REMEMBERING
5 ideasThe class is structured around understanding transformers plus training and applications.
He explains that the course aims to demystify the “underlying mechanism” behind modern LLMs—especially the transformer—and also cover how LLMs are trained and applied, with conceptual (non-coding) exams.
NLP problems fall into three broad task buckets with different evaluation challenges.
The lecture frames NLP tasks as (1) classification (text→label), (2) multi-classification (token/entity-level labels), and (3) generation (text→text), and connects each to typical evaluation approaches (accuracy/precision/recall/F1; token/entity-level aggregation; BLEU/ROUGE/perplexity).
Tokenization is a core design choice with accuracy and efficiency trade-offs.
Afshine Amidi describes tokenization as choosing the unit of text (word, subword, character), highlighting trade-offs like out-of-vocabulary risk, robustness to misspellings, and computational cost via longer sequences.
Dense token embeddings are learned via proxy prediction tasks rather than fixed encodings.
The lecture contrasts one-hot encodings (orthogonal vectors) with learned dense embeddings, and uses Word2vec-style proxy tasks (CBOW/Skip-Gram) to motivate how predicting context/targets yields representations that capture linguistic associations.
RNN-based sequence modeling adds order but struggles with long context and parallelism.
RNNs are introduced as a way to incorporate word order via a hidden state, but are described as slow to train (sequential dependence) and prone to long-range dependency issues and vanishing gradients; LSTMs are presented as an improvement via an added cell state.
WORDS WORTH SAVING
5 quotesLLMs are basically, uh, everywhere now, and I guess our goal here is twofold.
— Afshine Amidi
Models, they understand numbers, they don't really understand text.
— Afshine Amidi
This task is commonly called a proxy task, 'cause at the end of the day, in this exercise, what we care about is not necessary to predict the next word, or at least not yet. Our goal is to learn a representation of these words that are meaningful.
— Afshine Amidi
So the reason why you have not really heard of RNNs like this, like these days is because they had, uh, you know, some pros, but a lot of cons.
— Afshine Amidi
What the authors tried to do was to move away from this sequential way of processing the text, and instead let the model just have direct connections with all parts of the text at once.
— Afshine Amidi
SPEAKERS
Afshine Amidi
hostCo-instructor for Stanford CME295 (Transformers & Large Language Models), teaching transformer/LLM fundamentals.
High quality AI-generated summary created from speaker-labeled transcript.
