Stanford OnlineStanford CS25: V2 I Introduction to Transformers w/ Andrej Karpathy
Summaries, chapters, transcripts, and FAQs on this page are made by uListen, not YouTube.
CHAPTERS
- 0:06 – 2:39
Welcome to CS25 and Meet the Instructors
The course is introduced as being about deep learning transformer models rather than robots, and the instructors introduce themselves. They say transformers have spread from NLP to vision, reinforcement learning, biology and robotics.
- •The course was held at Stanford in winter 2023
- •Transformers are used across NLP, vision, RL, biology and robotics
- •Instructors share their backgrounds and research interests
- •Upcoming lectures feature speakers on how they apply transformers in different fields
- 2:39 – 5:01
Course Goals and the Attention Timeline
The instructors outline what students will learn and walk through a timeline from RNNs and LSTMs to Attention Is All You Need and the generative era. They cover how transformers spread into NLP, vision, biology and generative models up to ChatGPT.
- •Goals: how transformers work, where they are applied, and new research directions
- •Attention Is All You Need (Vaswani et al., 2017) began the transformer era
- •The prehistoric era had RNNs, LSTMs and simple attention that did not scale
- •2018-2020 saw spread into vision and biology; 2021 began the generative era
- •2022-2023 brought models like ChatGPT and Whisper
- 5:01 – 7:33
Why Attention Beats RNNs, and Where Transformers Stand
The instructor explains why RNNs and LSTMs struggle with long sequences and context, using the 'I grew up in France... I speak fluent' example. The talk then reviews progress from 2021 to 2022 in reasoning, RLHF and applications.
- •RNNs and LSTMs encode history but handle long sequences and context poorly
- •Attention maps help resolve references such as what 'it' points to
- •2021 was on the verge of takeoff, with AlphaFold, few-shot learning and DALL-E
- •2022 brought audio, art, music, reasoning, and RLHF as used for ChatGPT
- 7:33 – 10:14
The Future of Transformers and Missing Ingredients
The instructor looks ahead to video, finance, longer sequences, generalized agents and domain-specific models. The missing ingredients are external memory, lower computational complexity, controllability and alignment with the human brain.
- •Video understanding and generation, finance and business are expected applications
- •Most transformers are limited to around four thousand tokens, so long sequences must improve
- •Generalized agents like Gato and domain-specific models such as a 'doctor GPT' or 'lawyer GPT'
- •Missing: external memory, cheaper (quadratic) attention, controllability, brain alignment
- 10:14 – 15:09
Karpathy's Historical Context: Convergence of AI Toolkits
Andrej Karpathy describes entering AI around 2012, when computer vision pipelines relied on piles of hand-crafted features and each AI subfield had separate vocabulary. Neural networks unified the toolkit, and the transformer then converged architectures into one.
- •Pre-2012 vision pipelines combined many feature descriptors with an SVM and often failed
- •Each AI subfield, such as NLP, had its own vocabulary
- •AlexNet-style scaling showed neural nets scale with compute and data
- •The transformer converged architectures into one that can be copy-pasted
- •He suggests this may hint at the brain's uniform learning algorithm
- 15:09 – 20:11
From Neural Language Models to Attention
Karpathy traces the lineage from the 2003 neural language model through the 2014 sequence-to-sequence LSTM and its encoder bottleneck to soft attention. He shares an email from the first author, Dmitry, explaining the idea came from gaze shifting during translation exercises, with the name 'attention' suggested by Yoshua Bengio.
- •2003 MLP language model predicted the fourth word from three
- •Sequence-to-sequence LSTMs had an encoder bottleneck
- •Soft search over encoder states is the first use of attention
- •Dmitry's idea was inspired by translation exercises and worked on the first try
- •Yoshua suggested the name 'attention'
- 20:11 – 22:33
Attention Is All You Need and Its Resilience
Karpathy describes how the 2017 paper deleted the RNN and kept attention, combining residual connections, layer norms, multiple heads, positional encodings and MLPs. The architecture has stayed remarkably stable, with only small changes like pre-norm and rotary positional encodings.
- •The paper combined multiple ideas into a very good point in architecture space
- •Positional encoding is needed because attention operates over sets
- •Residual connections, LayerNorm, multi-head attention and the 4x MLP expansion persist
- •Pre-norm and rotary or relative positional encodings are the main adopted changes
- 22:33 – 27:23
Attention as Message Passing on a Graph
Karpathy presents attention as the communication phase of the transformer, with the MLP as the computation phase. Nodes in a directed graph emit queries, keys and values, and updates come from dot products, softmax and weighted sums, with encoder and decoder connectivity patterns.
- •Query is what a node seeks, key is what it has, value is what it communicates
- •Dot-product scores are softmax-normalized and used to weight values
- •Heads run in parallel and layers in series
- •Encoder tokens are fully connected; decoder tokens attend only to the past, giving a triangular structure
- •Cross-attention takes keys and values from the top of the encoder
- 27:23 – 30:50
Q&A on Self-Attention, Multi-Head and Cross-Attention
In response to an audience question, Karpathy explains multi-headed attention, layers, and the difference between self- and cross-attention. Questions about mapping the graph analogy to layers and nodes lead into the code implementation.
- •Multiple heads are independent parallel applications of attention with different weights
- •Layers repeat the process in series
- •Self-attention and cross-attention differ only in where keys and values come from
- •Nodes are token vectors, starting as embeddings
- 30:50 – 43:11
NanoGPT Walkthrough: Building a Decoder-Only Transformer
Karpathy steps through NanoGPT, a minimal implementation reproducing GPT-2, using the Tiny Shakespeare dataset. He covers tokenization, batching, embeddings, blocks with communicate and compute phases, causal masking, and generation.
- •Text is converted to integers; batches use block size and batch size, with many parallel training examples
- •Token and positional embeddings are added and fed through blocks
- •Each block has communication (masked multi-head attention) and computation (MLP)
- •Masking with negative infinity before softmax prevents seeing the future
- •Generation feeds samples back in and crops at block size
- 43:11 – 52:40
Encoder, Cross-Attention Variants and Audience Q&A on the Architecture
Karpathy explains how to get encoder-only or encoder-decoder models from the decoder code, mentioning BERT and T5. In Q&A he discusses fixed connectivity, why the transformer arrived as a package, his dislike of autoregressive sampling in favor of diffusion-like approaches, and graph neural network analogies.
- •Removing the mask gives an encoder; adding a cross-attention line gives encoder-decoder
- •GPT is decoder-only, BERT encoder-only, T5 encoder-decoder
- •Connectivity is usually fixed rather than data-dependent
- •The paper's authors were unaware of its impact
- •He finds diffusion-like drafting more appealing than committing token by token
- 52:40 – 55:43
Transformers Across Modalities and Their Flexibility
Karpathy describes how transformers are applied to images, speech, RL and proteins by chopping inputs into pieces and feeding them as tokens. He notes the flexibility of adding extra sensor information and freeing networks from Euclidean-space constraints.
- •Vision transformers chop images into patches
- •Whisper and decision transformers apply the same approach to audio and RL
- •AlphaFold has a transformer at its core
- •Extra inputs like radar or maps can be added as additional tokens, with attention figuring out communication
- 55:43 – 1:04:15
Why Transformers Work: In-Context Learning and General-Purpose Computing
Karpathy argues that in-context learning, as shown in the GPT-3 paper, is what makes transformers special, and may resemble gradient descent in the activations. He says the transformer is expressive, optimizable and efficient, and compares GPT to a general-purpose computer programmed through prompts.
- •Accuracy improves with more in-context examples without fine-tuning
- •Papers hint that transformers may implement something like gradient-based learning internally
- •Three properties: expressive forward pass, optimizable, and efficient on GPUs
- •Transformers are shallow and wide, unlike long thin RNN graphs
- •GPT is framed as a general-purpose computer reconfigurable by prompts
- 1:04:15 – 1:11:40
Final Q&A: Multimodal Inputs, Inductive Bias, Memory and Next Projects
Karpathy answers questions on feeding multiple modalities via special learnable tokens, positional encodings and inductive bias with limited data. He discusses scratchpad-style memory, ChatGPT forgetting, and says he is working on NanoGPT and sees potential in improving ChatGPT-like products.
- •Different modalities use learnable special embedding tokens, and positional vectors can be trained
- •With enough data, less inductive bias is needed; with little data, biases like convolutions help
- •Masking in attention can add structure and local communication
- •A scratchpad could extend memory beyond the finite context
- •He is working on NanoGPT and sees ChatGPT-like products as interesting
