Summaries, chapters, transcripts, and FAQs on this page are made by uListen, not YouTube.
CHAPTERS
Jump to publisher timestampsChapter summaries
Course logistics: midterm format, timing, and final exam details
The speaker opens with administrative announcements about the upcoming midterm and final. They describe timing, coverage, exam format, and what students can and cannot bring.
- •Midterm is one week away, held in the same classroom but shorter than a normal class session
- •Midterm covers Lectures 1–4 and includes multiple-choice plus free-form questions
- •Exam is closed book; only a pen is needed and no calculator is required
- •A study cheat sheet exists on the class website but cannot be brought to the exam
- •Final exam time/location are confirmed and it covers Lectures 5–9
Recap of prior lecture: Mixture of Experts and decoding strategies
The speaker briefly reviews key ideas from the previous lecture. This includes sparse vs. dense Mixture of Experts and common token decoding methods, plus a few inference optimizations.
- •Sparse MoE activates only some experts per forward pass to reduce inference cost while scaling parameters
- •Dense MoE combines expert outputs as weighted by a gating mechanism
- •Three decoding methods are reviewed: greedy decoding, beam search, and sampling
- •Temperature is described as shaping how “spiky” the sampling distribution is
- •Inference optimizations are mentioned, with KV cache called out as important
Training paradigm shift: transfer learning, pretraining, and tuning
The lecture transitions into how LLMs are trained, contrasting older task-specific training with transfer learning. The speaker frames LLM training as a two-stage approach: large-scale pretraining followed by task adaptation.
- •Traditional ML often trained separate models per task (e.g., spam detection vs. sentiment)
- •Transfer learning reuses a pre-trained model instead of starting from scratch for each task
- •LLM training is described as pretraining on vast data followed by a tuning stage
- •Pretraining aims to learn broad language/code structure; tuning adapts to specific tasks
- •LLMs are framed as text-to-text systems commonly using decoder-only transformers
What pretraining uses: internet-scale data and token-count scale
The speaker explains pretraining as the most expensive part of LLM training and describes typical data sources. They also give a sense of dataset scale in tokens using example figures mentioned in class.
- •Pretraining trains the model to predict the next token over very large corpora
- •Common Crawl is highlighted as a frequently cited source, alongside Wikipedia and social media conversations
- •Code sources like GitHub and Stack Overflow are mentioned as part of training data
- •Dataset sizes are discussed in tokens, on the order of hundreds of billions up to trillions (and beyond)
- •Example figures stated: GPT-3 trained on 300B tokens; Llama 3 trained on 15T tokens
Compute vocabulary: FLOPs vs. FLOP/s and why context matters
Two compute-related terms are introduced to help interpret training cost and hardware speed. The speaker emphasizes that papers can be inconsistent, so interpretation should follow context.
- •FLOPs (floating point operations) are presented as a unit measuring total compute required
- •An order-of-magnitude figure is given for LLM training compute (around 10^25 FLOPs)
- •Compute is described as depending on both number of tokens and number of model parameters (and architecture)
- •MoE models are noted as potentially requiring less compute than dense models due to partial activation
- •FLOP/s (floating point operations per second) is introduced as hardware throughput / speed
Scaling laws and the Chinchilla-style compute-optimal tradeoff
The speaker summarizes findings from scaling-law experiments and then explains compute-optimal training tradeoffs. They describe an empirical “sweet spot” linking model size and dataset size under fixed compute.
- •A 2020 scaling-laws paper is cited as finding performance improves with more compute, more data, and larger models
- •Larger models are described as more sample efficient given the same number of processed tokens
- •The lecture frames the practical constraint of fixed compute budgets
- •An empirical compute-optimal relationship is presented, sometimes called the Chinchilla law
- •The rule of thumb stated: training tokens around 20× the number of parameters is compute-optimal; GPT-3 is described as under-trained under this lens
Pretraining constraints: cost, knowledge cutoff, and memorization/plagiarism risks
The speaker discusses practical and societal challenges of pretraining. They explain knowledge cutoff as a consequence of training on a dataset frozen in time and mention difficulties of updating knowledge without regressions.
- •Pretraining is described as extremely expensive, potentially reaching tens or hundreds of millions of dollars
- •Time and ecological/environmental costs are mentioned as concerns included in discussions and papers
- •Knowledge cutoff date is defined as the latest time represented by the training data
- •Updating or editing knowledge in weights is described as tricky because it can cause regressions elsewhere
- •Memorization and plagiarism risk are noted when models reproduce training content
Why training needs parallelism: memory limits, forward/backward passes, and optimizer states
The lecture shifts to systems concerns: what must be stored during training and why GPU memory becomes the limiting factor. The speaker walks through forward pass, backward pass, and optimizer bookkeeping requirements.
- •Training is broken into forward pass (activations), loss computation, backward pass (gradients), and weight update
- •Memory usage depends on model size, batch size, and context length (attention’s sequence-length scaling is referenced)
- •Gradients must be stored, and optimizer methods like Adam require additional stored state (first and second moments)
- •A GPU example is used to show limited device memory (e.g., 80GB on a cited GPU)
- •The need to distribute work across multiple GPUs is motivated by these memory constraints
Data parallelism and ZeRO: sharding redundant states to fit large training jobs
The speaker introduces data parallelism and explains how gradients are synchronized across workers. They then present ZeRO as a way to reduce duplication by sharding optimizer state, gradients, and even parameters across GPUs, trading memory for communication.
- •Data parallelism splits the batch across GPUs while keeping a model replica on each device
- •Gradients are aggregated across GPUs (averaged) to perform consistent weight updates
- •Communication overhead is called out as a downside of multi-GPU training
- •ZeRO is introduced as “Zero Redundancy Optimization” to reduce duplicated storage across devices
- •ZeRO variants shard optimizer states, gradients, and/or parameters, reducing memory but increasing communication
Model parallelism catalog: expert, tensor, and pipeline parallel approaches
A second family of parallelization methods is described: model parallelism. The speaker gives a high-level overview of multiple subtypes and how they split computation within a batch.
- •Model parallelism is presented as parallelizing computation within a batch rather than just splitting data
- •Expert parallelism is connected to MoE, placing different experts on different devices
- •Tensor parallelism is described as splitting large matrix multiplications across devices
- •Pipeline parallelism assigns different layer ranges to different GPUs to stage the forward pass
- •The lecture emphasizes knowing the broad categories rather than detailed mechanics
FlashAttention: using HBM vs. SRAM and tiling to reduce IO bottlenecks
The speaker explains FlashAttention by first describing GPU memory hierarchy. They show how standard attention repeatedly reads/writes large intermediates to slower memory and how tiling computes attention in fast on-chip memory to reduce memory traffic without approximation.
- •GPU memory hierarchy is framed as large-but-slower HBM versus smaller-but-faster on-chip SRAM
- •Standard attention implementation incurs heavy HBM read/write due to intermediate matrices and softmax
- •FlashAttention is presented as an exact (non-approximate) method that reduces HBM IO via tiling blocks into SRAM
- •A key idea is that softmax can be computed in blocks using a scaling trick across submatrices
- •Questions address practical details like computing portions of rows and computing scaling quantities iteratively and exactly
FlashAttention recomputation: trading extra FLOPs for lower memory and faster runtime
The lecture highlights a second FlashAttention idea: not storing all activations from the forward pass. Instead, activations are recomputed during backprop because attention computation becomes fast enough, reducing memory and even improving runtime in reported comparisons.
- •Backprop requires activations; a strategy is introduced to discard and recompute them during the backward pass
- •This is described as recomputation enabled by faster attention kernels
- •Reported comparisons show significantly fewer HBM reads/writes versus standard attention
- •Despite more operations (more FLOPs), runtime is described as lower, making it “best of both worlds”
- •The existence of FlashAttention-2 and FlashAttention-3 is noted as adapting techniques to newer hardware
Quantization and mixed precision training: float formats, memory, and speed tradeoffs
The speaker introduces quantization as reducing numeric precision to save memory and increase throughput. They connect this to floating-point bit layouts and to GPU speed differences by datatype, then describe mixed precision training as a practical compromise.
- •Quantization is defined as converting numbers from higher precision to lower precision to save memory
- •Floating-point representations are described via sign, exponent, and mantissa bits (e.g., FP32 vs. FP16 vs. BF16)
- •GPU throughput varies by datatype; lower precision can yield higher compute speed
- •Mixed precision training keeps model weights in higher precision (FP32) while running forward/backward in lower precision (FP16) and updating weights in FP32
- •A question notes that precision strategies can vary by layer/weight type and are the subject of additional research
From pretraining to supervised fine-tuning: making models helpful and instruction-following
A new speaker transitions to post-pretraining stages and motivates fine-tuning with an example showing that a base next-token model may not answer user questions helpfully. They define supervised fine-tuning (SFT) and explain how instruction tuning is a form of SFT for assistant behavior.
- •A base next-token model may continue text rather than act like a helpful assistant, illustrated with a teddy-bear washing prompt
- •Fine-tuning is introduced as training on labeled input/output pairs to shape desired behavior
- •SFT is defined as supervised fine-tuning: starting from pretrained weights and further training on curated examples
- •The loss is applied to the output tokens conditioned on the input; the input itself is not trained via teacher forcing
- •Instruction tuning is introduced as SFT specifically focused on responding to user instructions
Instruction-tuning data mixtures: assistant tasks, synthetic data, and safety behaviors
The speaker describes what instruction-tuning datasets look like and why they differ from pretraining corpora. They discuss task categories, increasing use of model-generated data with review, and inclusion of safety-oriented examples to shape refusals and hedging.
- •Instruction-tuning data is organized around tasks users ask for (e.g., story writing, poem creation, list generation, explanation)
- •The model is trained to predict the response given an instruction prompt
- •Data can be human-written and increasingly can be generated by existing LLMs with human or model-based quality review
- •Safety-oriented data is included to reduce harmful behaviors and to teach refusal patterns when appropriate
- •The speaker mentions behaviors like hedging/nuance and refusal responses being embedded into model behavior rather than relying on simple pattern matching
How big is SFT compared to pretraining, and what changes in model behavior
Orders of magnitude are given to show SFT datasets are far smaller than pretraining datasets. The speaker revisits the teddy-bear example to show that instruction-tuned models respond more directly and helpfully, then outlines key challenges in SFT.
- •Dataset scale is contrasted using example counts: GPT-3 SFT examples (about 13K) versus Llama 3 (about 10M examples)
- •A rough estimate illustrates SFT token volume is several orders of magnitude smaller than pretraining
- •Instruction tuning is framed as aligning the model to be useful for user tasks rather than only modeling text continuation
- •Challenges include the cost of high-quality data and the importance of matching the prompt distribution to intended use
- •Sampling randomness and temperature are discussed as affecting whether outputs reproduce training stories verbatim
Evaluating instruction-tuned models: benchmarks, leaderboard pitfalls, and alignment framing
The speaker surveys evaluation approaches, from standardized benchmarks to human-preference leaderboards. They warn about pitfalls like training on benchmark task types, noise and manipulability in pairwise ranking systems, and subjectivity in user preferences and safety tradeoffs. They close by framing “alignment” as fine-tuning plus preference tuning and mention an emerging “mid-training” stage.
- •Benchmarks are grouped by capability (general knowledge, reasoning, math, and code), with MMLU mentioned as a common headline score
- •A paper is cited discussing “training on the test task,” emphasizing comparisons must consider training data parity
- •Chatbot Arena is described as a user-preference leaderboard based on pairwise comparisons, with noted issues like early-match noise and potential rigging
- •User preferences can diverge (e.g., emojis) and may bias rankings; safety refusals can be penalized by preference voting
- •Alignment is defined here as fine-tuning plus preference tuning; mid-training is mentioned as an emerging intermediate step after pretraining
Parameter-efficient finetuning with LoRA and the QLoRA idea
The speaker introduces LoRA as a way to fine-tune models efficiently by freezing base weights and training low-rank adapter matrices. They describe practical observations about where LoRA is applied and training dynamics, then briefly outline QLoRA as combining frozen-weight quantization with LoRA adapters trained in higher precision.
- •LoRA freezes pretrained weights and learns an additive low-rank update using two smaller matrices
- •The low-rank dimension is described as small compared to the full weight matrix, greatly reducing trainable parameters
- •LoRA adapters can be task-specific, enabling multiple task variants derived from one base model
- •Empirical notes include using higher learning rates and that LoRA may perform worse with larger batch sizes; rank is a design choice and can be tuned
- •QLoRA is summarized as quantizing the frozen base weights (using an NF4 quantization scheme described as assuming normally distributed weights) while training LoRA adapters in higher precision, aiming for large VRAM savings
Publisher timestamps
- 0:00Introduction
- 7:19Pretraining
- 13:26FLOPs, FLOPS
- 16:34Scaling laws, Chinchilla law
- 24:49Training optimizations overview
- 31:09Data parallelism with ZeRO
- 35:51Model parallelism
- 38:26Flash Attention
- 52:37Quantization
- 56:00Mixed precision training
- 1:02:31Supervised finetuning
- 1:09:21Instruction tuning
- 1:37:53Parameter-efficient finetuning with LoRA
- 1:45:16QLoRA
