Stanford OnlineStanford CS229 I Machine Learning I Building Large Language Models (LLMs)
Summaries, chapters, transcripts, and FAQs on this page are made by uListen, not YouTube.
CHAPTERS
- 0:05 – 3:43
Overview: Key Components of Building LLMs
Yann Dubois introduces LLMs as the chatbots people know today and lists five key components: architecture, training loss and algorithm, data, evaluation, and systems. He says he will skip architecture and argues that data, evaluation and systems matter most in practice, then outlines pretraining versus post-training.
- •LLMs include ChatGPT, Claude, Gemini and Llama
- •Five components: architecture, loss/algorithm, data, evaluation, systems
- •Academia focuses on architecture and losses, while industry focuses on data, evaluation and systems
- •Pretraining models the internet; post-training turns models into AI assistants
- 3:43 – 6:39
Language Modeling and Autoregressive Models
Dubois recaps language models as probability distributions over token sequences, with examples of syntactic and semantic knowledge. He explains generative models and the autoregressive decomposition via the chain rule, including its downside of sequential sampling.
- •Language models give a probability for a sequence such as "The mouse ate the cheese"
- •Sampling from a distribution makes a model generative
- •Autoregressive models decompose the distribution with the chain rule, with no approximation
- •Generation is a for loop, so longer outputs take more time
- 6:39 – 10:43
Training Pipeline and Cross-Entropy Loss
The task is next-word prediction: tokenize, pass through the model, get a distribution, sample and detokenize. Dubois walks through embedding, a transformer, a linear layer and softmax, and shows the cross-entropy loss is equivalent to maximizing text log likelihood. A student question covers vocabulary size and the output dimension.
- •Only the sampling and detokenizing steps are needed at inference time
- •The output dimension equals the vocabulary size
- •The model embeds tokens, applies a transformer, a linear layer and a softmax
- •Cross-entropy against a one-hot target equals maximizing log likelihood
- 10:43 – 19:05
Tokenizers and Byte Pair Encoding
Dubois explains why tokenizers are needed: words are not general enough and characters give sequences that are too long. He describes byte pair encoding and answers questions on pre-tokenizers, keeping smaller tokens, and token uniqueness. He also discusses drawbacks such as math and code handling.
- •Tokens are more general than words and shorter than full character sequences
- •Byte pair encoding repeatedly merges the most common pairs of tokens
- •Pre-tokenizers avoid merging across spaces for efficiency
- •Apply the largest matching token; small tokens are kept to handle typos
- •Numbers and code tokenization cause problems, and some want to move away from tokenizers
- 19:05 – 26:05
Evaluating Pretrained LLMs: Perplexity and Benchmarks
Dubois introduces perplexity as the exponentiated average per-token loss and explains its range and intuition. He then covers aggregated academic benchmarks like HELM and MMLU, how multiple-choice evaluation works, and why perplexity is not used in academic benchmarking.
- •Perplexity ranges from one to vocabulary size and reflects how many tokens the model hesitates between
- •Perplexity went from about 70 to under 10 between 2017 and 2023
- •HELM and the Hugging Face Open LLM Leaderboard aggregate benchmarks
- •MMLU is evaluated by comparing the likelihood of answer choices
- •Perplexity depends on tokenizer choice
- 26:05 – 28:37
Evaluation Challenges: Inconsistency and Contamination
Dubois discusses two evaluation challenges: different organizations evaluate MMLU in different ways and get very different results, and train-test contamination. He describes a trick of comparing the likelihood of test examples in order versus shuffled to detect contamination.
- •Llama 65B scored 63.7 on HELM but 48.8 on another benchmark
- •Prompting and evaluation method both change results
- •Contamination matters more for academia because outsiders do not know the training data
- •Order-likelihood test detects whether a test set was in the training data
- 28:37 – 40:40
Pretraining Data: From Common Crawl to Clean Datasets
Dubois describes the pipeline for turning raw internet crawls into training data: text extraction, filtering undesirable content, deduplication, heuristic and model-based filtering, domain reweighting, and high-quality data at the end. He gives dataset sizes and notes the secrecy and open research problems.
- •Common Crawl holds around 250 billion pages and about a petabyte
- •Pipeline steps include extraction, safety filtering, deduplication and heuristic filtering
- •A classifier trained on Wikipedia-referenced pages helps filter quality
- •Domains are up- or down-weighted, and the end of training uses high-quality data
- •Best models train on about 15 trillion tokens; companies keep data practices secret
- 40:40 – 55:15
Scaling Laws and Optimal Training Resources
Dubois explains that more data and bigger models improve performance predictably on log-log plots, with no sign of plateauing. He shows how scaling laws change the tuning pipeline, compare architectures like transformers and LSTMs, and give the Chinchilla result of about 20 tokens per parameter.
- •Loss decreases linearly in log scale with compute, data and parameters
- •Tune on small models, fit a scaling law, then train one large model
- •Architecture tweaks mostly change the intercept; data quality matters more
- •Chinchilla found about 20 tokens per parameter, and about 150 when inference cost is included
- •The bitter lesson: leverage compute rather than overcomplicate
- 55:15 – 59:54
Back-of-the-Envelope Cost of Training Llama 3 400B
Dubois estimates the compute, time, cost and carbon footprint of training Llama 3 400B. He uses the 6NP FLOPs approximation and the executive order threshold, and notes that each generation multiplies FLOPs about tenfold.
- •15.6 trillion tokens and 405 billion parameters, about 40 tokens per parameter
- •About 3.8e25 FLOPs, just below the 1e26 threshold
- •Roughly 26 million GPU hours on 16,000 H100s, about 70 days
- •About $52 million in compute plus salaries gives roughly $75 million
- •About 4,000 tons of CO2 equivalent
- 59:54 – 1:09:54
Post-Training and Supervised Fine-Tuning
Dubois explains why a pure language model like GPT-3 is not an assistant, and introduces alignment through supervised fine-tuning on human-written answers. He covers LLM-generated data with Alpaca, the finding from LIMA that little data is needed, and questions on synthetic data and loss.
- •GPT-3 completes a question with similar questions rather than answering
- •SFT is language modeling on human-desired answers, using the same loss
- •Alpaca used 52,000 LLM-generated examples to fine-tune Llama 7B
- •LIMA showed scaling from 2,000 to 32,000 examples barely helps
- •SFT mainly specializes the model to one type of user it already saw in pretraining
- 1:09:54 – 1:27:39
RLHF: Reward Models, PPO and DPO
Dubois describes the problems with SFT behavioral cloning, including human limits, hallucination and cost. He explains RLHF with a Bradley-Terry reward model and PPO as used for ChatGPT, then DPO as a simpler maximum-likelihood alternative. He also discusses human versus LLM preference data.
- •SFT is bound by human ability and may encourage hallucination
- •A reward model trained on preferences supplies continuous rewards for PPO
- •PPO is complex, and PPO-tuned models no longer give meaningful likelihoods
- •DPO maximizes preferred outputs and minimizes rejected ones, performing as well as PPO
- •Human annotators favor length and are expensive; LLM annotators are cheaper with high agreement
- 1:27:39 – 1:37:06
Evaluating Post-Trained Models: Chatbot Arena and AlpacaEval
Dubois explains why post-trained models can't be evaluated with validation loss or perplexity, and describes Chatbot Arena and LLM-based AlpacaEval. He covers length bias in LLM judges and a regression fix, then answers questions on how few fine-tuning examples can change the weights.
- •Validation loss and perplexity aren't comparable or calibrated after alignment
- •Chatbot Arena uses blind human comparisons
- •AlpacaEval has 98% correlation with Chatbot Arena at under ten dollars and three minutes
- •Prompting for verbosity swings win rate, so length is controlled via regression
- •SFT changes all weights; RLHF uses around a million examples
- 1:37:06 – 1:44:30
Systems Tricks and Outlook
In the final minutes Dubois covers GPU basics, low-precision training, and operator fusion with torch.compile. He then lists topics he did not cover and recommends Stanford courses CS224N, CS324 and CS336.
- •GPUs are optimized for throughput and matrix multiplication, and compute outpaces memory
- •Model FLOP utilization of around 50% is considered good; Llama reached about 45%
- •Mixed precision computes in 16 bits and stores weights in 32 bits
- •Operator fusion and torch.compile cut communication and can double speed
- •Uncovered topics include inference, multimodality, legality and data scarcity
