Stanford OnlineStanford CS229 I Machine Learning I Building Large Language Models (LLMs)
Summaries, chapters, transcripts, and FAQs on this page are made by uListen, not YouTube.
At a glance
WHAT IT’S REALLY ABOUT
Building LLMs: Pretraining, Scaling Laws, Post-Training, Evaluation and GPU Systems Explained
- Dubois argues that data, evaluation and systems matter more in practice than architecture and loss tweaks, which is where academia tends to focus.
- Pretraining models the distribution of text with an autoregressive next-token objective trained with cross-entropy, relying on tokenizers (such as byte pair encoding) and evaluated with perplexity and benchmarks like MMLU.
- Scaling laws let teams predict performance from compute, data and parameters, guiding choices like model size versus tokens (Chinchilla's roughly 20 tokens per parameter) and informing cost estimates such as his roughly $75M back-of-envelope figure for Llama 3 400B.
- Post-training turns a raw language model into an assistant through supervised fine-tuning and preference optimization (PPO-based RLHF or the simpler DPO), increasingly using LLM-generated data and LLM judges instead of humans.
- Because compute is the bottleneck, systems tricks such as low-precision arithmetic, operator fusion and torch.compile help GPUs avoid sitting idle.
IDEAS WORTH REMEMBERING
5 ideasData, evaluation and systems matter more than architecture in practice
Dubois says academia over-focuses on architectures and losses, while industry spends its effort on data, evaluation and systems; he ties this to the 'bitter lesson' that scalable compute and data dominate small architectural differences.
Tokenizers are a quiet but important design choice
Tokens are more general than words and shorter than character sequences, which would be too long for quadratically scaling transformers. Tokenization quirks (such as numbers and code whitespace) can hurt math and code ability, so some hope to move away from tokenizers eventually.
Scaling laws make large training runs plannable
Loss falls predictably as a straight line on log-log plots of compute, data and parameters, so teams can tune on small models and extrapolate to the final run. Dubois says small architectural changes mostly shift the intercept, whereas data quality changes scaling more.
Evaluation is harder than it looks
Different implementations of MMLU gave very different scores for the same model (Llama 65B at 63.7 versus 48.8), and train-test contamination is a real problem for academics. Perplexity depends on the tokenizer and stops being meaningful after PPO-style alignment.
SFT mostly teaches format, not new knowledge
Dubois cites LIMA showing that going from 2,000 to 32,000 SFT examples helps little, because the knowledge already lives in the pretrained model. He also notes that SFT can encourage hallucination when humans write answers the model never actually learned.
WORDS WORTH SAVING
5 quotesIn reality, honestly, what matters in practice is mostly the three other topics, so data, evaluation, and systems, which is what most of industry actually focuses on.
— Yann Dubois
This is actually pretty different than what you see in this class. In this class, we teach you about overfitting. Overfitting doesn't happen with large language models. Larger models, better performance.
— Yann Dubois
All the architecture differences that we can make, like the small minor ones, all they do is maybe change a little bit the intercept. But really that doesn't matter, 'cause just train it for ten hours longer or, like, wait for the next GPUs.
— Yann Dubois
So don't spend time overcomplicating. Do the simple things, do it well, scale them.
— Yann Dubois
If you've ever been annoyed at ChatGPT answering you super long sentences, this is because of RLHF.
— Yann Dubois
High quality AI-generated summary created from speaker-labeled transcript.
