Summaries, chapters, transcripts, and FAQs on this page are made by uListen, not YouTube.
CHAPTERS
Jump to publisher timestampsChapter summaries
Course recap: pretraining, SFT, and RLHF foundations for reasoning
The lecturer frames the session as a continuation of preference tuning, emphasizing that the RL tools from the prior lecture will be reused for reasoning. They recap the three-stage training pipeline (pretraining, supervised fine-tuning, preference tuning) and how PPO-style objectives keep models from drifting too far from prior or base behavior.
- •Pretraining produces a next-token autocomplete model with broad language/code knowledge
- •SFT uses curated instruction data to make the model useful for tasks like answering questions
- •Preference tuning aligns outputs with human preferences via RLHF
- •PPO mechanisms (ratio/clipping and KL penalty) limit overly large policy updates
- •Modern RLHF commonly applies KL regularization relative to the SFT/base model
Why “vanilla” LLMs struggle: limited reasoning, stale knowledge, no actions, hard evaluation
The lecturer contrasts strong generative abilities with key limitations that motivate reasoning-focused training. They highlight limited multi-step problem solving, static training cutoffs, inability to take actions, and the difficulty of evaluating free-form outputs with traditional NLP metrics.
- •Vanilla LLMs excel at fluent text and code generation but can fail on multi-step math/coding problems
- •Models’ knowledge is bounded by a training cutoff date
- •LLMs are “all talk” in the sense that prompting alone doesn’t execute real-world actions
- •Free-form generation complicates evaluation compared with tasks scored by BLEU/ROUGE
- •The lecture’s focus will be improving reasoning; other weaknesses are deferred to later lectures
Defining reasoning and connecting it to chain-of-thought at scale
Reasoning is defined (imprecisely, but pragmatically) as the ability to solve problems—especially math and coding—via multi-step processes. The lecturer ties this to chain-of-thought prompting and explains why generating intermediate steps can help models solve hard problems by decomposing them into tractable parts and using more generation steps as additional compute.
- •Reasoning is treated as solving problems that require multi-step decomposition
- •Example contrast: factual recall vs a small multi-step math question
- •Chain-of-thought encourages “thinking in steps” before the final answer
- •Decomposition helps because hard problems may not appear directly in training data
- •Longer reasoning chains can be viewed as allocating more compute budget during generation
What “reasoning models” look like in products: hidden thoughts, token billing, and UI cues
The lecturer explains how reasoning models differ operationally: they generate a reasoning chain plus an answer, though UIs often show only a summary of the thoughts. They note possible reasons for hiding raw chains and highlight that users are typically billed for reasoning tokens as output tokens, creating pressure to achieve strong reasoning with fewer tokens.
- •Reasoning-model output conceptually includes reasoning tokens plus final answer
- •Interfaces often indicate “thinking” but display only a summary rather than raw chains
- •Hypothesized reasons: raw chains may be unintelligible, too long, or useful for imitation/training by others
- •APIs generally charge for output tokens including reasoning tokens
- •This incentivizes efficient reasoning: high capability with minimal reasoning length
Reasoning benchmarks overview: coding and math evaluation setups
The lecture surveys common benchmark types used to quantify reasoning. Coding benchmarks are evaluated by passing test cases, while math benchmarks rely on parsing a model’s final answer into a comparable form and checking against ground truth.
- •Coding evaluation: correctness verified via compilation/tests and passing test cases
- •Examples named: HumanEval, Codeforces-style problems, and SWE-bench derived from real issues
- •Math evaluation: parse a standardized final answer and compare to ground truth
- •Prompts can force answers into parsable formats (e.g., boxed/bracketed answers)
- •Benchmarks are designed to be non-trivial and stress multi-step reasoning
Pass@k explained: motivation, derivation intuition, and temperature tradeoffs
The lecturer introduces Pass@k as the probability that at least one of k generated attempts succeeds, motivating it for settings where multiple tries are affordable and verifiable. They walk through the sampling-without-replacement logic leading to the standard estimator and discuss how sampling temperature affects diversity and success rates.
- •Pass@k measures the chance at least one of k generated solutions is correct
- •Multiple attempts are useful when correctness is automatically verifiable (e.g., tests/ground truth)
- •Estimator uses n samples with c successes to estimate success for subsets of size k
- •Pass@1 reduces to the intuitive single-attempt success probability
- •Temperature impacts Pass@k: too low lacks diversity; too high harms per-sample quality; papers must report temperature settings
Why RL is the scalable route for reasoning: verifiable rewards vs SFT chains
The lecture argues that large-scale human-written reasoning chains are hard to obtain and may not match how models “want” to reason. Because many reasoning tasks offer verifiable correctness signals, RL can directly optimize for producing a reasoning format and a correct result using simple reward checks.
- •SFT is limited when high-quality reasoning chains are unavailable or expensive to create
- •Human reasoning demonstrations may not align with model-internal reasoning strategies
- •Reasoning tasks often provide verifiable rewards (tests for code, exact answers for math)
- •Proposed RL reward combines format compliance (presence of think markers) and correctness
- •Training curves shown indicate substantial benchmark gains when optimizing these verifiable rewards
Controlling and representing “thinking”: dynamic budgets, budget forcing, and continuous thoughts
The lecturer notes that not all prompts require the same amount of thinking and that long reasoning competes with context-window constraints. They describe ideas for controlling reasoning length, including classifiers for allocating budget, prompt-based forcing tokens to extend/stop thinking, and research directions that replace discrete token thoughts with compressed continuous representations.
- •Some prompts merit extensive reasoning; others risk “overthinking” if not controlled
- •Context window limits require awareness of remaining length during reasoning
- •Budget forcing can push models to continue (e.g., injecting prompts like “wait”) or to stop and answer
- •Models may backtrack mid-reasoning, producing alternate solution paths
- •Research explores “continuous thoughts” as hidden-state reasoning to compress and improve thought representations
GRPO: group-based RL that removes the value function bottleneck
GRPO (Group Relative Policy Optimization) is introduced as the dominant RL algorithm for reasoning training, differing from PPO mainly in advantage estimation. Instead of training a value function, GRPO samples multiple completions per prompt and uses relative performance within the group to compute advantages.
- •GRPO still aims to maximize advantage while limiting deviation from a reference via KL regularization
- •Key change vs PPO: no learned value function; advantage comes from group-relative rewards
- •Multiple completions per prompt provide a baseline (group mean) for relative scoring
- •Advantages are normalized using group statistics (mean and standard deviation) in the described setup
- •Motivation includes avoiding expensive joint value training and better contextualizing difficulty across prompts
GRPO vs PPO mechanics: pipelines, objectives, and where KL appears
The lecturer compares GRPO and PPO end-to-end: how many completions are sampled, what models are trained, and how KL regularization is incorporated. They emphasize that, for reasoning with verifiable rewards, a learned reward model may be unnecessary, and the main architectural distinction becomes whether a value model is trained.
- •PPO commonly samples one completion and trains both a policy model and a value model
- •GRPO samples multiple completions and trains only the policy model
- •In practice, PPO often folds token-level KL penalties into per-token rewards used for advantage estimation
- •GRPO includes KL explicitly in its objective while using group-relative advantages
- •For reasoning tasks, reward can be computed verifiably, so neither approach strictly requires a learned reward model in this setting
Length bias in GRPO and fixes: DAPO and Dr. GRPO-style adjustments
The lecture analyzes why GRPO training can lead to ever-increasing output lengths, even after performance saturates. By inspecting the objective’s length normalization, the lecturer explains how token contributions differ for short vs long outputs and how recent modifications adjust or remove this factor to reduce incentives for long incorrect outputs.
- •Empirically, RL training can correlate improved performance with longer outputs, then continue lengthening after performance plateaus
- •A length-normalization factor can overweight tokens in short outputs relative to long ones
- •This creates an unintended incentive structure that can prefer longer incorrect outputs over shorter incorrect ones
- •DAPO modifies normalization to equalize token-level contributions across outputs
- •Another approach (described as Dr. GRPO) removes the length factor; comparisons show reduced length growth particularly for incorrect solutions
Additional GRPO tweaks: advantage standard deviation issues and asymmetric clipping
The lecturer briefly notes further refinements that practitioners apply beyond length normalization. These include concerns about standard deviation normalization in group advantages for hard problems and adjusting clipping behavior asymmetrically to better handle tokens with very low probability without destabilizing high-probability tokens.
- •Group standard deviation in advantage normalization can behave poorly on very hard prompts with uniformly bad completions
- •Clipping epsilon influences how much the policy can change per iteration
- •A fixed symmetric epsilon can be unfair for very low-probability tokens (limited room to grow)
- •Too-large epsilon risks collapsing high-probability tokens; asymmetry can mitigate this
- •These are presented as pragmatic modifications emerging from early-2025 follow-on work
DeepSeek R1-Zero: RL-only reasoning with formatting rewards (and observed issues)
A second lecturer (Shervin) shows how DeepSeek’s R1-Zero starts from a base pretrained model and applies RL directly, without a prior instruction-tuning stage, using verifiable correctness and formatting-based rewards. While benchmark performance rises, the generated reasoning chains exhibit problems like language mixing and syntax issues.
- •R1-Zero begins with a pretrained base model and proceeds directly to RL on reasoning data
- •Training uses simple templates requiring thoughts in designated think blocks and answers in answer blocks
- •Rewards include correctness and adherence to required output formatting
- •Performance improves over RL steps on math reasoning benchmarks as shown in their results
- •Observed drawbacks include mixed languages and poor readability/structure in reasoning chains due to lack of prior supervision
DeepSeek R1 full pipeline: cold-start SFT, RL with extra rewards, then mixed SFT and final RL
Shervin outlines DeepSeek R1 as a multi-stage process designed to fix R1-Zero’s weaknesses. It starts with a small human-rewritten “cold start” SFT set for clean formatting, runs RL with added language-consistency rewards, then performs larger-scale SFT mixing reasoning and non-reasoning data, and ends with a final RL stage that also targets helpfulness and harmlessness on non-reasoning prompts.
- •Cold-start stage: humans rewrite reasoning chains for format and language consistency, then SFT trains on these pairs
- •First RL stage: uses verifiable rewards plus formatting and a language consistency heuristic reward
- •Second SFT stage: mixes non-reasoning instruction data with a larger amount of reasoning data
- •Reasoning SFT data is produced via rejection sampling, filtering to high-quality responses using automated judging
- •Final RL stage mixes reasoning rewards with assistant-style alignment signals including helpfulness and harmlessness, with harmlessness applied across the full output including thoughts
Distilling reasoning into smaller models: sequence-level imitation vs RL from scratch
The lecture closes by revisiting distillation, adapted for reasoning models. Instead of matching teacher probability distributions token-by-token, the described approach has a strong teacher generate full reasoning sequences (including thinking tokens), then trains smaller student models via SFT to reproduce those sequences; the authors report this can outperform training smaller models via RL alone.
- •Teacher reasoning model generates offline datasets containing reasoning (think) and answers
- •Student training uses SFT to imitate the full token sequence output by the teacher
- •This differs from earlier distillation framing that emphasized matching full probability distributions
- •Results shown indicate competitive performance for distilled smaller models versus certain closed-source small reasoning offerings
- •Authors claim distillation at smaller scales can be more performance-efficient than applying RL techniques from scratch
