Summaries, chapters, transcripts, and FAQs on this page are made by uListen, not YouTube.
At a glance
WHAT IT’S REALLY ABOUT
How LLMs learn reasoning via verifiable RL and GRPO training
- The lecture defines LLM reasoning as multi-step problem solving and frames “reasoning models” as models that output a reasoning chain before giving a final answer.
- It reviews how reasoning is benchmarked in coding and math, emphasizing verifiable correctness checks and sampling-based metrics such as Pass@k, along with the role of temperature in diversity vs. quality.
- It argues that RL is a natural fit for reasoning training because writing high-quality reasoning chains for SFT is difficult and because many reasoning tasks provide direct, verifiable reward signals.
- It introduces GRPO as a reasoning-focused alternative to PPO that computes advantages from groups of sampled completions rather than training a value function, and discusses recent modifications aimed at mitigating output length growth.
- It uses DeepSeek’s R1-Zero and R1 pipelines to show how verifiable-reward RL, small cold-start SFT, rejection-sampled SFT data, and mixed RL objectives can be combined, and it closes with distillation via teacher-generated reasoning traces to train smaller student models.
IDEAS WORTH REMEMBERING
5 ideasReasoning training is positioned as an extension of the same RL machinery used for preference alignment.
The lecture frames pre-training as producing an autocomplete model, fine-tuning as making it useful for tasks like answering questions, and preference tuning (e.g., RLHF) as aligning to human preferences; the RL stage mechanics (advantages plus constraints against drifting too far) become the foundation for reasoning training.
In this lecture, a “reasoning model” is a model that generates intermediate reasoning text plus an answer.
Reasoning is defined (with a caveat that there’s no agreed definition) as the ability to solve problems—especially math and coding—via multi-step decomposition, operationalized by having the model output a reasoning chain before the final answer.
Verifiable rewards make RL an attractive way to train reasoning without large human-written chain-of-thought datasets.
The speaker argues that supervised fine-tuning is a poor starting point if you don’t already have high-quality reasoning chains, and also notes that human-written reasoning may not match the model’s most effective internal reasoning style; meanwhile many reasoning tasks have verifiable correctness signals.
Reasoning evaluation often relies on sampling multiple solutions and measuring success-at-k rather than single-shot accuracy.
For coding, correctness can be checked via test cases; for math, answers can be parsed and matched to ground truth; metrics include Pass@k (probability at least one of k sampled attempts succeeds), with temperature affecting the tradeoff between diversity and per-sample quality.
GRPO’s key practical appeal is dropping the value function by using group-relative advantages.
GRPO is introduced as a go-to RL algorithm for reasoning because it avoids training a separate value function: it estimates advantage by comparing each completion’s reward to the group’s reward statistics for the same prompt, rather than using a learned value model as in PPO.
WORDS WORTH SAVING
5 quotesSo the output of reasoning models are not only the answer, but also the reasoning itself.
— Afshin
we want to use RL to incentivize our model to think more if we start from scratch, meaning we don't have reasoning chains.
— Afshin
This is a bad incentive because what you're telling your model is, if you have a short bad sentence, it is worse than w- if you have a long bad sentence.
— Afshin
First, into showing how powerful can the RL stage be if you apply the verifiable rewards that Afshin, uh, mentioned.
— Shervin
So instead of fitting a probability distribution over the next token, you just fit the entire sequence.
— Shervin
High quality AI-generated summary created from speaker-labeled transcript.
