Stanford OnlineStanford CS329A Self-Improving AI Agents | Part 1 | Course Overview
Summaries, chapters, transcripts, and FAQs on this page are made by uListen, not YouTube.
CHAPTERS
Chapter summaries
Welcome to CS329A and instructor introductions
Dr. Aakanksha Chowdhery opens the course, checks the audience mix (master’s/PhD/undergrad), and introduces herself. Azalia Mirhoseini introduces herself as co-instructor and notes this is the second offering of the course with updates to lectures and course operations.
- •Course focus: self-improving AI agents
- •Quick show-of-hands for student backgrounds (master’s, PhD, undergrad)
- •Dr. Aakanksha Chowdhery shares her background and affiliation with Reflection AI
- •Azalia Mirhoseini shares her background and excitement to teach the updated course
- •Instructors note strong interest in enrolling
Course website, lecture plan, and why scaling matters
Dr. Aakanksha Chowdhery points students to the course website and outlines the lecture structure: LLM trends first, course focus next, and logistics second. She then frames scaling laws as a key driver behind recent progress in large language models.
- •Course website includes materials, reading list, homework schedule, and project structure
- •Lecture will cover LLM scaling trends, then course content focus, then grading-relevant logistics
- •Scaling is introduced as the central story behind rapid improvements since GPT-3
- •Performance improvements are discussed in terms of lower test loss with more resources
- •Sets up the transition from LLM basics to self-improving agent themes
Scaling axes: compute, data, parameters—and signs of saturation
Dr. Aakanksha Chowdhery explains that scaling compute, dataset size, and parameter count reduces test loss and improves model quality. She also notes the rapid increase in model sizes from 2018 through 2024 and mentions that scaling began to show signs of saturation more recently.
- •More compute correlates with lower test loss and better models
- •Bigger datasets also reduce test loss and improve performance
- •Increasing parameter count (layers/parameters) improves loss and model capability
- •Model sizes grew dramatically from hundreds of millions to hundreds of billions, with GPT-4 estimated much larger
- •Mentions scaling appeared to start hitting a saturation point more recently
Capabilities from scale: benchmark gains, few-shot/zero-shot, and emergence
Dr. Aakanksha Chowdhery connects larger models to improvements on language and reasoning benchmarks. She explains zero-shot and few-shot prompting and introduces the idea of emergent behaviors that become visible only at larger scales.
- •Bigger models improve across NLP and reasoning benchmarks
- •Few-shot learning reduces need for task-specific fine-tuning by learning from a handful of examples
- •Zero-shot learning is framed as performing a task from instruction alone
- •Prompt examples are used to illustrate few-shot vs. zero-shot translation
- •Emergent behaviors (like reasoning) are described as appearing in larger models
Chain-of-thought as an emergent reasoning behavior
Azalia Mirhoseini introduces chain-of-thought prompting and contrasts short answers with showing intermediate reasoning. She emphasizes that chain-of-thought remains a key property for reasoning and ‘thinking’ models and becomes more useful as models scale.
- •Chain-of-thought: providing reasoning steps can improve performance on problems
- •Example shows walking through a simple math word problem rather than giving only the final answer
- •Smaller models may not benefit much, while larger models can leverage reasoning traces
- •Results are described across models (Lambda, GPT, PaLM) with benefits increasing with scale
- •Other abilities are mentioned as appearing at certain sizes (e.g., better performance on new tasks)
From ChatGPT to alignment: instruction tuning and RLHF overview
Azalia Mirhoseini describes ChatGPT’s impact and highlights that post-pretraining steps were key to making it more usable. She outlines the pipeline from pretraining to higher-quality fine-tuning, instruction tuning, and reinforcement learning from human feedback (RLHF) aimed at aligning model behavior with human preferences.
- •ChatGPT’s launch and rapid user growth are mentioned as a milestone
- •Pretraining is described as next-token prediction over broad data
- •Fine-tuning on higher-quality data improves model behavior and output quality
- •Instruction tuning uses instruction–question–answer pairs (including chain-of-thought-style data)
- •RLHF uses human preference data to train a reward model and optimize generations for desired qualities
Inference-time scaling: repeated sampling and the “Large Language Monkeys” idea
Azalia Mirhoseini shifts to inference as a newer frontier, describing repeated sampling with a verifier to select correct outputs. She explains why generating many candidate solutions can reveal capability that a single sample may miss, and shows how parallel sampling can outperform stronger single-sample baselines when evaluated by coverage.
- •Inference scaling: improve results without changing model parameters by changing how you sample/aggregate outputs
- •Repeated sampling + verifier: generate many answers and select the correct one (e.g., unit tests for code)
- •Temperature and nondeterminism are discussed as sources of output diversity
- •Coverage improves as the number of samples increases; small models can beat a stronger model with enough samples
- •Notes that for hard problems, only a few correct samples may appear among many generations
Q&A on verifiers, latency/cost, and temperature limits
Students ask practical questions about how repeated sampling differs from traversing all possible answers, latency trade-offs, what to do without verifiable domains, and how temperature affects usefulness. Azalia Mirhoseini answers by emphasizing the enormous output space, parallelism to reduce latency, research on LLM-based judging when verifiers are absent, and the downside of excessively high temperature.
- •Generating ‘all possible answers’ is infeasible; sampling is a practical way to explore a huge space
- •Parallel sampling can reduce latency impact, but cost/compute trade-offs remain
- •Without verifiers, evaluation becomes harder; the course will cover judge/reward-model approaches
- •Very high temperature can degrade outputs into gibberish; tuning diversity is an active area
- •Course will include content dedicated to verifiers and evaluation strategies
Reasoning models and test-time compute scaling (and how they ‘think’)
Azalia Mirhoseini describes how newer ‘thinking’ or reasoning models combine training improvements with test-time scaling, including generating synthetic data. She explains test-time reasoning steps such as problem analysis, task decomposition, self-correction, and backtracking, and discusses how increased test-time compute can improve pass@1 performance.
- •Models and approaches discussed as examples of combining fine-tuning and test-time scaling
- •Synthetic data generation is described as enabled by test-time scaling and used for fine-tuning
- •Test-time compute can improve pass@1 accuracy without changing parameter count
- •Reasoning behaviors include analysis, decomposition, self-evaluation, self-correction, and alternative proposals
- •Reasoning models are described as stronger in math/data analysis/programming than in personal writing/editing
Discussion: where reasoning gains come from and using multiple models
Students ask whether improvements come from producing explicit reasoning traces or from decomposition more generally, and whether different models can be used for reasoning versus final answers. The instructors discuss how training and feedback can increase pass@1 by learning which reasoning chains are correct, and mention that models often prefer their own traces and that cross-model evaluation/feedback is also a topic in the course.
- •Question raised about whether ‘thinking out loud’ is necessary versus silent decomposition
- •Instructors distinguish coverage gains from repeated sampling versus pass@1 gains from training/feedback
- •Using a larger model for reasoning traces and a smaller model to summarize is discussed as a possible pattern
- •Comment that models tend to prefer their own generated traces
- •Instructors note upcoming coverage of outcome vs. process rewards and multi-step reasoning topics
From LLM chatbots to agents: goals, actions, feedback, tools, and memory
Dr. Aakanksha Chowdhery explains why agents are the next step beyond single-turn chat: agents pursue goals via planning, tool use, environment interaction, feedback, and deciding when to stop. She uses examples like research and coding workflows to illustrate how agentic systems can execute end-to-end tasks rather than only respond conversationally.
- •Agents are framed as moving beyond chat to accomplishing end-to-end tasks
- •Core ingredients: goal specification, planning steps, taking actions, feedback-driven correction, and stopping criteria
- •Examples discussed include research-style workflows and coding agents that operate through instructions
- •Tool interaction is highlighted (e.g., web search, external information retrieval)
- •Memory and tracking are presented as necessary to stay aligned with the task over multiple steps
Agentic workflow patterns: orchestration graphs, routing, parallelism, judges, and verifiers
Dr. Aakanksha Chowdhery describes common agentic workflow components and why many real systems use structured orchestration rather than fully open-ended loops. She outlines building blocks such as chaining, routing by complexity, parallelization, an orchestrator/planner, LLM-as-judge, and verifiers (including unit tests) to provide reliable feedback.
- •Many deployments use hand-constructed workflow graphs with generation plus evaluation loops
- •Workflow building blocks: LLM calls, critics/judges, tool calls, and verifiers
- •Prompt chaining decomposes tasks into subtasks across multiple calls
- •Routing selects more complex workflows only when needed; parallelization supports research-style aggregation
- •Verifiable domains (math/code/rule-based tasks) make feedback loops more effective
Where agents are used: coding, customer support, deep research, and ‘AI scientist’ workflows
Dr. Aakanksha Chowdhery gives concrete areas where agentic systems are already useful: coding tasks through terminal/tool loops, customer support assistance, and generating comprehensive research reports. She also describes research-agent roles across idea generation, experiment iteration, and paper write-up, while noting limitations like hallucinations alongside brainstorming value.
- •Coding agents: repository navigation, file viewing/editing, running commands, iterating based on terminal outputs
- •Reliability improvements attributed to stronger models and better reinforcement learning with verifiable rewards
- •Customer support applications: transcription, knowledge assist, smart reply, and call summaries
- •Research/report generation workflow: selecting references, outlining, summarizing sources, synthesizing a full article
- •Research-agent framing: idea generation, experiment iteration support, and assistance in paper write-up
Course logistics: prerequisites, assignments, projects, grading, and policies
The instructors transition to logistics, summarizing prerequisites, where to find updates, and what work is required. They explain homework and project structure, project expectations (research-oriented outcomes), milestones, grading breakdown, and operational details like office hours, discussion forums, submission tools, and late policy.
- •Canvas is the source of latest updates; lectures are uploaded in advance when possible
- •Three homeworks and a course project; projects can be solo or teams up to four, with API credits mentioned
- •Project expectations emphasize a research question/hypothesis and evaluation, not just a demo app
- •Milestones include a proposal, a midterm presentation requiring progress, and a final report plus poster presentation
- •Grading: homeworks 50% and project 50%; EdStem for Q&A, Gradescope for submissions, and a fixed late policy with no exceptions due to class size
