Stanford OnlineStanford CME295 Transformers & LLMs | Autumn 2025 | Lecture 7 - Agentic LLMs
Summaries, chapters, transcripts, and FAQs on this page are made by uListen, not YouTube.
CHAPTERS
Jump to publisher timestampsChapter summaries
Lecture roadmap and recap: reasoning models, GRPO, and length bias
Afshine Amidi opens the lecture by positioning it as a shift from standalone LLMs to systems that interact with external information and services. He briefly recaps the prior lecture on reasoning models, focusing on GRPO training and the length-bias issue that can cause outputs to grow longer even when performance plateaus.
- •The lecture’s goal is to connect LLMs to outside systems via RAG, tool calling, and agents
- •Recap of reasoning models vs. “vanilla” LLMs: producing a hidden reasoning chain before the final answer
- •GRPO described as a core RL method that uses group-relative advantages and does not train a value function
- •Observed length bias during RL training: output length increases even when accuracy plateaus
- •Mitigation strategies mentioned: DAPO’s normalization approach and removing a normalization term (“GRPO Done Right”)
Why LLM knowledge cutoffs create a real product limitation
Afshine explains why an LLM trained up to a cutoff date cannot reliably answer about newer events and why simply continuing training is often undesirable. He emphasizes the operational and regression risks of frequently updating weights across many fine-tuned use cases.
- •LLMs answer based on pretraining data and fail on events after the cutoff date
- •Model cards explicitly list knowledge cutoff dates as a limitation
- •Continuing training to inject new knowledge can cause regressions and is tricky to do safely
- •Operational overhead: updating weights becomes hard across multiple downstream fine-tuned variants
- •Motivation for methods that avoid frequent weight updates while still accessing fresh information
Why “just stuff everything into the prompt” fails: context limits, confusion, and cost
Afshine argues that naively adding all new information into the context is constrained by context windows and can degrade performance even if the window were large. He also highlights cost incentives to keep prompts short.
- •Context windows are large but finite; “hundreds of thousands of tokens” still isn’t enough for dumping everything
- •Needle-in-a-haystack behavior: retrieval quality depends on prompt length and where the key fact appears
- •Irrelevant context can confuse models and reduce accuracy
- •Token-based pricing increases costs with larger inputs
- •A better strategy is to add only relevant information to the prompt
RAG core loop: Retrieve → Augment → Generate
Afshine introduces retrieval-augmented generation (RAG) as a way to fetch only relevant information and insert it into the prompt. He breaks RAG into the three named stages and stresses that retrieval quality dominates overall system quality.
- •RAG augments the prompt with relevant retrieved information instead of everything
- •Three steps: retrieve relevant documents, augment the prompt, then generate the answer
- •Retrieval errors directly harm the final answer, making retrieval the key focus
- •RAG is motivated by knowledge freshness, context limits, and cost constraints
- •The method’s name directly maps to its pipeline stages
Building a knowledge base: chunking, embeddings, and key hyperparameters
Afshine explains how documents are collected into a knowledge base and split into chunks, each embedded for retrieval. He discusses practical hyperparameters such as embedding dimensionality, chunk size, and overlap, plus caveats about chunking structure.
- •Documents are split into chunks (often hundreds of tokens) to enable retrieval over smaller units
- •Each chunk gets an embedding; chunking choices affect meaning and retrievability
- •Hyperparameters: embedding dimension (trade-off between nuance and compute/storage)
- •Hyperparameters: chunk size (too small loses context; too large blurs representation)
- •Hyperparameters: chunk overlap to preserve continuity between adjacent chunks; structure-aware chunking can matter for formats like JSON/markdown
Two-stage retrieval architecture: candidate retrieval then (optional) re-ranking
Afshine frames retrieval as a two-stage pipeline borrowed from search and recommender systems: a fast high-recall candidate step followed by a more expensive ranking step. He explains why maximizing recall early and precision later is computationally practical.
- •Stage 1: candidate retrieval filters massive corpora down to a manageable candidate set with high recall
- •Stage 2: ranking/re-ranking refines ordering to surface the best few items (often optional depending on needs)
- •The second stage can be more compute-intensive because it runs on far fewer candidates
- •This mirrors common patterns in search and recommendation systems
- •Quality depends on both stages, but stage 1 strongly shapes what is even possible downstream
Candidate retrieval via embeddings: similarity search, ANN, bi-encoders, and SBERT
Afshine details semantic similarity search over embeddings as a standard candidate-retrieval method. He introduces approximate nearest neighbor (ANN) indexing for scale, the bi-encoder architecture, and points to SentenceBERT as a practical embedding approach.
- •Compute an embedding for the query and compare to precomputed chunk embeddings
- •Cosine similarity is the typical comparison; other distances appear but often relate closely
- •ANN methods help avoid naive linear search in large knowledge bases
- •Bi-encoder setup: query and document are encoded independently, then compared
- •SentenceBERT is highlighted as an extension aimed at sentence/document embeddings suitable for similarity search
Heuristic and hybrid retrieval: BM25 keyword matching alongside semantic search
Afshine explains why semantic similarity alone may miss exact keyword constraints and introduces BM25 as a heuristic relevance score based on term overlap. He notes that many systems use hybrid retrieval combining embedding similarity with BM25 depending on the use case.
- •Semantic similarity can retrieve meaning-alike text even without shared keywords
- •BM25 provides relevance based on keyword overlap between query and document
- •BM25 is useful when exact terms in the prompt must appear in retrieved text
- •A hybrid approach can combine embedding-based retrieval and BM25-style retrieval
- •Method choice should be driven by the requirements of the specific application
Bridging query–document mismatch: HyDE and contextual retrieval, plus prompt caching economics
Afshine discusses a mismatch between short question-like queries and longer document chunks, introducing HyDE as a method that generates a hypothetical document for embedding. He then addresses chunk coherence by adding LLM-generated context per chunk and explains prompt caching as a way to reduce cost when many prompts share a prefix.
- •Queries and documents can be different “types” of text, making naive embedding comparison less aligned
- •HyDE: generate a synthetic document from the query, embed it, then retrieve against chunk embeddings
- •Alternative mentioned: separate encoders for queries vs. documents (not always used due to maintenance)
- •Contextual retrieval: prepend chunk-specific context derived from the full document to improve standalone interpretability
- •Prompt caching: reuse shared-prefix activations to lower repeated compute; providers often discount cached input tokens
Re-ranking with cross-encoders: joint scoring of query + chunk
Afshine introduces re-ranking as a second-stage model that jointly encodes the query and candidate chunk to produce a relevance score. He contrasts cross-encoders with bi-encoders, emphasizing cross-attention interactions and higher compute per pair.
- •Re-ranking refines the candidate list to improve top-K quality
- •Cross-encoder: feed query and chunk together into a single encoder to score relevance
- •Joint encoding allows interaction (attention) between query and document text
- •This is more expensive but feasible because it runs only on a reduced candidate set
- •Pretrained and custom rankers exist; training details vary and can be tailored to the task
Evaluating retrieval: nDCG, MRR, precision@K, recall@K, and benchmarking
Afshine explains how to evaluate ranking quality using metrics that account for position and relevance labels. He walks through the intuition of nDCG and MRR and relates precision/recall to the top-K retrieval setting, then points to a popular embedding benchmark for comparison.
- •Evaluation compares the ranked top-K results to ground-truth relevance labels
- •nDCG rewards placing relevant items earlier in the ranking and normalizes by the ideal ranking for that query
- •MRR focuses on the rank of the first relevant result and ignores later relevant items
- •Precision@K and Recall@K adapt standard classification metrics to top-K retrieval
- •A named benchmark is referenced for evaluating retrievers and comparing methods
Tool (function) calling basics: structured data access and the teddy-bear example pipeline
Shervine Amidi transitions to tool calling, defining it as enabling autonomous systems to complete tasks by accessing external resources. Using a “find a teddy bear near me” example, he describes the three-step flow: present APIs in the prompt, execute the function call outside the LLM, then feed structured results back for a natural-language answer.
- •Tool calling/function calling reframes structured input-output relationships as callable functions
- •Tools complement RAG: they inject structured, real-time information via external APIs
- •The model sees function signatures and documentation (not the backend implementation)
- •Three-stage workflow: tool selection/argument generation → function execution → natural-language response generation
- •Tool outputs are structured (e.g., JSON-like objects) and are then summarized into user-friendly text
Training and prompting for tool use: SFT pairs, prompt iteration, and reasoning-based prompt writing
Shervine explains how models can be trained or prompted to map user queries into correct tool calls and to turn tool results into the desired final response. He describes using two kinds of supervised examples and also an alternative approach: iterating on a fixed explanatory prompt using evaluation feedback and a strong reasoning model.
- •Two supervised mappings are often needed: query → tool call (arguments) and tool result + history → final response
- •Tool-use training examples should vary: missing vs. explicit location, different phrasings, multi-turn contexts
- •Some models may require less dedicated SFT due to strong code/tool priors, motivating prompt-based approaches
- •A workflow is described for improving a tool-use explanation prompt using an evaluation set and a reasoning model to propose edits
- •The offline goal is a fixed explanation prompt that generalizes at inference time when combined with the tool APIs
Scaling to many tools: tool selection/routing and standardizing tool interfaces with MCP
Shervine highlights the “too many tools” problem: limited context windows and confusion when many APIs are provided. He introduces a tool selector/router approach to first choose relevant tools from a large catalog, then include only those tool specs; he also outlines MCP as a protocol to standardize how tools are exposed to models.
- •Too many tools can overwhelm context and reduce reliability, echoing “needle in a haystack” issues
- •Tool selection/router approach: choose relevant tools from lightweight descriptions, then include only selected full APIs
- •Tool selection can be implemented with an LLM and can also be connected to retrieval ideas
- •MCP (Model Context Protocol) is presented as a standard for exposing tools across models/providers
- •MCP vocabulary described: servers serving tools, prompts as templates, resources as external data, and clients connecting hosts to servers
Agents with ReAct loops: observe–plan–act, multi-agent coordination, and safety risks
Shervine defines agents as systems that autonomously pursue goals through iterative reasoning and tool use, then presents ReAct-style loops through a thermostat example. He briefly discusses agent-to-agent protocol ideas for coordinating multiple agents and closes with safety concerns, including data exfiltration and the need for training-time and inference-time safeguards, plus practical caveats about reliability.
- •Agents differ from simple tool calls by adding iterative loops and goal pursuit across multiple steps
- •ReAct-style decomposition: observe the situation, plan next steps, act via tools, then observe again until completion
- •Example loop: infer temperature relevance, call a temperature tool, decide it’s cold, call a thermostat-adjust tool, then stop when goal is met
- •Multi-agent setups motivate standardized communication; an agent-to-agent protocol is referenced with skills, status updates, and cancellation concepts
- •Safety concerns discussed: data exfiltration via tools; mitigations include safety data during training and inference-time safety classifiers; agent reliability remains a practical limiting factor
Publisher timestamps
- 0:00Introduction
- 6:38RAG overview
- 27:39Similarity search with SBERT and bi-encodersRead this chapter from 27:40
- 34:25Heuristic search with BM25
- 37:54HyDE and contextual retrievalRead this chapter
- 41:00Prompt caching
- 45:24Re-ranking with cross-encoders
- 47:49Retrieval evaluation with NDCG, MRRRead this chapter
- 59:28Tool calling
- 1:26:22Tool selection
- 1:29:17Model Context Protocol (MCP)
- 1:31:56Agents with ReActRead this chapter
- 1:42:16Safety and closing thoughtsRead this chapter
