Stanford OnlineStanford CME295 Transformers & LLMs | Autumn 2025 | Lecture 7 - Agentic LLMs
Summaries, chapters, transcripts, and FAQs on this page are made by uListen, not YouTube.
At a glance
WHAT IT’S REALLY ABOUT
Practical RAG, tool calling, and agent loops for LLMs
- Afshine Amidi introduces RAG as a way to augment LLM prompts with only relevant external information to overcome knowledge cutoffs, context-length limits, degraded performance from irrelevant context, and per-token cost.
- He breaks RAG into retrieve–augment–generate, emphasizing knowledge-base construction (chunking, overlap, embeddings) and a two-stage retrieval pipeline of candidate retrieval followed by optional re-ranking.
- Candidate retrieval is explained through semantic similarity search with bi-encoders (e.g., SentenceBERT) and contrasted with keyword-focused heuristic search using BM25, with hybrid combinations suggested depending on use case.
- He covers practical retrieval improvements such as HyDE (hypothetical document embeddings), contextual retrieval for chunks, prompt caching for repeated prefixes, and evaluation via nDCG, MRR, precision@K, and recall@K.
- Shervine Amidi then introduces tool/function calling, scalable tool selection (routing), standardization via MCP, agentic workflows via ReAct loops, emerging agent-to-agent communication, and the safety risks and mitigations that come with action-taking systems.
IDEAS WORTH REMEMBERING
5 ideasRAG is presented as a practical workaround for knowledge cutoffs and prompt-length limits.
Afshine Amidi frames the core motivation as avoiding incorrect or outdated answers from a model that only knows what was in its training data, while also avoiding the context-length, performance, and cost issues that come from stuffing everything into a prompt.
RAG quality starts with knowledge-base design choices (chunking, overlap, embedding size).
He describes building a knowledge base by chunking documents (hundreds of tokens), embedding each chunk, and tuning hyperparameters like embedding size, chunk size, and chunk overlap to balance context preservation with representation quality and efficiency.
RAG retrieval is treated as a two-stage system: maximize recall first, then improve ranking.
For retrieval, he outlines a two-stage pipeline: a fast candidate retrieval stage that prioritizes recall (often via embedding similarity/approximate nearest neighbors), followed by an optional ranking/re-ranking stage that is more compute-intensive but aims to improve precision at the top.
Embedding retrieval and BM25 solve different problems; hybrids can cover both semantics and exact keywords.
Candidate retrieval is contrasted between semantic similarity (bi-encoders like SentenceBERT) and lexical/heuristic keyword matching (BM25), with the note that hybrid approaches can be useful when keyword inclusion matters.
HyDE, contextual chunking, and prompt caching are presented as practical RAG extensions for better retrieval and lower cost.
To address query/document mismatch and out-of-context chunks, he introduces HyDE (generate a hypothetical document from the query and embed it) and contextual retrieval (prepend chunk-specific context generated from the full document), alongside prompt caching to reduce repeated compute/cost.
WORDS WORTH SAVING
5 quotesSo today we're going to, uh, focus on practical techniques to let our LLM interact with the outside world, with, uh, other systems 'cause, um, un- up until now, uh, our LLM was purely on its own.
— Afshine Amidi
So for all these reasons, I hope I convinced you that we need a more clever approach where instead of putting all the new information all at once in the prompt, what we do is we only somehow find the relevant information and put that in the prompt.
— Afshine Amidi
So nowadays what people do is to look at the use cases that they have and think about whether having some heuristic as well in the relevance score is useful for their use case.
— Afshine Amidi
The takeaway here is I do not recommend you write it end-to-end, maybe just a draft, and you let some very powerful model with excellent knowledge of, uh, logic do it for you.
— Shervine Amidi
Generating code is cheap, but judging whether a code is correct and does the right thing, this is the hard part.
— Shervine Amidi
SPEAKERS
Afshine Amidi
hostCo-instructor for Stanford CME295 (Transformers & LLMs), teaching practical LLM/agent methods.
Shervine Amidi
hostCo-instructor for Stanford CME295 (Transformers & LLMs), focusing on tool-calling/agents and applied LLM techniques.
High quality AI-generated summary created from speaker-labeled transcript.
