CHAPTERS
- 0:00 – 0:30
Why “context engineering” beats prompt engineering and RAG (and what Jeff/Chroma do)
Jeff (founder of Chroma) frames the talk around building practical, reliable AI software rather than chasing buzzwords. He argues that “context engineering” is the most useful umbrella term for what builders are actually doing when they make LLM systems work.
- •Speaker and company context: Jeff founded Chroma (search/retrieval database)
- •Pushback on AI buzzwords and over-complicated “RAG variants” discourse
- •Reframing the discipline as context engineering, not prompt tricks
- •Suggestion that this implies a new role/tooling: “context engineers” and maybe a “context engine”
- 0:30 – 1:31
AI systems as software programs: inputs, tools, and a “magic box”
The talk models an AI system as a program: fixed instructions, selected tools and information, and changing user input. This framing emphasizes engineering rigor over mysticism—LLMs are software components with predictable failure modes and tradeoffs.
- •Instruction set + relevant info/tools + user input → model output
- •Treat the model as a programmable component rather than a “machine god”
- •Engineering mindset: design inputs and interfaces deliberately
- •Sets up why controlling context is the central lever
- 1:31 – 2:02
Definition of context engineering: deciding what enters the context window
Jeff gives a direct definition: context engineering is choosing what goes into the model’s context window for a given turn. Prompts, retrieved documents, and other artifacts are all just parts of that context selection problem.
- •Context engineering = deciding what’s in the context window
- •Includes prompts and optionally retrieval depending on the use case
- •Focus on explicit ownership of context quality
- •Motivation: clearer, more durable concept than narrower terms
- 2:02 – 2:32
The real objective: reliable, fast, cheap AI software (in that order)
The target isn’t flashy demos—it’s dependable systems. Jeff advocates the classic engineering progression: make it work reliably first, then optimize speed and cost.
- •Shared goal: build reliable software with new AI primitives
- •AI usefulness depends on supplying the right context
- •Reliability first, then performance, then cost optimization
- •Many teams are still in the “make it work” phase
- 2:32 – 3:03
Why long-context alone isn’t the answer (yet)
Despite headline-grabbing million+ token context announcements, Jeff argues that simply stuffing more tokens into the prompt is not a dependable solution today. Builders should focus on techniques that work now, under real constraints.
- •Long-context models are rapidly expanding (1M, 10M tokens, etc.)
- •Bigger context windows don’t automatically yield better real-task performance
- •Uncertainty: may improve next year—or may not solve the core issue
- •Engineering focus: prioritize what is effective today
- 3:03 – 4:03
Chroma’s technical report: performance degrades as context length grows
Jeff summarizes Chroma’s findings: even on simple tasks, model performance can drop sharply as token length increases. This challenges the assumption that longer prompts are uniformly beneficial.
- •Chroma report shows accuracy declines with longer inputs on simple tasks
- •Observed degradation can happen earlier than many expect
- •Token length vs performance curve drops “precipitously”
- •Motivates context curation rather than raw expansion
- 4:03 – 4:33
“Needle in a haystack” is a weak benchmark for long context
Needle-in-a-haystack tests can look great while masking real-world difficulty. Jeff explains that the task requires minimal reasoning and minimal attention across the full context, unlike most practical applications.
- •Needle tasks are solved across token dimensions, but are too easy
- •Model attends to a single salient snippet, not the whole context
- •Reasoning required is near zero (simple string/semantic match)
- •Long-context success on this benchmark doesn’t generalize
- 4:33 – 5:03
A better lens: attention required vs reasoning difficulty (and why real tasks are harder)
Jeff proposes plotting tasks by how much of the context the model must use and how much reasoning is needed. Many valuable tasks—agents, summarization, complex QA—sit in harder regions than needle benchmarks.
- •Axes: fraction of context needing attention vs reasoning difficulty
- •Needle-in-haystack sits at low attention + low reasoning
- •Real workloads often require broader context, deeper reasoning, or both
- •Raises the question: how much context can models use effectively?
- 5:03 – 5:34
Focused (curated) context beats full context on evaluations
Experiments show that giving models the right subset of information can dramatically improve outcomes compared to dumping everything in. The practical takeaway: curate context aggressively.
- •Long-context evals: full context underperforms focused (oracle/curated) context
- •Large gains come from selecting and shaping inputs
- •Curation is a core competency, not a minor optimization
- •Leads into a concrete process for context engineering
- 5:34 – 6:04
Core goals of context engineering: find, remove, optimize
Jeff boils context engineering down to three steps: identify relevant info, strip irrelevant info, and optimize what remains. Each model turn becomes a selection problem: from “all information,” choose what belongs now.
- •Find relevant information for the current turn
- •Remove irrelevant/distracting information
- •Optimize/format the remaining information for best model use
- •Framing: per-turn decision about what enters the window
- 6:04 – 6:34
Gather & Glean model: maximize recall first, then precision
The proposed workflow separates context building into two stages. First “gather” broadly to avoid missing key signals; then “glean” to narrow down to a clean, high-precision context set.
- •Stage 1 (Gather): maximize recall; accept some noise
- •Stage 2 (Glean): maximize precision; remove irrelevant items
- •Analogy to ML/IR pipelines: recall-then-precision architecture
- •Fits modern multi-tool, multi-query retrieval patterns
- 6:34 – 7:35
Practical gathering pipelines: query planning + multi-source probes
Developers increasingly use an LLM to plan searches, issue multiple probes, and pull from diverse sources. The result is a large candidate pool from structured data, unstructured corpora, tools, and conversation history.
- •LLM-generated query plans and multiple “search probes” per user query
- •Pull from structured stores (SQL/APIs) and unstructured stores (vector DBs)
- •Include local tools, web search, MCP-style tools, and chat history
- •Creates a big pool that must be “gleaned” down effectively
- 7:35 – 8:06
Gleaning methods: from top‑K to re-rankers to “LLM as search brute force”
Jeff surveys common techniques for narrowing candidates: vector top‑K, rank fusion, learning-to-rank, dedicated rerankers, and increasingly LLM-based selection. A pragmatic trend is to use many small/cheap models in parallel—spending tokens instead of building overly complex retrieval heuristics.
- •Common pipeline steps: top‑K similarity, RRF, LTR, reranking models
- •Increasing use of LLMs directly for filtering/reranking (“cheating at search”)
- •Tradeoff: spend more tokens/intelligence rather than complex heuristics
- •Parallelism + smaller models can make brute-force curation practical
- 8:06 – 11:16
Context engineering for agents: loops, histories, and compaction as leverage
Agentic systems repeat gather/glean over many steps and sub-agents, making context growth and history management central challenges. Jeff shares findings on using past failures vs successes and emphasizes compaction—distilling long traces into usable next-turn context—while noting current summarization/compaction approaches often underperform.
- •Agents run iterative loops; gather/glean happens repeatedly across steps
- •Agent conversation/log history becomes a major context component (can be huge)
- •Past failures can improve performance; past successes can encourage lazy pattern matching
- •Compaction is critical but current defaults can be nearly as bad as no summary; better prompts help
