By playing, you agree to uListen's Terms and Privacy Policy and YouTube's Terms of Service.

Jeff Dean: The 1% Rule for Building in AI

In 2001, Jeff Dean and Sanjay Ghemawat did the math and realized Google’s entire search index would fit in RAM — then shipped it in a few days, and search got fast.

Diana HuhostJeff Deanguest

Jul 30, 202657mWatch on YouTube

Summaries, chapters, transcripts, and FAQs on this page are made by uListen, not YouTube.

CHAPTERS

Jump to publisher timestamps

Chapter summaries

AI at “junior engineer” level: what improved faster than expected
Diana Hu asks Jeff Dean to revisit his public claim that AI had reached a junior-engineer level. Jeff says agent-based, longer-running coding has improved significantly, and that complexity has grown faster than he expected, with agents shining beyond coding as well.
- •Jeff says models have become more capable at longer-running, agent-based coding tasks
- •He says the speed of progress on complex tasks outpaced his expectations
- •He notes agent-based systems are increasingly strong in domains beyond coding
- •Diana asks what he underestimated in the original prediction
A 2027 prediction: automated self-improvement loops for ML and science
Asked for a new bold prediction, Jeff forecasts more automation of ML system improvement. He describes systems that decompose problems, run many experiments automatically, and recombine results, and says this approach extends to other scientific and engineering fields with measurable objectives.
- •Jeff predicts more automation of ML systems improving themselves
- •He describes problem decomposition into subproblems plus tight experimentation loops
- •He expects fully automated experimentation to yield improved systems
- •He says similar loops can accelerate progress in science and engineering when objectives are measurable
The next “fits-in-memory” moment: inference hardware, latency, and energy efficiency
Diana references an early Google Search shift from disk to RAM and asks what today’s equivalent is. Jeff points to high-performance, low-energy inference hardware as key to making agent-based systems broadly available, emphasizing latency and the value of specialized hardware for inference.
- •Jeff highlights inference as central to scaling agent-based systems to more people
- •He emphasizes latency as a key constraint on what’s possible
- •He argues specialized inference hardware can be more energy efficient and lower latency than general-purpose devices
- •He asks listeners to imagine dramatically better latency (e.g., much faster responses)
A false assumption: agents can run for days or weeks
Jeff challenges the audience’s intuition about how long agents can operate effectively. He says with capable underlying models and the right domains, agents can run for days or weeks doing complicated work, such as implementing major software rewrites in different languages for safety or performance.
- •Jeff says many people haven’t internalized that agents can run for days or weeks
- •He frames long duration as enabling much more complicated tasks
- •He gives an example: agents implementing new software versions in different programming languages
- •Diana probes for a concrete example of a multi-week agent task
Napkin math to TPU: predicting demand and building specialized chips
Diana recounts the TPU origin story: Jeff’s back-of-the-envelope calculation showed speech usage could require doubling Google’s servers. Jeff explains deep-learning speech models were far more computationally expensive but dramatically improved error rates, motivating specialized hardware optimized for low-precision dense linear algebra and delivering major energy-efficiency and latency gains.
- •Jeff explains why improved speech quality would drive much higher usage and compute demand
- •He contrasts expensive deep-learning speech models with large quality gains (reduced error)
- •He describes TPUs as specialized for low-precision dense linear algebra at the core of modern ML
- •He cites big improvements in energy efficiency and latency from specialized chips
- •He explains TPUs were designed as general-purpose linear algebra hardware to remain useful as ML evolves
Finding breakthrough ideas: bottlenecks, first-principles thinking, and system “numbers”
Jeff responds to what founders should do their own napkin math on: look for bottlenecks and rethink solutions from first principles to unlock order-of-magnitude improvements. Diana also brings up Jeff’s well-known latency numbers list, and Jeff proposes an AI-systems version focused on bandwidth, energy per operation, and scaling interconnects across many chips.
- •Jeff suggests founders identify bottlenecks and seek 10x–100x shifts in capability or performance
- •He encourages not being anchored to today’s solutions and instead reasoning from first principles
- •He suggests key AI-system metrics: memory bandwidth hierarchy and energy per operation
- •He highlights interconnect bandwidth and scaling behavior when coordinating many chips
- •He says these “numbers to know” influence how you design AI systems
AI as an energy and data-movement problem: batching, latency, and inference-focused hardware
Diana asks about Jeff’s view that energy becomes the core unit of measurement, highlighting that moving data can cost far more than computing. Jeff explains how this shapes batching and system design, why batching conflicts with low-latency goals, and why he’s focusing more on inference—minimizing data movement and using low precision with targeted hardware support.
- •Jeff explains the large energy gap between moving data and computing on it
- •He describes batching as a way to amortize data movement costs
- •He notes batching is problematic for very low-latency requirements
- •He says he’s been thinking more about inference than training due to latency demands
- •He highlights strategies: minimizing data movement and using very low-precision operations
Context engineering and tool orchestration: making systems, not just models, effective
Diana frames recent progress as driven by system components around the model—retrieval, tools, memory, and orchestration—often called context engineering. Jeff agrees the model is only one piece and describes how clear context, tool use, problem decomposition, multi-step plans, and evaluation across approaches form the heart of capable agent and multi-agent systems.
- •Jeff says solving real problems requires an overall system, not only a stronger model
- •He contrasts “soup” of training data with explicit, clear context provided at run time
- •He emphasizes models that can retrieve information and use tools effectively
- •He describes decomposing problems into sequences of tool calls and trying multiple approaches
- •He points to orchestration of complex agent and multi-agent systems as increasingly important
How to get better at context engineering: skills, harnesses, and a performance-optimization example
Jeff suggests improving context engineering by using models to solve real tasks, diagnosing failures, and adding better guidelines and tool-usage “skills.” He shares an example where he and Sanjay created a skill to automate microbenchmark-driven performance optimization loops, and notes they published a document on performance hints that others have used to improve model reasoning about performance.
- •Jeff recommends hands-on iteration: use models, observe failures, and refine the setup
- •He suggests adding tool-usage skills and guidelines to increase success without changing model weights
- •He describes a workflow skill for microbenchmarks: measure, modify, rerun, and iterate
- •He notes these skills mirror the approach humans would take and encode it for the agent
- •He mentions a published performance-hints document that people summarize for models to improve performance reasoning
Why long-running agents fail: out-of-distribution behavior, skills, multi-agent search, and evaluation
Diana asks why agents go off the rails after many steps. Jeff explains degradation often happens when agents drift off the distribution of what the model has experience with, and he outlines mitigations: providing skills and hints to keep agents on “brighter paths,” using multiple agents to try approaches, and applying inference-time compute for search and evaluation of candidate solution paths.
- •Jeff attributes failures to agents doing tasks the model lacks experience with
- •He describes performance degrading as the agent moves farther from familiar distributions
- •He recommends skills and hints to keep the agent on more reliable paths
- •He suggests multi-agent approaches with evaluation to keep promising trajectories and discard failures
- •He frames inference-time compute as search over solution paths to improve reliability
Where startups can win vs. general models: domain focus, private data advantages, and niche models
Diana asks where small teams can outperform a company building broad, general models. Jeff says startups can win by creating delightful, high-quality domain-specific products, but must consider how quickly general models will catch up; he suggests looking for areas where general models fail nearly completely, or where product advantage comes from access to user-specific data, or from training specialized niche models for hard domains.
- •Jeff says general models aim to do almost anything, which can leave domain gaps in product focus
- •He argues small teams can build high-quality, domain-specific experiences with specialized skills or models
- •He cautions that general models are improving quickly; founders should assess durability over 6–12 months
- •He suggests targeting problems where general models succeed 0–1% of the time, not 20%
- •He highlights advantages from access to user-specific/personal data and from affordable specialized niche model training
Becoming an AI-native founder: crisp specs, and “taste” as the scarce skill
Jeff says managing many agents depends on clearly specifying what you want—crisp design docs and specs reduce ambiguity and improve outcomes. He gives translation between programming languages as a case where the spec is explicit (the existing code and tests), and argues that as code writing becomes easier, the scarce skill becomes choosing what to work on—having taste in what to ask agents to do.
- •Jeff says clear specifications are increasingly important when delegating to agents
- •He notes vague requests force agents to infer intent and can produce mismatches
- •He gives code translation as an example of success due to an extremely detailed specification
- •He argues the scarce skill becomes choosing the right problems and directing AI-assisted work
- •He says models may not be very good at high-level problem selection, so humans must steer
Questioning assumptions and building taste: prediction review, thought experiments, and examples like MapReduce
Jeff explains taste is hard to measure and often comes from experience, including writing down predictions and reviewing them later to calibrate judgment. He also advocates occasional “crazy” thought experiments that challenge long-held assumptions, illustrating with an example of designing systems from unreliable transistors; he then cites TPUs and MapReduce as past cases where rethinking assumptions led to consequential systems.
- •Jeff says experience across problems helps develop intuition for what might be important next
- •He suggests writing down ideas and revisiting them later to evaluate what mattered
- •He recommends thought experiments that challenge standard assumptions to explore new design points
- •He describes a thought experiment: building transistor-level systems that tolerate frequent errors via redundancy
- •He describes the MapReduce insight: separating simple computation from parallelization and reliability via an abstraction
AI that builds better AI: automating the scientific method and speeding up evaluation loops
Diana asks about systems that compound improvements, and Jeff describes automating the propose–implement–evaluate loop across domains like ML, chip design, science, and engineering. He emphasizes that faster evaluators can transform what’s feasible, giving an example where a learned approximation model made a previously expensive simulator dramatically faster and changed the pace of screening and experimentation.
- •Jeff describes automating experimentation loops with orchestration that decomposes objectives into subproblems
- •He argues low-latency iteration enables running many experiments and combining results into better solutions
- •He says this approach can accelerate ML progress, science, engineering, and chip design
- •He emphasizes that evaluator speed is often a key bottleneck
- •He gives an example of training a neural approximation to a computationally intensive simulator to enable much faster screening while staying nearly as accurate
Rejections, career choices, and building what matters: impact, teams, and open problem areas
Jeff reflects on a rejected paper that later proved useful, emphasizing persistence. He then answers what he’d do as a young engineer today—choosing work he cares about that could make a positive difference—and discusses how to find and work with strong teammates; he closes by listing areas he finds exciting, including new hardware approaches, more data-efficient learning, continual learning, multi-agent interaction, and tools for better discourse among people.
- •Jeff recounts a paper rejection and says the lesson is to keep going despite setbacks
- •He frames career choices around caring about the problem and creating positive world impact
- •He contrasts large organizations (structure, colleagues, platform for impact) with startups (risk, rewards)
- •He describes valuing teammates who are low ego, collaborative, and bring complementary skills
- •He lists motivating problem areas: hardware innovation, more data-efficient ML, continual learning, multi-agent interactions, and improving discourse and connections among people

Publisher timestamps

Get more out of YouTube videos.

Read along on your phone, search it beside the video, or ask your favorite AI app what was said.