Dwarkesh PodcastJeff Dean & Noam Shazeer — 25 years at Google: from PageRank to AGI
CHAPTERS
- 0:00 – 3:29
25 years at Google, and when no one person could “know everything” anymore
Dwarkesh opens by framing Jeff Dean and Noam Shazeer’s long tenures and outsized influence on Google’s infrastructure and modern AI. They joke about early Google when Jeff effectively mentored everyone, and describe the phases of organizational complexity as Google scaled.
- •Jeff/Noam introductions and their roles leading Gemini
- •Early Google mentorship culture; Noam’s mentor was Jeff
- •How understanding shifts from “everyone and everything” to high-level awareness
- •Importance of internal networks to find the right owner (“one level of indirection”)
- 3:29 – 6:19
Joining Google in the early days: job fairs, exponential query charts, and AI ambitions
Jeff and Noam recount how they ended up at Google around 1999–2000, when it was still tiny but growing explosively. They connect Google’s founding mission—organizing the world’s information—to an implicit need for advanced AI.
- •Jeff reached out; Noam applied “on a whim” after assuming Google was already huge
- •The famous wall chart of daily queries and early exponential growth
- •Noam’s early motivation: earn money to fund AI research—then realizes Google is great for AI
- •Google’s mission as an unusually broad mandate that naturally expands into AI
- 6:19 – 11:04
Moore’s Law slows, specialization rises: why ML hardware changed everything
The conversation shifts to system design under changing hardware trends: CPUs no longer deliver easy generational speedups, pushing the field toward accelerators. Noam highlights the hardware-algorithm co-evolution: arithmetic is cheap, data movement is expensive, and deep learning exploits that balance.
- •From effortless CPU speedups to slower process improvements and weaker general-purpose gains
- •Rise of TPUs and ML-optimized GPUs as specialized linear-algebra engines
- •Deep learning’s fit with hardware: lots of compute per byte moved (matrix multiply)
- •Opportunity cost mindset: fill chip area with low-precision arithmetic units
- 11:04 – 13:17
Future TPUs and quantization: FP4, INT4, and the co-design mindset
Jeff explains TPU design tradeoffs, especially the steady march toward lower precision for both inference and training. Both emphasize that quantization only works when chip designers and algorithm designers coordinate, because each side otherwise resists the risk and inconvenience.
- •TPU v1 bet: serving with INT8 when it wasn’t yet “obviously possible”
- •Progression to INT4/FP4 and even 2-bit/1-bit experiments
- •Quantization as a throughput-per-dollar lever, despite engineering irritation
- •Necessity of hardware/algorithm co-design to realize performance gains
- 13:17 – 15:58
Jeff’s 1990 thesis on parallel backprop—and the long wait for enough compute
Dwarkesh digs into Jeff’s early work on neural networks, where he implemented data and model parallelism on a 32-processor machine. Jeff reflects that the abstraction felt powerful, but real-world impact required roughly a million-fold more compute, which arrived decades later.
- •Early implementation of data parallelism and model/pipeline parallelism
- •Neural nets as the “right abstraction” even when they only solved toy problems
- •Compute shortfall in 1990 vs breakthroughs starting around 2008–2010
- •Re-entry into neural nets as hardware finally made scaling practical
- 15:58 – 25:08
LLMs in 2007: the 2-trillion-token n-gram model that made translation practical
Jeff recounts optimizing Google’s machine translation system after discovering it took 12 hours per sentence despite winning a DARPA-style contest. By building an in-memory compressed 5-gram store across ~200 machines with batched lookups, they cut latency to ~100 milliseconds—unlocking broader LM applications inside Google.
- •Translation system wins, but is unusable due to extreme latency
- •Design of compressed in-memory n-gram representation at massive scale
- •Batched API to parallelize ~100,000 lookups per decoding step
- •From translation to autocomplete/spelling correction and broader LM usage
- 25:08 – 27:28
“Holy shit” moments: scaling neural nets, the cat neuron, and the lesson of scale
Jeff describes early Google Brain efforts to train large neural nets on CPU clusters before GPUs were common in data centers. The “cat neuron” result from unsupervised learning and large-scale training helped validate the strategy that bigger models plus infrastructure could unlock new capabilities.
- •Google Brain’s early focus: infrastructure for very large nets on CPUs
- •2,000 machines / 16,000 cores training on 10M YouTube frames
- •Emergent high-level features: a neuron responding strongly to cat faces
- •Scaling yields big supervised gains (ImageNet) and reinforces ‘keep pushing’
- 27:28 – 31:59
AI as Google’s mission fulfillment: from retrieval to synthesis and multimodal understanding
Dwarkesh challenges whether Google remains an “information retrieval company” if it builds AGI-like systems. Jeff and Noam argue the mission expands to organizing, transforming, and generating information across modalities—text, video, audio, and even specialized data like LiDAR or genomics.
- •Shift from retrieval to synthesis: answers require combining many sources
- •Multimodal capabilities broaden ‘information’ beyond text
- •Vision: universal accessibility across languages (any content in any language)
- •Value creation as models become “doers,” not just information routers
- 31:59 – 36:12
Search in-context: long context windows, factuality, and ‘the entire internet’ aspiration
They explore how transformer attention makes context “sharp,” while parametric memory is “squishy,” contributing to hallucinations. Jeff outlines the ambition to scale effective attention from millions to trillions of tokens—linking search, personal data, and massive codebases—requiring new approximations beyond quadratic attention.
- •Why hallucinations happen: facts mixed into parameters vs crisp in-context tokens
- •Current frontier: millions of tokens (hundreds of pages, many papers, hours of media)
- •Goal: attend over internet-scale and personal-data-scale corpora with permission
- •Compute challenge: quadratic attention forces approximate retrieval/attention methods
- 36:12 – 45:46
Internal coding models and the ‘millions of employees’ future of research productivity
Jeff confirms Google has already further-trained Gemini on internal code, enabling a meaningful fraction of production code to be AI-generated. They discuss the near-term possibility of models generating experiments, writing systems from specs, and requiring new workflows to manage many asynchronous AI tasks.
- •Gemini trained on Google’s internal codebase for internal developers
- •Reported metric: ~25% of checked-in characters generated by AI coding models
- •Examples of autonomous coding: implementing a SQL engine from a paragraph prompt
- •Need for new UIs/workflows to coordinate many background agentic tasks
- 45:46 – 53:06
2027 capabilities and rapid feedback loops: automated architecture search and chip design
Dwarkesh presses on the implication of algorithmic progress and automation: could breakthroughs arrive “every day”? Jeff and Noam argue many improvements are algorithmic (better per flop), and AI-assisted exploration could accelerate the iteration loop—though largest-scale experiments remain bottlenecked by time and hardware availability.
- •Model improvements often driven more by algorithms/data/architecture than raw scale
- •Automated experimentation: winnow ideas at small scale, scale winners up
- •Chip design acceleration (AlphaChip-style): compress 12–18 month cycles toward fab-time limits
- •Possibility of rapid capability acceleration as models improve both software and hardware design
- 53:06 – 1:02:38
Inference-time scaling: reasoning ‘dials,’ search, asynchronous tasks, and new data-center demands
Noam argues inference is underexploited: LLM interaction is often cheaper than human labor or even books, so there’s headroom to spend more compute for smarter answers. Jeff connects this to search and tool use, asynchronous “deep research” workflows, and specialized inference hardware and algorithms like drafting/verifying models.
- •Economic framing: massive headroom to spend more inference compute for better outputs
- •Inference algorithms: iterative exploration, tool use, and search-style approaches
- •UX shift: long-running background tasks that ask clarifying questions mid-flight
- •Efficiency ideas: drafter + verifier decoding to parallelize token generation
- •Implication: much larger inference compute demand and possible inference-specialized chips
- 1:02:38 – 1:12:45
Multi-datacenter training and debugging at scale: sync vs async, bandwidth vs latency
Jeff notes Google already trains across multiple metro areas, where step times make latency tolerable and bandwidth is the key. They revisit early async CPU training, its practical success but poor reproducibility, and discuss why modern TPU networking enables synchronous training—while future scaling may reintroduce controlled asynchrony.
- •Multi-metro training already in production (Gemini 1.5 report references)
- •Training tolerates higher latency; bandwidth and step-time dominate constraints
- •Async training worked on slow CPUs but harmed reproducibility/debuggability
- •TPU pods and strong networks enable large-scale fully synchronous runs
- •As scale grows, systems complexity makes it harder to know ‘what’s fighting you’
- 1:12:45 – 1:24:22
Fast takeoff, safety, and ‘a million evil Jeff Deans’: shaping deployment with safeguards
Dwarkesh raises fast-takeoff and misalignment concerns in a world of AI-driven feedback loops. Jeff advocates a middle stance: steer AI’s trajectory through engineering safeguards, policy, and evaluation, including using models to analyze and filter model outputs; they emphasize controlled access via APIs and interfaces rather than unconstrained self-improvement.
- •Acceleration plausibility: models getting substantially better generation to generation
- •Risks: misinformation, automated hacking, and scalable malicious capability
- •Strategy: ‘shaping AI’—steering deployment toward benefits and away from harms
- •Use models for self-critique and content analysis (recognition easier than generation)
- •Control lever: API/UI boundaries and monitoring to limit dangerous use cases
- 1:24:22 – 1:27:51
Fun times at Google and the ‘Gradient Canopy’ micro-kitchen culture
To lighten the mood, Jeff reflects on the early Google years of rapid growth and hands-on scaling search infrastructure, and compares it to today’s Gemini momentum. He describes how in-person collaboration hubs—like a micro-kitchen turned workspace—create high-bandwidth idea exchange alongside global chatrooms.
- •Nostalgia for early search/crawling/indexing days amid explosive growth
- •Today’s Gemini work as similarly energizing due to rapid capability gains
- •In-person collaboration: micro-kitchen workspace dynamics and serendipity
- •Distributed coordination: many Gemini chatrooms and globally shared results
- 1:27:51 – 2:03:10
2030 compute demand, modularity, Pathways ‘blob’ models, and what’s missing from distillation
They explore a future where inference compute dominates and assistants become ubiquitous, driving enormous demand. Jeff outlines a Pathways-inspired vision: more organic, modular MoE systems that match hardware topology, allow continual learning via module upgrades, and support distillation into efficient serving models—while acknowledging distillation and training objectives still need major advances.
- •Compute outlook: inference scaling and broader adoption imply orders-of-magnitude growth
- •Vision of modular/organic models: specialized regions, variable-cost experts, routing
- •Hardware-topology-aligned connectivity: dense local, sparse distant communication
- •Continual learning via modular upgrades and versioning; parallel development across teams
- •Serving challenges: keeping large MoE models in memory; paging rare experts to DRAM
- •Open problems: faster/better distillation and richer pretraining objectives for data efficiency
- 2:03:10 – 2:15:35
Open research vs competitive advantage—and career longevity through curiosity and humility
Dwarkesh asks whether publishing breakthroughs like Transformers helped competitors too much; Jeff and Noam describe shifting publication norms and the value of community progress. They close with advice on sustaining breadth: actively learning new fields, collaborating across expertise, staying humble, and creating incentives that support both exploration and honest failure reporting.
- •Publishing trade-offs: immediate openness vs product-first then publication vs secrecy
- •Community benefits of conferences and shared advances, alongside strategic restraint
- •Jeff’s approach: follow research, talk to domain experts, learn by collaboration
- •Noam’s emphasis on humility and dropping ideas quickly when better ones appear
- •Organizational incentives: bottom-up ‘UBI’ compute credits vs top-down coordination
