Dwarkesh PodcastJeff Dean & Noam Shazeer — 25 years at Google: from PageRank to AGI
At a glance
WHAT IT’S REALLY ABOUT
Jeff Dean and Noam Shazeer Envision AI’s Self‑Improving, Compute‑Hungry Future
- Jeff Dean and Noam Shazeer reflect on 25 years at Google, from early search and massive n‑gram language models to today’s Gemini and TPUs. They describe how hardware and algorithms have co‑evolved, enabling deep learning, mixture‑of‑experts architectures, massive context windows, and increasingly capable coding and reasoning systems. A major theme is the coming feedback loop where AI designs better AI and hardware, driving rapid progress in algorithms, chips, and large‑scale systems, with huge implications for productivity and global GDP. They also discuss modular, continually‑learning models, inference‑time scaling, multi‑datacenter training, and the need to shape powerful systems safely while exploiting their vast economic and social upside.
IDEAS WORTH REMEMBERING
5 ideasHardware specialization and co‑design have been crucial to modern AI progress.
General‑purpose CPUs stopped scaling as fast, so Google built TPUs and leaned into reduced‑precision linear algebra; algorithms like deep learning and transformers then evolved to exploit cheap arithmetic and tolerate quantization.
Language models evolved from n‑grams to neural nets and now power many Google products.
Early massive n‑gram models and spelling correction showed that modeling word sequences at web scale was powerful, but only with later neural architectures and huge compute did these ideas become today’s LLMs and Gemini.
Inference‑time compute is an underexploited axis for improving AI capability.
Because tokens are extremely cheap compared to human labor, there is enormous headroom to spend 10–1,000× more compute at inference, using search, multi‑step reasoning, and drafter–verifier setups to get significantly better answers.
AI is already materially boosting software development productivity and will increasingly assist research.
About 25% of characters in Google’s code commits are AI‑generated, and Dean and Shazeer foresee near‑term systems that can generate, run, and iterate on complex research experiments from high‑level natural‑language specs.
Future models may be large, sparse, modular “blobs” that grow organically and are continually updated.
They propose mixture‑of‑experts‑style architectures with specialized modules, different compute depths per query, modular training by many teams, and frequent distillation into efficient sub‑models for serving.
WORDS WORTH SAVING
5 quotesOrganizing information is clearly like a trillion‑dollar opportunity, but a trillion dollars is not cool anymore. What's cool is a quadrillion dollars.
— Noam Shazeer
The world GDP is almost certainly going to go way, way up to, like, orders of magnitude higher than it is today... due to the fact that we have all of these artificial engineers.
— Noam Shazeer
25% of the characters that we're checking into our code base these days are generated by our AI‑based coding models.
— Jeff Dean
I think one of the beauties of deep learning is you don't need to understand or hand‑engineer every last feature... as long as the collective output and characteristics of the overall system are good.
— Jeff Dean
We’re going to need, like, a million automated researchers to invent all of this stuff.
— Noam Shazeer
High quality AI-generated summary created from speaker-labeled transcript.
Get more out of YouTube videos.
High quality summaries for YouTube videos. Accurate transcripts to search & find moments. Powered by ChatGPT & Claude AI.
Add to Chrome