Jeff Dean & Noam Shazeer — 25 years at Google: from PageRank to AGI
Noam Shazeer (guest), Jeff Dean (guest), Jeff Dean (guest), Dwarkesh Patel (host), Dwarkesh Patel (host), Noam Shazeer (guest), Jeff Dean (guest)
In this episode of Dwarkesh Podcast, featuring Noam Shazeer and Jeff Dean, Jeff Dean & Noam Shazeer — 25 years at Google: from PageRank to AGI explores jeff Dean and Noam Shazeer Envision AI’s Self‑Improving, Compute‑Hungry Future Jeff Dean and Noam Shazeer reflect on 25 years at Google, from early search and massive n‑gram language models to today’s Gemini and TPUs. They describe how hardware and algorithms have co‑evolved, enabling deep learning, mixture‑of‑experts architectures, massive context windows, and increasingly capable coding and reasoning systems. A major theme is the coming feedback loop where AI designs better AI and hardware, driving rapid progress in algorithms, chips, and large‑scale systems, with huge implications for productivity and global GDP. They also discuss modular, continually‑learning models, inference‑time scaling, multi‑datacenter training, and the need to shape powerful systems safely while exploiting their vast economic and social upside.
Jeff Dean and Noam Shazeer Envision AI’s Self‑Improving, Compute‑Hungry Future
Jeff Dean and Noam Shazeer reflect on 25 years at Google, from early search and massive n‑gram language models to today’s Gemini and TPUs. They describe how hardware and algorithms have co‑evolved, enabling deep learning, mixture‑of‑experts architectures, massive context windows, and increasingly capable coding and reasoning systems. A major theme is the coming feedback loop where AI designs better AI and hardware, driving rapid progress in algorithms, chips, and large‑scale systems, with huge implications for productivity and global GDP. They also discuss modular, continually‑learning models, inference‑time scaling, multi‑datacenter training, and the need to shape powerful systems safely while exploiting their vast economic and social upside.
Key Takeaways
Hardware specialization and co‑design have been crucial to modern AI progress.
General‑purpose CPUs stopped scaling as fast, so Google built TPUs and leaned into reduced‑precision linear algebra; algorithms like deep learning and transformers then evolved to exploit cheap arithmetic and tolerate quantization.
Get the full analysis with uListen AI
Language models evolved from n‑grams to neural nets and now power many Google products.
Early massive n‑gram models and spelling correction showed that modeling word sequences at web scale was powerful, but only with later neural architectures and huge compute did these ideas become today’s LLMs and Gemini.
Get the full analysis with uListen AI
Inference‑time compute is an underexploited axis for improving AI capability.
Because tokens are extremely cheap compared to human labor, there is enormous headroom to spend 10–1,000× more compute at inference, using search, multi‑step reasoning, and drafter–verifier setups to get significantly better answers.
Get the full analysis with uListen AI
AI is already materially boosting software development productivity and will increasingly assist research.
About 25% of characters in Google’s code commits are AI‑generated, and Dean and Shazeer foresee near‑term systems that can generate, run, and iterate on complex research experiments from high‑level natural‑language specs.
Get the full analysis with uListen AI
Future models may be large, sparse, modular “blobs” that grow organically and are continually updated.
They propose mixture‑of‑experts‑style architectures with specialized modules, different compute depths per query, modular training by many teams, and frequent distillation into efficient sub‑models for serving.
Get the full analysis with uListen AI
AI will likely design better chips much faster, tightening the AI–hardware feedback loop.
If AI can shrink chip‑design cycles from ~18 months to months with small teams, hardware can track evolving ML algorithms more closely, enabling more specialized accelerators and faster capability gains.
Get the full analysis with uListen AI
Safety and control must keep pace with rapidly increasing capabilities and self‑improvement.
They argue for shaping and constraining deployment—via APIs, post‑training, AI‑for‑AI oversight, and policy—so that powerful models amplify beneficial uses (e. ...
Get the full analysis with uListen AI
Notable Quotes
“Organizing information is clearly like a trillion‑dollar opportunity, but a trillion dollars is not cool anymore. What's cool is a quadrillion dollars.”
— Noam Shazeer
“The world GDP is almost certainly going to go way, way up to, like, orders of magnitude higher than it is today... due to the fact that we have all of these artificial engineers.”
— Noam Shazeer
“25% of the characters that we're checking into our code base these days are generated by our AI‑based coding models.”
— Jeff Dean
“I think one of the beauties of deep learning is you don't need to understand or hand‑engineer every last feature... as long as the collective output and characteristics of the overall system are good.”
— Jeff Dean
“We’re going to need, like, a million automated researchers to invent all of this stuff.”
— Noam Shazeer
Questions Answered in This Episode
How realistic is the envisioned feedback loop where AI rapidly designs better AI and hardware, and what are the concrete failure modes if alignment lags?
Jeff Dean and Noam Shazeer reflect on 25 years at Google, from early search and massive n‑gram language models to today’s Gemini and TPUs. ...
Get the full analysis with uListen AI
What specific algorithmic ideas or architectures do Dean and Shazeer see as most promising for enabling 100–1,000‑step reliable reasoning rather than today’s 5–10 steps?
Get the full analysis with uListen AI
How would a modular, continually‑learning “blob” model actually be governed in practice—who decides which modules get added, updated, or restricted, and based on what criteria?
Get the full analysis with uListen AI
Given the projected explosion in inference demand, how will power, cooling, and environmental constraints shape AI deployment and access across different regions and economic classes?
Get the full analysis with uListen AI
To what extent should foundational breakthroughs like transformers or future ‘blob’ architectures be openly published versus held back as competitive or safety‑critical secrets?
Get the full analysis with uListen AI
Transcript Preview
Organizing information is clearly like a trillion-dollar opportunity, but a trillion dollars is not cool anymore. What's cool is a quadrillion dollars. (laughing)
(laughing)
The world GDP is almost certainly going to go way, way up to, like, orders of magnitude higher than it is today-
(laughs)
... due to the fact that we have all of these artificial engineers.
25% of the characters that we're checking into our code base these days are generated by our AI-based coding models.
We're going to need, like, a million automated researchers to invent all of this stuff. (laughs)
(laughs) Yeah.
(laughs) If this is where things go, this is actually like getting, like, Noam on a podcast in 2018 and being, like, "Yeah, so I think like, you know, language models will be a thing."
I'm guessing that the amount of compute being used for AI to help each person will be astronomical.
(laughs)
(laughs) Today, I have the honor of chatting with Jeff Dean and Noam Shazeer. Jeff is Google's chief scientist, and through his 25 years at the company, he has worked on basically the most transformative systems in modern computing, from MapReduce, BigTable, TensorFlow, AlphaChip. Genuinely, the list doesn't end. Uh, Gemini now. And Noam is the per- single person most responsible for the current AI revolution. He has been the inventor or the co-inventor of all the main, uh, ar- architectures and techniques that are used for modern LLMs, from the transformer itself, to mixture of experts, uh, to mesh-TensorFlow, to many other things. Um, and they are two of the three co-leads of Gemini at Google DeepMind. Awesome. Thanks so much for coming on.
Thanks for having us.
(laughs)
(laughs) Thank you.
Super excited to be here.
Okay, first question. Uh, both of you have been at Google for 25 or close to 25 years. At some point early on in the company, you probably understood how everything worked.
Mm-hmm.
When did that stop being the case? Do you feel like there was a clear moment that happened?
I mean, I- I know I joined and, like, at that poi- this was, like, end of 2000, and, uh, they had this thing, everybody gets a mentor. And, you know, so, you know, I knew nothing. I would just ask my mentor everything, and my mentor knew everything. It turned out my mentor was Jeff. (laughs)
(laughs)
(laughs)
(laughs)
And it was not the case that everyone at Google knew everything.
(laughs)
It was just the case that Jeff knew everything (laughs) 'cause he, 'cause he had basically written everything.
(laughs) You're- you're very kind. I mean, I think, uh, as companies grow, you- you kind of go through these phases. Like, when I joined, you know, we were 25 people, 26 people, something like that. And so you eventually learned everyone's name, and even though we were growing, you kept track of all the people who were joining. Uh, at some point, then you kind of lose track of everyone's name in the company, but you still know everyone working on, you know, software engineering things. Uh, then you sort of lose track of, you know, all the names of people in the software engineering group, but, you know, you at least know all the different projects th- uh, that everyone's working on. And then, at some point, the company gets big enough that, you know, you get an email that Project Platypus-
Install uListen to search the full transcript and get AI-powered insights
Get Full TranscriptGet more from every podcast
AI summaries, searchable transcripts, and fact-checking. Free forever.
Add to Chrome