Jeff Dean & Noam Shazeer — 25 years at Google: from PageRank to AGI

Noam Shazeer 0:00 Organizing information is clearly like a trillion-dollar opportunity, but a trillion dollars is not cool anymore. What's cool is a quadrillion dollars. (laughing)

Jeff Dean 0:08 (laughing)

Noam Shazeer 0:09 The world GDP is almost certainly going to go way, way up to, like, orders of magnitude higher than it is today-

Jeff Dean 0:16 (laughs)

Noam Shazeer 0:16 ... due to the fact that we have all of these artificial engineers.

Jeff Dean 0:19 25% of the characters that we're checking into our code base these days are generated by our AI-based coding models.

Noam Shazeer 0:25 We're going to need, like, a million automated researchers to invent all of this stuff. (laughs)

Jeff Dean 0:30 (laughs) Yeah.

Dwarkesh Patel 0:30 (laughs) If this is where things go, this is actually like getting, like, Noam on a podcast in 2018 and being, like, "Yeah, so I think like, you know, language models will be a thing."

Noam Shazeer 0:37 I'm guessing that the amount of compute being used for AI to help each person will be astronomical.

Jeff Dean 0:45 (laughs)

Dwarkesh Patel 0:47 (laughs) Today, I have the honor of chatting with Jeff Dean and Noam Shazeer. Jeff is Google's chief scientist, and through his 25 years at the company, he has worked on basically the most transformative systems in modern computing, from MapReduce, BigTable, TensorFlow, AlphaChip. Genuinely, the list doesn't end. Uh, Gemini now. And Noam is the per- single person most responsible for the current AI revolution. He has been the inventor or the co-inventor of all the main, uh, ar- architectures and techniques that are used for modern LLMs, from the transformer itself, to mixture of experts, uh, to mesh-TensorFlow, to many other things. Um, and they are two of the three co-leads of Gemini at Google DeepMind. Awesome. Thanks so much for coming on.

Jeff Dean 1:35 Thanks for having us.

Dwarkesh Patel 1:36 (laughs)

Jeff Dean 1:36 (laughs) Thank you.

1:36 Super excited to be here.

Dwarkesh Patel 1:37 Okay, first question. Uh, both of you have been at Google for 25 or close to 25 years. At some point early on in the company, you probably understood how everything worked.

Jeff Dean 1:48 Mm-hmm.

Dwarkesh Patel 1:49 When did that stop being the case? Do you feel like there was a clear moment that happened?

Noam Shazeer 1:53 I mean, I- I know I joined and, like, at that poi- this was, like, end of 2000, and, uh, they had this thing, everybody gets a mentor. And, you know, so, you know, I knew nothing. I would just ask my mentor everything, and my mentor knew everything. It turned out my mentor was Jeff. (laughs)

Dwarkesh Patel 2:10 (laughs)

Jeff Dean 2:10 (laughs)

2:10 (laughs)

Noam Shazeer 2:11 And it was not the case that everyone at Google knew everything.

Jeff Dean 2:13 (laughs)

Noam Shazeer 2:13 It was just the case that Jeff knew everything (laughs) 'cause he, 'cause he had basically written everything.

Jeff Dean 2:18 (laughs) You're- you're very kind. I mean, I think, uh, as companies grow, you- you kind of go through these phases. Like, when I joined, you know, we were 25 people, 26 people, something like that. And so you eventually learned everyone's name, and even though we were growing, you kept track of all the people who were joining. Uh, at some point, then you kind of lose track of everyone's name in the company, but you still know everyone working on, you know, software engineering things. Uh, then you sort of lose track of, you know, all the names of people in the software engineering group, but, you know, you at least know all the different projects th- uh, that everyone's working on. And then, at some point, the company gets big enough that, you know, you get an email that Project Platypus-

Dwarkesh Patel 2:57 (laughs)

Noam Shazeer 2:57 (laughs)

Jeff Dean 2:57 (laughs)

Noam Shazeer 2:57 ... is launching on Friday, and you're like, "What the heck is Project Platypus?"

Dwarkesh Patel 3:00 (laughs)

Noam Shazeer 3:00 So I think...

Jeff Dean 3:00 Usually it's a very good surprise.

Dwarkesh Patel 3:02 (laughs)

Jeff Dean 3:02 (laughs) Like, you're like, "Wow."

Noam Shazeer 3:03 Project Platypus.

Jeff Dean 3:04 Yeah. Wow. Project Platypus.

Noam Shazeer 3:04 It's like, I had no idea we were doing that, and, uh, then it turns out to be great.

Jeff Dean 3:07 Yeah, but I think it is good to keep track of, like, what's going on in the company, even at a very high level, even if you don't know every last detail. And it's good to know lots of people throughout the company so that you can go ask someone for more details or figure out who to talk to. I think, like, with one level of indirection, you can usually find the right person in the company if you have a good, you know, network of people that you've built up over time.

Dwarkesh Patel 3:29 Um, ho- how did Google recruit you, by the way?

Jeff Dean 3:31 Um, I kind of reached out to them, actually.

Dwarkesh Patel 3:35 And- and Noam, how did you get recruited? Wha- what was it that you did there?

Noam Shazeer 3:38 Yeah. I mean, I, um, I actually saw i- uh, Google at a job fair in, like, 1999, and I assumed that it was, like, already this huge company, that no point in joining-

Dwarkesh Patel 3:48 (laughs)

Noam Shazeer 3:49 ... um, because everyone I knew used Google. I guess that was because I was a grad student at Berkeley at the time.

Dwarkesh Patel 3:54 (laughs)

Jeff Dean 3:55 (laughs)

Noam Shazeer 3:55 I- I guess I've dropped out of grad programs a few times, but, um, but, you know, it turns out that, like, actually it wasn't really that, uh, that large (laughs) . So-

Dwarkesh Patel 4:04 (laughs)

Jeff Dean 4:04 ... so it tur- turned, turns out I did not apply in 1999, but, like, just kind of sent them a resume on a whim in 2000 'cause I figured I should let, it was, like, my favorite search engine and figured I should apply to multiple places for a job. Um, but then, yeah, turned out to be, uh, be, uh, really, uh, really fun. Looked like a bunch of smart people, uh, doing good stuff, and they had this really nice crayon chart on the wall of the daily, uh, number of search queries-

Dwarkesh Patel 4:32 I love that chart. Uh-huh.

Noam Shazeer 4:32 ... uh, that, you know, somebody had just been maintaining, uh. And, um, yeah, it looked very exponential. (laughs)

Dwarkesh Patel 4:39 (laughs)

Jeff Dean 4:39 (laughs)

Noam Shazeer 4:39 I was like, "These guys are going to be very successful." And it looks like they have a lot of good problems to work on. So I was like, "Okay, maybe I'll, uh, yeah, go work there for a little while and then have enough money to just go work on AI for as long as I want after that." (laughs)

Dwarkesh Patel 4:53 Yeah, yeah. In a way, you did that, right?

Jeff Dean 4:55 (laughs)

Dwarkesh Patel 4:55 (laughs)

Noam Shazeer 4:55 Yeah, yeah. It, uh, yeah, it- it totally worked out exactly according to plan.

Dwarkesh Patel 4:59 Right. So you were thinking about AI in 1999?

Noam Shazeer 5:02 Um, yeah, this was like 2000. Yeah, I- I- I remember in, um, yeah, in grad school, uh, a- a friend of mine at the time had, um, told me that his, um, uh, New Year's resolution for 2000 was to live to see the year 3000 and that he was going to achieve this by inventing AI.

Dwarkesh Patel 5:21 Uh-huh.

Noam Shazeer 5:21 Um, so I was like, "Huh, that's- that sounds like a good idea." Um, but, you know, then, you know, I- I- I didn't get the idea at the time that, oh, like, you could go do it at- at a big company, but, you know, I figured, hey, you know, a bunch of people seem to be making a ton of money at startups. Maybe I'll just make some money and then I'll have, uh, you know, enough to live on, just work on AI research, uh, for- for a long time.

Dwarkesh Patel 5:46 Yeah.

Noam Shazeer 5:46 Um, but yeah, it actually turned out that Google was a terrific place to work in AI.

Jeff Dean 5:51 I mean, one of the things I like about Google is our ambition has always been sort of something that would kind of require pretty advanced AI.

Dwarkesh Patel 5:59 Mm-hmm.

Jeff Dean 5:59 You know, organizing the world's information and making it universally accessible and useful. Like, actually, there's ton- there's a really broad mandate in there, uh, so it's not like the company was gonna do this one little thing and stay in- stay doing that. And also, you could see that what we were doing initially was...... in that direction, but you could do so much more in that direction.

Dwarkesh Patel 6:19 How

Dwarkesh Patel 6:20 has Moore's Law over the last two, three decades changed the kinds of considerations you have to take on board when you design new systems, when you figure out what projects are feasible? What has stayed, you know, like, what, what are, what are still the limitations? What are things you can now do that you obviously couldn't do before?

Jeff Dean 6:35 I mean, I, I, I think of it as actually changing quite a bit in the last couple decades.

Dwarkesh Patel 6:38 Mm-hmm.

Jeff Dean 6:38 So like the two decades ago to one decade ago, it was awesome 'cause you'd just, like, wait and, like, 18 months later, you'd get much faster hardware and you don't have to do anything (laughs) . Um, and then more recently, you know, I feel like the general purpose CPU-based s- uh, machines, scaling has not been as good. Like, the fabrication processes improvements are now taking three years instead of every two years. The, the architectural improvements in, you know, multi-core processors and so on are, you know, not giving you the same boost that you, that we were getting, you know, the pre-, you know, 20 to 10 years ago. Um, but I think at the same time you were seeing, uh, much more specialized computational devices, uh, like machine learning accelerators, TPUs, um, very ML-focused GPUs more recently, um, are making it so that we can actually get, you know, really high performance a- and good efficiency out of the more modern kinds of computations we wanna run that are, that are different than, you know, a twisty pile of C++ code trying to run Microsoft Office or something.

Dwarkesh Patel 7:45 Yeah, yeah. (laughs)

Jeff Dean 7:46 (laughs)

Noam Shazeer 7:46 I mean, it feels like the, the, um, the algorithms are following the hardware. Basically, like, what's happened is that at this point arithmetic is very, very cheap and moving data around is comparatively, like, much more expensive.

Dwarkesh Patel 8:03 Right.

Noam Shazeer 8:03 So pretty much all of deep learning has taken off roughly because of that, because it, you can build it-

Dwarkesh Patel 8:11 Interesting.

Noam Shazeer 8:11 ... out of matrix multiplications that are, you know, n, n cubed operations and n squared bytes of, uh, data communication basically.

Jeff Dean 8:22 Well, I would say that the, the pivot to hardware oriented around that was an important transition. 'Cause before that we had CPUs and-

Noam Shazeer 8:31 Right, yeah.

Jeff Dean 8:31 ... GPUs that were not, you know, especially well-suited for deep learning. And then, um, you know, we started to build, say, TPUs at Google, uh, that were really just reduced precision linear algebra machines.

Dwarkesh Patel 8:43 Yeah.

Jeff Dean 8:43 And then once you have that, then you wanna-

Noam Shazeer 8:45 R- right.

Jeff Dean 8:45 ... you know, exploit it.

Noam Shazeer 8:45 You have to see the insight that seems like it's all about, uh, all about kind of identifying opportunity cost. Like, okay, thi- this is something like Larry Page, I think used to always say, like, uh, "Our second biggest cost is taxes, and our biggest cost is opportunity cost."

Dwarkesh Patel 9:02 (laughs)

Jeff Dean 9:02 (laughs)

Noam Shazeer 9:02 And if he didn't say that, then I've been misquoting him for years, but-

Dwarkesh Patel 9:05 (laughs)

Noam Shazeer 9:06 ... uh, but, but, uh, but basically it's like, you know, what, what, what is the opportunity that you have that-

Dwarkesh Patel 9:12 Right.

Noam Shazeer 9:12 ... that you're missing out on? Um, and like in this case, I guess it was that, okay, you've got all of this chip area and you're putting a very small number of, uh, of arithmetic units on it. Like, fill the thing up with arithmetic units, you could have orders of magnitude more arithmetic getting done. Now what else has to change? Okay, the algorithms and the data flow and everything else-

Jeff Dean 9:35 And oh, by the way-

Noam Shazeer 9:35 ... about the chip.

Jeff Dean 9:36 ... the arithmetic can be, like, really low precision, so then you can squeeze even more multiplier units in.

Dwarkesh Patel 9:41 Um, Noam, I wanna follow up on what you said, that the algorithms have been following the hardware. If you imagine a counterfactual world where suppose that the cost of memory had declined more than arithmetic, or, uh, just like in- invert the dynamic you saw with the last decade.

Noam Shazeer 9:55 Right. Yeah, yeah. That, that, okay, data flow is, is extremely cheap and arithmetic-

Dwarkesh Patel 10:00 Yeah.

Noam Shazeer 10:00 ... is not cheap.

Dwarkesh Patel 10:01 Wh- wh- what would, what would AI look like today?

Noam Shazeer 10:03 That's interesting.

Jeff Dean 10:04 You'd have a lot more lookups into very large memories.

Noam Shazeer 10:06 Yes (laughs) .

Jeff Dean 10:07 (laughs) I think.

Noam Shazeer 10:08 Yeah. I mean, I think it might look more like AI looked like 20 years ago, or, but in the opposite direction? I, I, like, I, I'm not sure. I, I guess I joined, uh, Google Brain in 2012, um... Hap- you know, I'd left Google for a c- a few years, happened to, like, go back for lunch to visit my wife, and, uh, and, uh, we happened to sit ne- down next to Geoff and the early Google Brain team. And I thought, "Wow, that's a smart group of people doing something."

Jeff Dean 10:38 I think I said, "You should, you should, you should think about deep neural nets 'cause we're making some pretty good progress here."

Noam Shazeer 10:42 Like, "That, that sounds fun." So okay-

Jeff Dean 10:44 Yeah.

Noam Shazeer 10:44 ... so I jumped back in to join-

Jeff Dean 10:45 I wooed him back. It was great. (laughs)

Noam Shazeer 10:47 ... uh, to join Geoff. That was, like, 2012. Um, I, I seem to join Google every 12 years.

Dwarkesh Patel 10:53 (laughs)

Noam Shazeer 10:53 Uh, I rejoined Google in 2000, 2012 and 2024. Um, but-

Dwarkesh Patel 10:57 What's gonna happen in 2036? (laughs)

Noam Shazeer 10:59 Ooh, I don't know. I guess, uh (laughs) ... we shall see. Um...

Dwarkesh Patel 11:04 Wh- wh- what are the trade-offs that you're considering changing for future, uh, versions of TPU to s- to integrate how, how are you thinking about algorithms differently?

Jeff Dean 11:12 Um, I mean, I think one thing, one general trend is we're getting better at quantizing or having much re- more reduced precision models.

Dwarkesh Patel 11:21 Yeah.

Jeff Dean 11:22 Uh, you know, we started with TPU v1. We weren't even quite sure we could quantize a model for serving with eight-bit integers. But we sort of had some early evidence that seemed like it might be possible, so we're like, "Great, let's build the whole chip around that." (laughs)

Dwarkesh Patel 11:35 (laughs)

Jeff Dean 11:36 And then over time, I think you've seen people, uh, able to use much lower precision for training as well, but also f- the inference precision has, you know, gone... people are now using INT4 or FP4, which sounded like if you said to someone, like, "We're gonna use FP4,"-

Dwarkesh Patel 11:52 (laughs)

Jeff Dean 11:53 ... to s- like a super computing floating point person 20 years ago would be like, "What? That's crazy. We like 64 bits in our floats."

Dwarkesh Patel 11:59 (laughs)

Jeff Dean 12:00 Um, or, you know, even below that. You know, some people are quantizing models to two bits or one bit, uh, and I think that's a, that's a trend to definitely pay attention to.

Dwarkesh Patel 12:09 One bit, just like a zero or one? (laughs)

Jeff Dean 12:11 Yeah, just a zero or one. Um, and then you have, like, a, a sign bit for a group of bits or something.

Dwarkesh Patel 12:16 Ah.

Noam Shazeer 12:16 (laughs)

Dwarkesh Patel 12:16 (laughs)

Noam Shazeer 12:17 Really has to be a co-design thing because, you know, if-... you know, if, if, if the, um... You know, if the algorithm designer doesn't realize that he can get greatly improved, uh, performance, you know, uh, throughput with the lower precision, of course the algorithm designer's going to say, "Of course I don't want low precision." (laughs) Like, that-

Jeff Dean 12:38 Right.

Noam Shazeer 12:38 ... that, that introduces risk, and then, uh, you know, it adds irritation. And then the, uh... Then if you ask the chip designer, "Uh, okay. Uh, you know, what, what are, what do you wanna build?" And then th- they'll ask the person who's w- who's writing the algorithms today, who's gonna say, "No, I don't like, uh, I don't like quantization. It's irritating."

Jeff Dean 12:57 (laughs)

Noam Shazeer 12:57 So, y- you actually, you know, need to basically see the whole picture and-

Jeff Dean 13:02 Right.

Noam Shazeer 13:02 ... figure out, "Oh, wait a minute. We can, you know, we can incr- increase our, uh, throughput to cost ratio by a lot by, uh, you know-

Jeff Dean 13:10 Right.

Noam Shazeer 13:10 ... by quantizing."

Jeff Dean 13:11 Then you're like, "Yes, quantization is irritating." (laughs) But your model's gonna be three times faster, so you're gonna have to deal. (laughs)

Noam Shazeer 13:16 (laughs)

Jeff Dean 13:17 Uh, through your careers, at various times, you've had sort of an uncanny... You worked on things that have an uncanny resemblance to what is actually, what we're actually using now for generative AI. In 1990, Geoff, your senior thesis was about, uh, uh, back propagation. And in 2007... So, th- this is, this is the thing I didn't realize until I was prepping for this episode. In 2007, you guys trained a two trillion token n-gram model for language modeling. Um, I, I... Just walk me through, when you were developing that model, what, what... Like, was this kind of thing in your head? What, what did you think you guys were doing at the

Jeff Dean 13:56 time? Yeah. So, um, I mean, let, let me start with the undergrad thesis. Yes. So I kind of got introduced to neural nets in one section of one class on parallel computing that I was taking in my senior year, um, and I needed to do a thesis to graduate, like an honors thesis. And so I approached the professor and I said, "Oh, it'd be really fun to, like, do something around neural nets." So he, he and I decided we would, I would, uh, sort of implement a couple different ways of parallelizing, uh, back propagation training for neural nets in, in 1990, and I called them something funny in my thesis, like, like, uh, pattern partitioning or something. But really, I implemented a, you know, model, model parallelism and data parallelism-

Noam Shazeer 14:33 Mm-hmm.

Jeff Dean 14:33 ... on a 32 cube, 32 processor hypercube machine. Um, you know, in one, you split all the examples into different batches, and every model, every CPU has a copy of the, the model, and in the other one, you kind of pipeline a bunch of examples, uh, along to processors that have different, um, different, uh, parts of the model. And, you know, com- I compared and contrasted them. And, uh, it was interesting, you know. I was really excited about the abstraction, 'cause it felt like neural nets were the right abstraction that could solve tiny toy problems that no other approach could solve at the time. Um, but... And I thought, "Oh," you know, naive me, "Oh, 32 processors, we'll be able to train, like, really awesome neural nets." Uh, but it turned out, you know, we needed about a million times more compute before they-

Noam Shazeer 15:19 (laughs)

Jeff Dean 15:19 ... really started to work for real problems. But then starting, you know, in the, you know, late 2008, 2009, 2010 timeframe, we started to have enough compute, uh, thr- thanks to Moore's law, uh, to actually make neural nets work for real things. And that was kind of when I sort of reentered, uh, looking at neural nets. But prior to that, in 2007-

Noam Shazeer 15:38 Sorry, can I ask about this for just-

Jeff Dean 15:40 Oh, yeah. Sure, yeah.

Noam Shazeer 15:41 ... um, uh, um... First of all, uh, unlike other artifacts of academia-

Jeff Dean 15:45 Yeah.

Noam Shazeer 15:45 ... it's actually, like, a really... Like, it's, like, four pages, and you can just, like, read it. And, um, and it's-

Jeff Dean 15:49 Yeah. It was four pages and then, like, 30 pages of C code. (laughs)

Noam Shazeer 15:53 Sure. (laughs) But it's,

Noam Shazeer 15:54 like, just, like, a well-produced sort of artifact. Um, and then, yeah, tell me about how the 2007 paper came together.

Jeff Dean 15:58 Oh, yeah. So that, we had a, we had a machine translation research team at Google, um, uh, led by Franz Och, uh, who had joined Google maybe a year before, um, and a bunch of other people. And every year, they competed in a, uh, uh, uh, I guess it's a DARPA contest on translating a couple of different languages to English, I think. Chinese to English and Arabic to English, I think. Um, and, uh, the Google team had submitted an entry. And the way this works is you get, like, I don't know, 500 sentences on Monday, and you have to submit the answer on Friday. Um, and so the... I saw the results of this, and we'd won the contest, um, and, uh, by a pretty substantial margin measured in BlueScore, which is like a measure of translation quality. And so I reached out to Franz, uh, the, the head of this winning team. I'm like, "This is great. When are we gonna launch it?" And he's like, "Oh, well, we can't launch this. It's not really very practical, 'cause it takes 12 hours to translate a sentence."

Noam Shazeer 17:00 (laughs)

Jeff Dean 17:01 (laughs) I'm like, "Well, well, that seems like a long time." Um, "How could we fix that?" So it turned out, you know, they'd not really designed it for high throughput, obviously. (laughs)

Noam Shazeer 17:14 (laughs) Sure.

Jeff Dean 17:15 And so it was doing like 100,000 dist-seqs to... In a, in a large language model that they, they, uh, tra- uh, sort of computed statistics over. I wouldn't say trained, really. Um, and, you know, for each word that it wanted to translate. So, like, obviously doing 100,000 dist-seqs is not-

Noam Shazeer 17:33 Yeah. (laughs)

Jeff Dean 17:33 ... super speedy. But I said, "Okay, well, let's, let's dive into this." And so I spent about two or three months with them, uh, designing an in-memory compressed representation of n-gram, uh, data. And, and we were using n... An n-gram is basically statistics for how often every N-word sequence occurs in a large corpus. So you basically have... You know, in this case, we had like two trillion words, and most n-gram models of the day were, like, using two grams or maybe three grams. Uh, but we decided we would use five grams.

Noam Shazeer 18:03 Mm-hmm.

Jeff Dean 18:03 So how often every five-word sequence-

Noam Shazeer 18:05 Right.

Jeff Dean 18:05 ... occurs in basically as much of the web as we could, uh, process that, that... In that day. Um, and then you have a data structure that says, "Okay, you know, 'I really like this restaurant,'" uh, occurs, you know, 17 times in the web or something. Um, and-... and so, uh, I built, like, a data structure that would let you, uh, store all those in memory on 200 machines, and then have sort of a batched API where you could say, "Here are the 100,000 (laughs) things I need to look up in this round for this word." And it would give you the mall back in pa- in parallel. Um, and that enabled us to go from taking a night to translate a sentence to basically doing something in, you know, 100 milliseconds or something. (laughs)

Dwarkesh Patel 18:47 There's, um, there's this- there's this, uh, list of, uh, Jeff Dean facts, like Chuck Norris facts.

Jeff Dean 18:54 Facts. (laughs)

Noam Shazeer 18:55 (laughs)

Dwarkesh Patel 18:56 Um, like, for example, that, uh, for Jeff Dean, NP equals no problemo.

Noam Shazeer 19:00 (laughs)

Jeff Dean 19:01 (laughs)

Dwarkesh Patel 19:02 Um, and one of them-

Jeff Dean 19:04 No.

Dwarkesh Patel 19:04 ... uh, it- it's funny 'cause, uh, now that I hear you say it, it's like, actually, I think it's kind of true.

Jeff Dean 19:07 (laughs)

Dwarkesh Patel 19:07 Um, one of them is, um, the speed of light was 35 miles an hour-

Jeff Dean 19:10 (laughs)

Dwarkesh Patel 19:10 ... until Jeff Dean decided to optimize it over a weekend. (laughs)

Jeff Dean 19:14 (laughs)

Noam Shazeer 19:14 (laughs)

Dwarkesh Patel 19:14 Just going from 12 hours to, uh, like, 100 milliseconds or whatever is like-

Jeff Dean 19:18 Yeah.

Dwarkesh Patel 19:18 (laughs) I gotta do the orders of magnitude there, but- (laughs)

Jeff Dean 19:20 All of these- all of these are very flattering.

Dwarkesh Patel 19:22 (laughs)

Jeff Dean 19:23 They're- they're- they're pretty funny. They're like an April Fools' joke gone awry by my colleagues.

Noam Shazeer 19:29 (laughs)

Dwarkesh Patel 19:29 Um, okay, so, uh, obviously in retrospect, this idea that you can develop a latent representation of the entire internet through just considering the relationships between words, uh, is like, yeah, this is- this is large language models, this is, uh, Gemini. At the time, what... was it just a translation idea or did you see that as being the beginning of a different kind of paradigm?

Jeff Dean 19:54 I think once we built that for translation, the- the serving of large language models started to be used for other things like, uh, completion of, you know, you start to type and it suggests-

Dwarkesh Patel 20:05 Yeah.

Jeff Dean 20:05 ... like, what completions make sense.

Dwarkesh Patel 20:07 Right.

Jeff Dean 20:07 Um, so it was definitely the start of- of a lot of uses of language models in- in Google. And, you know, Noam has worked on a number of other things at Google-

Dwarkesh Patel 20:16 Mm-hmm.

Jeff Dean 20:16 ... like spelling correction systems that use language models for-

Dwarkesh Patel 20:19 Right.

Jeff Dean 20:19 ... transmitting words.

Noam Shazeer 20:19 Yeah, I think th- yeah, that that was, like, 2000-

Jeff Dean 20:22 That wa-

Noam Shazeer 20:22 ... 2000, 2001. And then, and there, I think it was just all in memory on one- on one machine.

Jeff Dean 20:27 Yeah, I think it was one machine.

Noam Shazeer 20:28 Yeah, it was a- but-

Jeff Dean 20:28 But his spelling correction-

Noam Shazeer 20:29 So, yeah.

Jeff Dean 20:29 ... system he built in 2001 was amazing.

Dwarkesh Patel 20:31 (laughs)

Jeff Dean 20:32 Like, he sent- he sent out this demo link to the whole company and, like, I just tried every butchered spelling of every-

Dwarkesh Patel 20:38 (laughs)

Jeff Dean 20:38 ... keyword query I could get. Like, scrumbled ugs bundict. (laughs)

Noam Shazeer 20:42 Oh, I remember that one. Yeah, yeah. (laughs)

Jeff Dean 20:44 Instead of scrambled eggs Benedict.

Noam Shazeer 20:45 (laughs)

Jeff Dean 20:46 And, like, it just nailed it every time.

Noam Shazeer 20:48 Yeah, and that- I guess that was language modeling.

Jeff Dean 20:50 Yeah.

Noam Shazeer 20:50 It's... Yeah.

Dwarkesh Patel 20:50 But- but, uh, at the time, when you were developing this system-

Noam Shazeer 20:53 Yeah.

Dwarkesh Patel 20:53 ... did, uh, did you have this sense of, look, you make these things more and more sophisticated, you don't consider five words, but if you consider 100 words, 1,000 words, then the latent representation is intelligence? Or was that... uh, like, basically, when did that insight hit?

Noam Shazeer 21:08 Not really. I mean, like, not... uh, like, I don't think I ever felt like, okay, n-gram models are going to, you know, are going to-

Jeff Dean 21:16 Sweep the world. (laughs)

Noam Shazeer 21:16 Yeah, be- be, uh, artificial intelligence. Um, I think- I think at the time, uh, I was... a lot of people were excited about the Bayesian networks. Uh, that was- that seemed exciting. Definitely seeing, like, those early neural language models, um, you know, there- there's b- both- both the magic in that, okay, this is doing something extremely cool, and also, uh, also it's just struck me as, like, the- the best problem in the world. Like, in that, like, for one, it is very, very simple to state. Like, give me a probability distribution over the next word. Um, also, uh, there's roughly infinite training data out there.

Dwarkesh Patel 21:56 Mm-hmm.

Noam Shazeer 21:56 There's like the text of the web, you have, like, trillions of training examples, like, um, you know, of- of- of unsupervised data. And- and- and then-

Jeff Dean 22:05 Yeah, or self-supervised really.

Noam Shazeer 22:05 Self-supervised.

Jeff Dean 22:06 Yeah, it's nice.

Noam Shazeer 22:06 Yeah.

Jeff Dean 22:06 'Cause you then have the right answer-

Noam Shazeer 22:08 Right.

Jeff Dean 22:08 ... and then you can train on, like, all but the current word and try to predict the current word. And it's this kind of amazing, you know, ability to just learn from observations of the world.

Noam Shazeer 22:19 Yep. And then it's AI-complete. If you can do a great job of that-

Jeff Dean 22:22 (laughs)

Noam Shazeer 22:22 ... then you can pretty much, uh, pretty much do anything.

Dwarkesh Patel 22:25 (swoosh sound) I'm excited to introduce our new sponsor, Meter. They're a networking company that is behind a growing fraction of the world's internet infrastructure. Fun fact, about three to four years ago, in the very early days of the podcast, I ran this podcast from a donation from Meter CEO, Anil, and I continue to benefit enormously from his advice to this day. The modern world runs on networks. Progress in fields as diverse as self-driving cars to giant LLM training runs to even broadcasting a podcast like this around the world is bottlenecked on designing and debugging large, complex networks. Meter wants to give network engineers 100X multiplier by training a large end-to-end foundation model using time series packet data and support tickets and networking textbooks and all the other proprietary data they have as a result of themselves building every layer of the networking stack in-house. Meter just announced a long-term compute partnership with Microsoft for access to tens of thousands of GPUs. They're currently recruiting a world-class AI research team. Their goal is to build autonomous networks that radically improve the digital world that we take for granted. To learn more, go to meter.com/gwarkesh. All right, back to Jeff and Noam. (swoosh sound) There's this interesting discussion in the history of science about whether ideas are just in the air and there's a sort of inevitability to big ideas, or whether-

Jeff Dean 23:53 Mm-hmm.

Dwarkesh Patel 23:55 ... it's sort of pl- uh, plucked out of some tangential direction. In this case, this way in which you're laying it out very logically, um, does that imply... like, basically, how- h- h- how inevitable does this, uh-

Noam Shazeer 24:05 Uh, it- it does feel like it's in the air. There were definitely some, um... there was, like, this neural Turing machine. So yeah, a bunch of ideas around this attention/ there's this, like, having these, uh, key value stores that, uh, that could be useful in- in neural networks to kind of focus on things.

Dwarkesh Patel 24:28 Yeah.

Noam Shazeer 24:28 Um, so yeah, I- I- I think in- in- i- in some sense, uh, in the air, and in some sense, you know, you need some group to- uh, to go do it.

Dwarkesh Patel 24:37 Sure.

Jeff Dean 24:37 I mean, I- I like to think of a lot of ideas as-... they're kind of partially in the air, where there's, like, a few different maybe par- separate research ideas that one is kind of squinting at when you're trying to solve a new problem-

Dwarkesh Patel 24:49 Yeah.

Jeff Dean 24:50 ... and you kind of draw on those for some inspiration. And then there's, like, some aspect that is not solved, and you sort of need to figure out how to solve that. And then the combination of, like, some morphing of the things that already exist and some new things lead to some new breakthrough or new research result that, that didn't exist before.

Dwarkesh Patel 25:08 Mm.

Dwarkesh Patel 25:09 Were there, uh, are there key moments that stand out to you where you, looking at a, a research area and you come up with this idea, and you have this feeling of like, "Holy shit, I can't believe that worked"? Um. (laughs)

Jeff Dean 25:22 One, one thing I remember was, you know, we'd been f- in the early days of the Brain Team, we were focused on, "Let's see if we can build some infrastructure that lets us train really, really big neural nets."

Dwarkesh Patel 25:32 Yeah.

Jeff Dean 25:33 And at that time, we didn't have GPUs in our data centers. We just had CPUs. But, you know, we know how to make lots of CPUs work together. (laughs) So, we built a system that enabled us to train, you know, uh, pretty large neural nets through both model and data parallelism. So we had a, a system for unsupervised learning on, uh, uh, actually 10 million randomly selected, uh, YouTube frames. Uh, and it was kind of a, you know, a, a, a spatially local representation, so it would build up unsupervised representations, uh, based on trying to reconstruct the, the thing from the high-level representations. And, um, so we got that working and training on 2,000 computers using 16,000 cores. Uh, and, um, you know, after a little while, that model was actually able to build a representation at the highest level where one neuron would get excited by, uh, you know, uh, images of cats that-

Dwarkesh Patel 26:32 Mm-hmm.

Jeff Dean 26:32 ... you know, it had never been told what a cat was-

Dwarkesh Patel 26:34 Right.

Jeff Dean 26:34 ... but it sort of had seen enough, uh, examples of them in the training data of head-on facial views of cats-

Dwarkesh Patel 26:41 Yeah.

Jeff Dean 26:41 ... that that neuron would turn on for that and not for much else.

Dwarkesh Patel 26:44 Yeah.

Jeff Dean 26:44 And similarly, you'd have other ones for human faces, and, you know, uh, backs of pedestrians and this kind of thing. Um, and so that was kinda cool 'cause it's sort of from unsupervised learning principles building up these really high-level representations. And then we were able to get, you know, very good results on the supervised ImageNet 20,000 category, uh, challenge that, like, advanced the state of the art by, like, 60% relative improvement, which was quite good at the time. So, that to m- and that neural net was probably 50X bigger than one that had been trained, uh, previously.

Dwarkesh Patel 27:17 Mm.

Jeff Dean 27:18 Um, and it got good results. So that sort of said to me, "Hey, actually scaling up neural nets seems like a (laughs) ... I thought it would be a good idea, and it seems to be, so we should keep pushing on that."

Dwarkesh Patel 27:28 So, the... these examples illustrate how these AI systems fit into what you were just mentioning, that Google is sort of a company that organizes information fundamentally. And then you can... basically what AI is doing in this context is finding relationships between information, between concepts to help get ideas to you faster, information you want to you faster. Um, now we're moving with current AI models, like obviously they're very... you know, you can use BERT in Google Search, and you can ask these things questions, and they obviously are still good at information retrieval. But more fundamentally, you know, like, they're like... uh, they can, like, write your entire code base for you and do all... (laughs) you know, like-

Jeff Dean 28:09 That seems

Noam Shazeer 28:09 ... actually useful.

Dwarkesh Patel 28:09 ... more like an actual worker-

Jeff Dean 28:11 Yeah, yeah.

Dwarkesh Patel 28:12 ... um, which is going beyond the, uh, just, like, information retrieval. So-

Jeff Dean 28:16 Yeah.

Dwarkesh Patel 28:16 ... has, um... yeah, has your... how are you thinking about, like, is Google still an information retrieval company if you're, like, building an AGI? Like, AGI can do information retrieval, but it can do many other things as well, right?

Jeff Dean 28:28 I think we're an organized inf- inf- the world's information company, and that's broader than information retrieval, right?

Dwarkesh Patel 28:33 Sure.

Jeff Dean 28:33 That's maybe organizing and creating new information from, you know, some guidance you give it. "Can you help me write a, a letter to my, to my veterinarian about my dog? It's got these symptoms," and it'll draft that. Or, "Can you feed in this video, and, you know, can you produce a summary of like what's happening in the video every few minutes?" And, you know, I think our sort of multimodal capabilities are showing that it's more than just text. It's about, you know, understanding the world and all the different kind of modalities that, that information exists in. Um, both kind of human ones, but also, uh, kind of non-human-oriented ones, like weird, uh, LiDAR sensors on autonomous vehicles or, you know, genomic information or health information. And then helping... how do you extract and transform that into useful insights for people and make use of that in, in helping them do all kinds of things they wanna do? And that's, you know, sometimes it's, "I wanna be entertained by chatting with a chatbot." Sometimes it's, "I want answers to this really complicated question." There is no single s- source to retrieve from. It's... you need to pull information from, like, 100 web pages and, like, figure out what's going on and make a organized, synthesized version of that data. And, uh, and then dealing with, you know, multimodal things or coding-related problems. I think it's super exciting what these models are capable of, and they're improving fast. So, I'm excited to see where we go. I don't know about you.

Noam Shazeer 30:00 I am also excited to see where we go.

Jeff Dean 30:02 (laughs)

Noam Shazeer 30:02 And, you know, yeah, I think, uh, definitely the, uh, uh, or- organizing, organizing information, you know, is, you know, is, is, is clearly like a, uh, you know, trillion-dollar opportunity. But, you know, a trillion dollars is not cool anymore. What's cool is a quadrillion dollars.

Dwarkesh Patel 30:20 (laughs)

Jeff Dean 30:20 (laughs)

Noam Shazeer 30:20 (laughs) I mean, and obviously the, the, the, the, the, the, the, the idea is not to just pile up some giant pile of money, but, uh, it's to just... it's create value in the world-

Dwarkesh Patel 30:29 Right.

Noam Shazeer 30:29 ... you know, and, uh, and so much more value can be, uh, created when these, when these, uh, systems can actually, like, go and do something for you, write your code, or figure out problems that, uh, that, that you wouldn't have been able to figure out yourself and to, uh, and to do that at scale. So I, I, I mean-... we, we're going to have to be very, very flexible and dynamic as, uh, as, as, uh, as we, we improve the capabilities of these models to, you know-

Jeff Dean 30:56 Yeah, I guess I'm, I'm pretty excited about kind of a lot of fundamental research questions-

Noam Shazeer 31:01 Yeah.

Jeff Dean 31:01 ... that sort of come about because you see something that we're doing could be substantially improved if we-

Noam Shazeer 31:06 Mm-hmm.

Jeff Dean 31:07 ... tried, you know, this approach or things in this rough direction and, you know, maybe that'll work, maybe it won't. But I also think there's, there's value in seeing what we could achieve for end users and then how, how can we work backwards from that to actually build systems that are able to do that. So as one example, you know, organizing information, that should mean any information in the world should be usable by anyone regardless of what language they speak.

Noam Shazeer 31:31 Yeah.

Jeff Dean 31:32 And that I think-

Noam Shazeer 31:33 Yeah.

Jeff Dean 31:33 ... you know, we've done some amount of but it's not nearly the full vision of, you know, no matter what language you speak out of thousands of languages, we can make any piece of content available to you and make, make it usable by you, um, and, you know, any video could be watched in any language. I think that would be pretty awesome. Um, and you know, we're not quite there yet but that's definitely things I see on the horizon that would, should be possible.

Dwarkesh Patel 31:59 The,

Dwarkesh Patel 32:00 the, speaking of different architectures you might try, I know one thing you're working on right now is longer context. Um, if you think of Google Search as, like, it's got the entire index of the internet in its context but it's, like, sort of very, like, shallow search, uh, and then obviously language models all have, like, uh, limited context right now but they can, like, really think... It's like dark magic, like, in-context learning, right? It just, like, can really think about what, what it's seeing. Um, how, how do you think about what it would be like to merge something like Google Search and something like in-context learning?

Jeff Dean 32:31 Yeah, maybe I'll, I'll take a first stab at it. I mean, I... 'Cause I've thought about this for a bit. I mean, I think one of the things you see with these models is they, they're quite good but they do hallucinate and, you know, have factuality issues sometimes.

Dwarkesh Patel 32:43 Yeah.

Jeff Dean 32:43 And part of that is, you know, you've trained on, say, tens of trillions of tokens and you've stirred all that together in your tens or hundreds of billions of parameters.

Dwarkesh Patel 32:53 Yeah.

Jeff Dean 32:54 But it's all a bit squishy 'cause you've, like, s- (laughs) churned all these-

Dwarkesh Patel 32:59 Right.

Jeff Dean 32:59 ... turk tokens together. And so, um, the model has, like, a reasonably clear view of that data but it sometimes, like, gets confused and will give the wrong date for something.

Dwarkesh Patel 33:10 Right.

Jeff Dean 33:10 Um, whereas information in the context window, uh, in the input of the model is, like, really sharp and clear-

Dwarkesh Patel 33:16 Yeah.

Jeff Dean 33:16 ... because we have this really nice attention mechanism in transformers that the model can pay attention to things-

Dwarkesh Patel 33:21 Yeah.

Jeff Dean 33:21 ... and it knows kind of the exact text or the exact-

Dwarkesh Patel 33:25 Right.

Jeff Dean 33:25 ... frames of the video or audio or whatever that it's processing. Um, and so right now we have a, uh, uh, models that can deal with kind of millions of tokens of context, uh, which is quite a lot. It's like, you know, hundreds of pages of a PDF or-

Dwarkesh Patel 33:40 Right.

Jeff Dean 33:40 ... you know, 50 research papers or, you know, hours of video or tens of hours of audio, or some-

Dwarkesh Patel 33:47 Yeah.

Jeff Dean 33:47 ... combination of those things, which is pretty cool. Um, but it would be really nice if the model could attend to trillions of tokens, right? Could it attend to the entire internet and find the right stuff for you?

Dwarkesh Patel 33:59 Yeah.

Jeff Dean 33:59 Could it attend to all your personal information for you, right?

Dwarkesh Patel 34:03 Yeah.

Jeff Dean 34:03 Like, uh, I would love a model that has access to all my emails and all my documents and all my photos, and I'm, when I ask it to do something it can sort of make use of that, uh-

Dwarkesh Patel 34:14 Right.

Jeff Dean 34:14 ... with my permission to, to sort of help solve what-

Dwarkesh Patel 34:17 Yeah.

Jeff Dean 34:17 ... it is I'm wanting it to do. Uh, but that's gonna be a big computational challenge 'cause the, the naive attention a- algorithm is, is quadratic and you can kind of barely make it work on a fair bit of hardware for millions of tokens but there's no hope of making that just naively go to trillions of tokens. So, we need a whole bunch of in- interesting algorithmic approximations to what you would really want, uh, to make, uh, a way for the model to attend kind of conceptually to, you know, lots and lots of more tokens, trillions of tokens, and attend to your tokens. You know, maybe we can put all of the Google codebase in context for every Google developer, uh, all the world's source code in context for any open source developer. That would be amazing.

Noam Shazeer 35:00 It would be, it would be incredible.

Dwarkesh Patel 35:02 (laughs)

Noam Shazeer 35:02 Yeah. I, I mean, right. The, the, the, yeah, the beautiful thing about, you know, model parameters is they are quite memory efficient at, you know, sort of memorizing facts.

Dwarkesh Patel 35:15 Right.

Noam Shazeer 35:15 Maybe, you know, you can probably memorize order of one, one fact or something per model parameter, whereas, you know, if it, you have some token in context there are, like, lots of key, of keys and values at every layer. It could be, it could be a kilobyte, a megabyte of, uh, of, uh, of memory per, per token that, that you have in-

Jeff Dean 35:36 Yeah, you take a word and you blow it up to 10 kilobytes or something. (laughs)

Noam Shazeer 35:38 Yes, yes. It's... Yeah. So, I mean there, so there are some... There, there, uh, there's actually a lot of innovation going on around, okay, A, how do you minimize that, and B, okay, what, what words do you need to have there or are there better ways of accessing bits of that, bits of that information? And, you know, Jeff seems like the right person-

Jeff Dean 35:59 (laughs)

Noam Shazeer 36:00 ... to figure this out. Like, okay, what are, what does our memory hierarchy look like, uh, you know, from the, uh, you know, well, as RAM all the way up to data center, uh-

Dwarkesh Patel 36:11 Yeah.

Noam Shazeer 36:11 ... worldwide level.

Noam Shazeer 36:12 Uh-

Dwarkesh Patel 36:12 I, I, I wanna talk more about the thing you mentioned about, look, you know, Google is a company with, like, lots of code and lots of examples, right? Um, if you just think about that one use case and what that implies, so you've got, like, the Google mono repo, um, and if you, maybe you figured out the long context thing, you could put the whole thing in context or you fine-tune on it. Um, uh, yeah. Basically, like, why, why, why don't, why hasn't this been already done and, you know, because you can imagine, like...... in- the amount of code that Google has, uh, proprietary access to, just, like, ma- e- even if you're just using it eternally for-

Jeff Dean 36:47 Yeah.

Dwarkesh Patel 36:47 ... it to make your developers more efficient and productive.

Jeff Dean 36:49 Oh, to be clear, we ha- we have actually already, uh, f- done further training on a Gemini model on our internal code base for our internal developers.

Dwarkesh Patel 36:58 Yeah.

Jeff Dean 36:59 But that's different than attending to all of it-

Dwarkesh Patel 37:01 Right.

Jeff Dean 37:01 ... uh, because it sort of stirs together the code base into a bunch of parameters.

Dwarkesh Patel 37:05 Mm-hmm.

Jeff Dean 37:06 Uh, and I think having it in context i- is, is, uh, he- makes things clearer. But even the f- the sort of further trained model internally is incredibly useful. Like, um, Sundar, I think, has said that 25% of the characters that we're checking into our code base these days are generated by our AI-based-

Dwarkesh Patel 37:24 Right.

Jeff Dean 37:24 ... coding models, uh, with, with kind of human, kind of-

Dwarkesh Patel 37:28 H- how do you imagine-

Jeff Dean 37:29 ... driving.

Dwarkesh Patel 37:29 ... in a year or two, based on the capabilities you see are on the horizon, your own personal work, what will it be like to be a researcher at Google? You have a new idea or something, uh, w- with the way in which you're interacting with these models in a year, what does that look like?

Jeff Dean 37:44 (laughs)

Noam Shazeer 37:44 Well, I mean, I assume the, we will be, we will have these models a lot better and, uh, (clears throat) hopefully be able to be much, much more productive.

Dwarkesh Patel 37:53 (laughs)

Jeff Dean 37:54 (laughs) Uh- Yeah. I mean, I think one, one of the f- i- in addition to kind of research-y context, like, any time you're seeing these models used, I think they're able to make software developers more productive 'cause they can kind of take sort of a high level sp- spec or in-sentence description of what you want done and give a pretty approximate, you know, pretty reasonable first cut at that. Um, and so from a research perspective, maybe you can say, "I'd really like, uh, you to explore, you know, this kind of idea, like, similar to the one in this paper, but maybe, like, let's try making it convolutional or something." Like, that, if you could do that and have the system automatically sort of generate a bunch of experimental code and maybe you look at it and you're like, "Yeah, that looks good. Run that," um, like, that seems like a nice dream direction-

Dwarkesh Patel 38:43 Yeah.

Jeff Dean 38:43 ... to go in and seems plausible in the next year or two years that you might make a lot of progress on that. And-

Dwarkesh Patel 38:48 Seems under-hyped 'cause you've got, like, ins- y- you could have, like-

Jeff Dean 38:51 (laughs)

Dwarkesh Patel 38:51 ... literally millions of extra employees, um, and you can immediately check their output, but employees can check either- each other's output. They, like, im- immediately stream tokens. You could say something-

Jeff Dean 39:01 Yeah. Sorry, I didn't mean to under-hype it.

Dwarkesh Patel 39:02 Yeah.

Jeff Dean 39:03 I think it's super exciting. (laughs)

Dwarkesh Patel 39:04 (laughs)

Noam Shazeer 39:04 (laughs) Uh, yeah, I'm with you.

Jeff Dean 39:06 I just don't like to hype things that aren't done yet. (laughs)

Dwarkesh Patel 39:11 (laughs)

Noam Shazeer 39:11 (laughs)

Dwarkesh Patel 39:11 Um, I w- uh, yeah, so l- let's... I, I do wanna play with this idea more 'cause, you know, it, it seems like a big deal, like, yeah, something, like, kind of like an autonomous software engineer, especially from the perspective of a researcher who's like, "I want to spec build this system." Um, yeah, and there's... Okay. So you l- legislate with this idea, like, w- um, y- as somebody who has worked on developing transformative systems through your, through your careers, the idea that instead of having to code something like whatever the, today's equivalent of MapReduce is or TensorFlow is, just like, "Here, here's how I want, like, distributed a- a- a AI library to look like. Write it up for me," um, could y- do you imagine you could be, like, 10X more productive, 100X more productive?

Jeff Dean 39:53 I was pretty impressed. I think it was on Reddit that I saw. Like, we have a new experimental coding, like, model that's much better at coding and math and so on. And someone external tried it and they basically prompted it and said, "I'd like you to implement a SQL processing database system with, uh, no external dependencies. And please, please do that in C."

Noam Shazeer 40:18 (laughs)

Jeff Dean 40:18 (laughs) And from what the person said, it actually did a quite good job. Like, it generated a SQL parser and a tokenizer and, you know, a, a, uh, query planning system and some storage format for the data on, on disk-

Dwarkesh Patel 40:31 Mm-hmm.

Jeff Dean 40:31 ... and actually was able to handle simple queries. So, you know, from that prompt, which is like, you know, a paragraph of text or something, to get, you know, even a cer- an initial cut at that seems like a big boost in productivity for software developers.

Dwarkesh Patel 40:48 Yeah.

Jeff Dean 40:48 And I think you might end up with other sy- other kinds of systems that maybe don't try to do that in a single, you know, in semi-interactive respond in 40 second kind of thing, but might go off for 10 minutes and, like, might interrupt you after five minutes saying, "Ah, I've, I've done a lot of this, but now I need to, you know, get some input. You know, do you, do you care about handling video or just images or something?" Um, and that, that seems like you'll need ways of managing the workflow if you have a lot of these kind of e- uh, m- background activities happening.

Dwarkesh Patel 41:23 Yeah. Actually, can you talk more about that? So what interface do you imagine we might need if we have, um, uh... if you could l- literally have, like, millions of employees you could spin up, hundreds of thousands of employees you could spin up on command who are able to type incredibly fast and who, um... So it, it's almost like you go from, like, 1930s, like, trading of, like, tickets or something to now modern, like, you know, w- chain suit or something. You know, like, you, you need a be- you need some interface to keep track of all this that's going on for the AIs to integrate into this big mono repo, um, and leverage their own, like, uh, strengths, um, for humans to keep track of what's happening. Wh- what... Basically, wh- wh- what is it like to be, uh, Jeff or Noam in three years working day to day?

Noam Shazeer 42:05 It might be kind of similar to what we have now 'cause we all- already have, uh, sort of parallelization as, uh, as a major issue 'cause, you know, we, we have, like, lots and lots of really, really brilliant machine learning researchers and we want them to work, all work together and, uh, a- and, and build AI. Um, you know, so actually the parallelization among people might be similar to-

Dwarkesh Patel 42:27 Mm-hmm.

Noam Shazeer 42:28 ... parallelization (laughs) among, uh, a- among, uh, machines. Um, but, um, I think they're... definitely it should be good for things that require, like, a lot of exploration, you know?

Dwarkesh Patel 42:41 Mm-hmm.

Noam Shazeer 42:42 Like, uh, come up with-

Dwarkesh Patel 42:42 (clears throat)

Noam Shazeer 42:42 ... come up with the next, uh, breakthrough because, you know, uh...... you, if you have a brilliant idea that it's just certain to work, uh, in the, uh, ML domain, then you know, it has a 2% chance of working if you're brilliant and, you know-

Dwarkesh Patel 42:58 Yeah.

Noam Shazeer 42:58 ... most- mostly these, these things, uh, fail, but if you try 100 things or 1,000 things or a million things, then, then you might hit on something, uh, something amazing. Um, and we have, we have plenty of compute, like modern la- you know, top labs these days have probably a million times as much compute as it took to train a transformer. So,

Noam Shazeer 43:20 uh ...

Dwarkesh Patel 43:20 Yeah. A- actually, so that's a really interesting idea. If you have, um, like suppose in the world today, there was li- on the order of 10,000 AI researchers, and this-

Noam Shazeer 43:29 Mm-hmm.

Dwarkesh Patel 43:29 ... community coming up with a breakthrough every-

Jeff Dean 43:31 Probably more than that. There were 15,000 at NeurIPS last year. (laughs)

Noam Shazeer 43:34 (laughs) Oh, wow. (laughs)

Dwarkesh Patel 43:37 100,000? I don't know. Um ...

Jeff Dean 43:37 Yeah, maybe. (laughs)

Dwarkesh Patel 43:39 Um ...

Jeff Dean 43:39 Sorry. (laughs)

Dwarkesh Patel 43:39 No, no. It's good to have (laughs) the correct order of magnitude. Um, and the odds that this community every year comes up with a breakthrough on the scale of a transformer is, let's say, 10%. Um, now suppose this community is 1,000 times bigger, and it is in some sense like this sort of parallel search of better architectures-

Noam Shazeer 43:58 Yeah.

Dwarkesh Patel 43:58 ... better techniques. Do we just, like, get, like, Transformer-sized-

Noam Shazeer 44:02 A breakthrough a day? Yeah.

Dwarkesh Patel 44:02 ... breakthroughs every year or every day?

Noam Shazeer 44:04 Maybe. Sounds-

Jeff Dean 44:06 Maybe.

Noam Shazeer 44:06 ... uh, sounds potentially good, you know?

Dwarkesh Patel 44:08 (laughs)

Noam Shazeer 44:09 (laughs)

Dwarkesh Patel 44:09 But exactly like what ML research is like is just if you have, uh, if you are able to try all these experiments?

Noam Shazeer 44:16 It's a good question, 'cause we, you know, I th- I don't know that folks have been, uh, haven't been doing that as much. Um, I mean, we definitely have lots of, lots of great ideas coming along. Everyone seems to want to run their experiment at maximum scale, but I think that's, (laughs) you know, that's a human problem. (laughs)

Dwarkesh Patel 44:35 (laughs)

Jeff Dean 44:35 Yeah, yeah. It's very helpful to have a 1/1000th scale problem and then vet, like, 100,000 ideas on that and then scale up the ones that are, that seem promising.

Dwarkesh Patel 44:45 Yeah. A quick word from our sponsor, Scale AI. Publicly available data is running out, so major labs like Meta and Google DeepMind and OpenAI all partner with Scale to push the boundaries of what's possible. Through Scale's Data Foundry, major labs get access to high-quality data to fuel post-training, including advanced reasoning capabilities. As AI races forward, we must also strengthen human sovereignty. Scale's research team, SEAL, provides practical AI safety frameworks, evaluates frontier AI system safety via public leaderboards, and creates foundations for integrating advanced AI into society. Most recently, in collaboration with the Center for AI Safety, Scale published Humanity's Last Exam, a groundbreaking new AI benchmark for evaluating AI systems' expert level knowledge and reasoning across a wide range of fields. If you're an AI researcher or engineer and you want to learn more about how Scale's Data Foundry and research team can help you go beyond the current frontier of capabilities, go to scale.com/dwarkesh. All right, back to Jeff and Noam. So, I think one thing the world might not be taking seriously, um, b- people are aware that it's exponentially harder to make, um, like, to, to do the scale, like make a model that's 100X bigger takes 100X more compute, right? So it's like people are aware that it's like an exponentially, uh, harder problem to go from Gemini 2 to 3 or so forth. But maybe people aren't aware of this other trend where Gemini 3 is coming up with all these different architectural ideas and trying them out, and you see what works, and you're constantly coming up with these algorithmic progress, uh, that makes training the next one easier and easier. Um-

Jeff Dean 46:24 Yeah, I mean-

Dwarkesh Patel 46:25 How far could you take that feedback loop?

Jeff Dean 46:26 I mean, I think one, one thing people should be aware of is the improvements from generation to generation of these models often are partially driven by hardware and larger scale, but equally, and perhaps even more so, driven by major algorithmic improvements and major changes in the model architecture and the training data mix and so on, that really make the model better per, per flop that is applied to the model. Uh, so I think that's a good realization. And then I think if we have automated exploration of ideas, we'll be able to vet a lot more ideas and bring them into kind of the actual, you know, production, uh, training for next generations of these models.

Dwarkesh Patel 47:06 Mm-hmm.

Jeff Dean 47:06 And that's gonna be really helpful, 'cause that's sort of what we're currently doing with a lot of machine learning research, brilliant machine learning researchers, is looking at lots of ideas, you know, winnowing ones that seem to, to work well at small scale, seeing if they work well at medium scale, bringing them into larger scale experiments, and then like settling on, like, adding a whole bunch of new and interesting things to the, to the final model recipe. Um, and then I think if we can do that, you know, 100 times faster through, uh, those machine learning researchers just gently steering a more automated search-

Noam Shazeer 47:42 Right.

Jeff Dean 47:42 ... process rather than sort of hand-babysitting lots of experiments themselves, that's gonna be really, really good.

Noam Shazeer 47:46 Yeah. The one thing it doesn't speed up is, like, experiments at the largest scale, 'cause you, uh, you still end up doing, like, these n=1 experiments, and they're really just, try to put a bunch of really brilliant people in the room and have them stare at, and-

Dwarkesh Patel 48:01 (laughs)

Noam Shazeer 48:01 ... stare at the thing-

Jeff Dean 48:02 Yeah.

Noam Shazeer 48:02 ... figure out wh- why this is working, why this is not working.

Jeff Dean 48:05 For that, more hardware is a good solution, and better hardware, and-

Noam Shazeer 48:08 Yes, yes, we're counting on you. (laughs)

Jeff Dean 48:11 Yes. (laughs)

Dwarkesh Patel 48:12 So, um, okay, th- and, uh, naively, I would s- uh, so there's a software, um, there, there's this like algorithmic side improvement that future AIs can make.

Jeff Dean 48:22 Mm-hmm.

Dwarkesh Patel 48:22 There's also, um, the stuff you're working on, on Offal Chip. I'll let you describe it, but if you get into a situation where just from a software level, you can be making better and better chips er- uh, in a matter of weeks and months, um, and better AIs can presumably do that better, basically I'm wondering, how does this feedback loop not just end up in like, uh, Gemini 3 takes, uh, two years, then Gemini 4 is like a six, or the equivalent level jump is, uh, now six months, then like the level five is like three months, then, uh, one month, and you get to, like, superhuman intelligence pre- much more rapidly than you might naively think because of this-

49:03 ... uh, software, uh, both on the hardware side and from the algorithmic side improvements.

Jeff Dean 49:09 Yeah. I mean, I've-

Jeff Dean 49:10 I've been pretty excited lately about how could we dramatically speed up the chip design process.

Dwarkesh Patel 49:15 Yeah.

Jeff Dean 49:15 'Cause as we were talking earlier, the, you know, the current way in which you design a chip takes you roughly, you know, 18 months to go from, "We should build a chip," to something that you then hand over to TSMC and- and then TSMC takes, you know, four, four months to- to fab it and then you get it back and you put it in your data centers. Um, so that's a pretty lengthy cycle. Uh, and the fab time in there is a pretty, uh, you know, small portion of it today. But if you could make that the dominant portion so that instead of taking, you know, 18 month- 12 to 18 months to design the chip, you could shrin- And with, you know, 100, 150 people, you could shrink that to, you know, a few people with a much more automated search process exploring the whole design space of- of chips and getting feedback from all aspects of the chip design process for the kind of choices that the system is trying to explore at the high level.

Dwarkesh Patel 50:11 Yeah.

Jeff Dean 50:11 Uh, then I think you could get, uh, you know, perhaps much more exploration and more rapid, uh, design of something that you actually want to give, uh, to a fab. And- and that would be great 'cause you can shrink the- that time, you can shrink the deployment time, uh, by kind of designing the hardware in the right way so that you just get the chips back and you just plug them in to some, uh, some system. Um, and that will then, I think, enable a lot more specialization. It will enable a shorter timeframe for th- the hardware design so that you don't have to look out quite as far into what kind of ML algorithms-

Dwarkesh Patel 50:45 Yeah.

Jeff Dean 50:45 ... would be interesting. Instead it's like you're looking at, you know, six to nine months from now-

Dwarkesh Patel 50:50 Yeah.

Jeff Dean 50:51 ... uh, what should it be, uh, rather than, you know, two, two and a half years. And that would be pretty cool. I do think that that fab- fabrication time is, if that's in your inner loop of improvement, you're gonna, like-

Dwarkesh Patel 51:02 How long is it?

Jeff Dean 51:03 The- the leading edge nodes unfortunately are taking longer and longer 'cause they have more metal layers-

Dwarkesh Patel 51:08 Yeah.

Jeff Dean 51:08 ... uh, than previous, you know, older nodes. So that- that tends to make- make it take anywhere from three to five months.

Dwarkesh Patel 51:15 Okay. But that's how long training runs take anyways, right? So you could potentially do both at the same time?

Jeff Dean 51:21 Yeah, potentially.

Dwarkesh Patel 51:22 Okay, so I guess like you can't get sooner than three to five months, but the idea that you could g- get, like... But also, yeah, you're, like, r- rapidly developing new algorithmic ideas-

Jeff Dean 51:30 Mm-hmm.

Dwarkesh Patel 51:30 ... between this time.

Noam Shazeer 51:30 Right. That- that can move fast. It'll- it'll-

Jeff Dean 51:31 That can move fast. That can run on, like, existing chips and explore lots of cool ideas.

Dwarkesh Patel 51:36 Right.

Noam Shazeer 51:36 Yeah.

Dwarkesh Patel 51:37 So does... Isn't that, like, a situation in which you're... Like, I think people-

Jeff Dean 51:41 Yeah.

Dwarkesh Patel 51:41 ... sort of expect, like, ah, there's gonna be a sigmoid. Um, again, this is not a sure thing, but just like is this a possibility, the idea that you have like sort of an explosion of capabilities, uh, very rapidly towards the tail end of human intelligence that, you know, gets like s- m- uh, smarter and smarter at a more and more rapid rate? Um...

Noam Shazeer 52:00 Quite possibly. Yeah, yeah.

Jeff Dean 52:01 Yeah. I mean, I th- I like to think of it like this, right? Like right now we have models that can take a pretty complicated problem and can break it down, you know, internally in the model into a bunch of steps, can sort of p- puzzle together the solutions for those steps, and can often give you a solution to the entire problem that you're asking it. But it, you know, isn't super reliable and it's good at breaking things down into, you know, five to 10 steps, not 100 to 1,000 steps. So if you could go from, yeah, 80% of the time it can give you a perfect- perfect answer to something that's 10 steps long to something that, you know, 90% of the time can give you a perfect answer to something that's 100 to 1,000 steps-

Dwarkesh Patel 52:45 Yeah.

Jeff Dean 52:45 ... of sub- subproblem long, that would be an amazing improvement in capability of these models. And, you know, we're not there yet, but I think that's what we're aspirationally trying to get to is-

Noam Shazeer 52:56 Right. Yeah, we don't need new hardware for that. We-

Dwarkesh Patel 52:59 (laughs)

Noam Shazeer 52:59 But I mean, we- we'll- we'll take it, but...

Jeff Dean 53:00 Yeah. (laughs)

Dwarkesh Patel 53:01 (laughs)

Noam Shazeer 53:01 (laughs)

Jeff Dean 53:01 (laughs) Yeah, exactly.

Noam Shazeer 53:02 Um...

Jeff Dean 53:03 Never look new hardware in the mouth.

Dwarkesh Patel 53:06 One of the,

Dwarkesh Patel 53:07 uh, you kn- Like, one- one of the big areas of improvement I think, uh, you know, in the near future is- is this, uh, inference time compute, like applying more compute, you know, a- a- at inference time. And I guess the way I've- I've liked to describe it is that, um, you know, a, uh, like even some giant, uh, language model, you know, if- even if you're doing say a trillion operations per token, which is, you know, more than- more than most people are- are doing these days, um, you know, operations cost something like 10 to the negative $18 and so you're getting like-

Jeff Dean 53:45 (laughs)

Dwarkesh Patel 53:45 (laughs)

Noam Shazeer 53:45 ... a million tokens to the dollar, right? So I mean, compare that to like a relatively cheap pastime. Like you- you- you go out and you buy a paper book and read it, you're paying like 10,000 tokens to the dollar. So it's... So like talking to a language model could be like... You know, is- is- is like a hundred times cheaper than reading a paperback. So, um, there is a huge amount of headroom there to say, "Okay, if we can make this thing more expensive, uh, but-"

Dwarkesh Patel 54:14 Ah, right.

Noam Shazeer 54:14 "... but smarter 'cause we're like two- you know, like 100X cheaper than reading a paperback, we're like 10,000 times cheaper than like talking to a customer support agent, we're like a million times or more cheaper than- than, you know, hiring a software engineer or talking to your doctor or lawyer, like can we add, you know, the-"

Dwarkesh Patel 54:33 Yeah.

Noam Shazeer 54:33 "... a- a- add computation and make it- make it smarter?" So like I think a lot of- a lot of the takeoff that we're go- that we're going to see in- in the very near future is of this form. Like we've- we've ex- been exploiting and improving, uh, pre-training a lot in the past and post-training. And those things will continue to improve, but like taking advantage of, you know, think harder at, uh, at inference time is going to just be a- a- an explosion.

Jeff Dean 55:04 Yeah, and an aspect of- of inference time is I think you want the system to a- be actively exploring a bunch of different potential solutions. You know, maybe it does some searches on its own and gets some information back and like consumes that information and figures out, "Oh, now I would really like to know more about this thing," so now it kind of iteratively kind of explores how to best solve the high level-... problem you pose to the system. And I think having a dial where you can make the model give you better answers with more inference time compute seems like we have a bunch of techniques now that seem like they can kind of, kind of do that, and the more you crank up the dial, the more it costs you in terms of compute, but the better the answers get. That, that seems like a, a nice trade-off to have, 'cause sometimes you wanna think really hard 'cause there's a super important problem. Sometimes you probably don't wanna spend-

Dwarkesh Patel 55:58 Right.

Jeff Dean 55:58 ... enormous amounts of compute to compute, you know, one plus, what's the answer to one plus one? (laughs) Um, maybe the system should decide to us-

Dwarkesh Patel 56:05 You, you take that to 100 and it comes up with, like, new axioms of set theory or something. (laughs)

Jeff Dean 56:08 Should decide to use the calculator tool or something instead of, (laughs) you know, a very large language model.

Dwarkesh Patel 56:13 Uh, i- i- are there any impediments to taking inference time, like, uh, ha- having some way in which we can just linearly scale up inference time compute? Or is this a, basically a problem that's sort of solved and we know how to s- throw, like, 100X compute or 1,000X compute and get correspondingly better results?

Jeff Dean 56:32 Yeah.

Noam Shazeer 56:32 Well, we're, we're working out the algorithms as we speak-

Dwarkesh Patel 56:35 (laughs)

Noam Shazeer 56:35 ... so I, I, I, I, I believe, you know, we'll, we'll see, we'll see better and better solutions to this as these many more than 10,000 researchers (laughs) are-

Dwarkesh Patel 56:46 (laughs)

Noam Shazeer 56:46 ... are, are hacking at it, uh, many of them-

Jeff Dean 56:48 Yeah.

Noam Shazeer 56:48 ... at Google.

Jeff Dean 56:48 I mean, I think we do see some examples in our own sort of experimental work of things where you, if you apply more inference time compute, the answers are better than if you just apply, you know, a, you know, X, you know? If you apply 10X, you can get better answers than X amount of in- computed inference time, and that seems useful and important. Um, but I think what we would like is when you apply 10X to get, you know, even a s- bigger improvement in the quality of the answers, uh, than we're getting today. And so that's about, you know, designing new algorithms, trying new approaches, you know, figuring out how best to spend that 10X instead of X to, to improve things.

Dwarkesh Patel 57:27 Does it look more about... Look more like search or does it look more like just keeping going in that linear direction for a longer time?

Jeff Dean 57:32 I mean, I think search is l- I, I, I really like Rich Sutton's paper that he wrote about the-

Dwarkesh Patel 57:38 Mm-hmm.

Jeff Dean 57:38 ... the bitter lesson, and the bitter lesson effectively is this nice one-page paper. But the, the essence of it is, um, you can try lots of approaches, but the, the two techniques that are incredibly effective are learning and search. (laughs) And you can apply and scale those algorithmic or, you know, computationally, and you often will then get better results than any other kind of approach you can apply to, to a pretty broad variety of problems.

Dwarkesh Patel 58:04 Yeah.

Jeff Dean 58:05 Um, and so I think search has gotta be part of the solution to spending more inference time, is you wanna maybe explore a few different ways of solving this problem and, like, "Oh, that one didn't work, but this one worked better, so now I'm gonna explore that a bit more."

Dwarkesh Patel 58:18 How does this change your plans for future data center plan- you know, planning and so forth, where if, um, you know, c- can this kind of search be done asynchronously? Does it have to be online, offline? How does that change how, how big of a, a campus you need and those kinds of considerations?

Jeff Dean 58:38 Um, I mean, I think one general trend is it's clear that inference time compute, you know, where you have a model that's pretty much already trained and you wanna do inference on it, is going to be a growing and important class of computation, uh, that maybe you wanna specialize hardware more around that. Um, you know, actually the, the first TPU was specialized, uh, for inference-

Dwarkesh Patel 59:02 Right.

Jeff Dean 59:02 ... and wasn't really designed for training, and then subsequent TPUs were really designed more around training and also for, uh, inference. Um, but it may be that, you know, when you have, uh, something where you really wanna crank up the, the amount of compute you use at inference time-

Dwarkesh Patel 59:17 Yeah.

Jeff Dean 59:17 ... that even more specialized solutions will make a lot of sense.

Dwarkesh Patel 59:20 Hmm. Does that mean you can accommodate more a- asynchronous training?

Jeff Dean 59:23 Uh, training or inference?

Dwarkesh Patel 59:24 Uh, or j- just you can have e- the different data centers don't need to talk to each other. You can just, like, have them do a bunch of, uh...

Jeff Dean 59:31 Oh. Yeah. I mean, I think, uh, I, I like to think of it as, is the inference that you're trying to do latency-sensitive-

Dwarkesh Patel 59:38 Yeah.

Jeff Dean 59:38 ... like, the user's actively waiting for it?

Dwarkesh Patel 59:40 Right.

Jeff Dean 59:41 Or is it kind of a background thing? And maybe that's, uh, I have some inference tasks that I'm trying to, to run over a whole batch of data, but it's not for a particular user. It's just I wanna-

Dwarkesh Patel 59:51 Yeah.

Jeff Dean 59:51 ... you know, run inference on it-

Dwarkesh Patel 59:52 Right.

Jeff Dean 59:52 ... and extract some information. And then there's probably a bunch of things that we don't really have very much of right now, but you're seeing inklings of it in our deep research, like-

Dwarkesh Patel 1:00:02 Mm-hmm.

Jeff Dean 1:00:02 ... tool that we just, uh, released, I forget exactly when, like, a week ago, um, where you can give it a pretty complicated high-level task, like, "Hey, can you go off and research the history of renewable energy and all the trends and costs for-"

Dwarkesh Patel 1:00:15 Yeah.

Jeff Dean 1:00:15 "... for wind and solar and other kinds of techniques and put it in a table and give me a full eight-page report?" And it will come back with an eight-page report with, like, 50-

Dwarkesh Patel 1:00:24 Yeah.

Jeff Dean 1:00:24 ... entries in the bibliography. (laughs) It's, it's pretty, pretty remarkable. But you're not actively waiting for that e- for one second. It takes, like, you know, a minute or two to go, go do that.

Dwarkesh Patel 1:00:33 Yeah. Yeah.

Jeff Dean 1:00:34 And I think there's gonna be a fair bit of that kind of compute, and that's the kind of thing where you have, uh, some UI questions around, okay, if you're gonna have a user with 20 of these kind of asynchronous tasks in the background happening and maybe e- each one of them needs to, like, get from our, more information from the user, like, "I found your flights to Berlin, but, uh, there's no non-stop ones. Is, d- are you okay with a, you know, a non-stop one?" Um, how does that flow work when you kind of need a bit more information and then you wanna put it back in the background for it to continue doing, you know, finding the hotels in Berlin or whatever? Um, I think it's gonna be pretty interesting, uh, and inference will be useful.

Noam Shazeer 1:01:15 Inference will be useful. I mean, there's also a, uh, compute efficiency thing in inference that you don't have in training in that, uh, you know, in general, transformers can use the sequence length as a batch during training-... but they can't really in inference because you're, uh, when you're generating one token at a time.

Jeff Dean 1:01:32 Yeah.

Noam Shazeer 1:01:33 So, so there, uh, the, uh, there, there may be different hardware and, uh, inference algorithms that, that we design for the purposes of being efficient at the inference.

Jeff Dean 1:01:43 Yeah, like, as a good example, uh, out of an algorithmic improvement, uh, is, like, the use of, uh, drafter models. So you have, like, a really small language model that you do one s- token at a time when you're decoding and predict, like, four tokens. (laughs) And then you give that to the big model, and you say, "Okay, here's the four tokens the, the little model came up with. Check which ones y- you agree with, and if you agree with the first three, then you just advance." And then you've basically been able to do a four, four token with parallel computation instead of a one token with-

Noam Shazeer 1:02:17 Right.

Jeff Dean 1:02:17 ... thing in the big model. Um, and so th- those are the kinds of things that people are looking at to improve inference efficiency, um, so you're not... don't have this single token decode bottleneck.

Noam Shazeer 1:02:27 Right. Basically, the big model's being used as a verifier-

Jeff Dean 1:02:30 Right, can you verify, yeah.

Noam Shazeer 1:02:31 ... uh, as opposed to as, as a generator in verification you can do-

Jeff Dean 1:02:34 Hello, how are you?

Noam Shazeer 1:02:35 ... in parallel.

Jeff Dean 1:02:35 That sounds great to me. I'm gonna, like, advance past that.

Dwarkesh Patel 1:02:38 S- so a, a big discussion has been about, you know, you're, uh, we're already tapping out, like, nuclear power plants in terms of de- delivering power into one single campus, and so do we have to, like, have just, like even more... like, two gigawatts in one place, uh, five gigawatts in one place, or can it be more distributed and still be able to train a model? Um, does this new regime of inference scaling make different considerations there plausible, or how are you thinking about multi-data center training now?

Jeff Dean 1:03:12 Uh, I mean, we're already doing it, so-

Dwarkesh Patel 1:03:13 Yeah.

Jeff Dean 1:03:14 ... we're, we're, we're pro multi-data center training.

Dwarkesh Patel 1:03:16 (laughs)

Jeff Dean 1:03:16 I think, uh, I think in the Gemini 1.5 tech report we said we use multiple metro areas-

Dwarkesh Patel 1:03:21 Mm.

Jeff Dean 1:03:21 ... uh, and trained with some of the compute in each place, and then a, you know, pretty, uh, long latency but high bandwidth connection between those data centers, and that works fine.

Dwarkesh Patel 1:03:32 Yeah.

Jeff Dean 1:03:32 You know, it, it's great. Actually, training is kind of interesting 'cause each step in a training process is, you know, usually for a large model is a few seconds or something at least, so the latency of it being, you know, 50 milliseconds away doesn't matter that much.

Dwarkesh Patel 1:03:46 Mm.

Jeff Dean 1:03:46 Um-

Noam Shazeer 1:03:47 Just the bandwidth, you know?

Jeff Dean 1:03:48 Yeah, just bandwidth.

Noam Shazeer 1:03:49 As long as you can sync, uh, you know, sync all of the parameters of the model across the different data centers and then accumulate all the gradients. So it's, uh, in the time it takes to do one step, you're, you're, you're pretty good.

Jeff Dean 1:04:01 Yeah, and then we have a bunch of work on in, you know, in even early brain days when we were using CPU machines and they were really slow, uh, so needed to do asynchronous training to help scale-

Dwarkesh Patel 1:04:13 Yeah.

Jeff Dean 1:04:13 ... uh, where each copy of the model would kind of do some local-

Dwarkesh Patel 1:04:16 Right.

Jeff Dean 1:04:17 ... computation and then send gradient updates to a centralized system, and then apply them, uh, asynchronously, and another copy of the model would be doing the same thing. Uh, you know, it makes your, your model parameters kinda wiggle around a bit, and it makes people uncomfortable with the theoretical guarantees, but it actually seems to work-

Dwarkesh Patel 1:04:34 (laughs)

Noam Shazeer 1:04:34 (laughs)

Jeff Dean 1:04:34 ... in practice. And our research has generally-

Noam Shazeer 1:04:35 In practice, it works. It, it was so pleasant-

Jeff Dean 1:04:38 (laughs)

Noam Shazeer 1:04:38 ... to go from async to sync-

Jeff Dean 1:04:40 Yeah.

Noam Shazeer 1:04:40 ... because, because your experiments are now replic-

Jeff Dean 1:04:43 (laughs)

Noam Shazeer 1:04:43 ... replicable, like-

Dwarkesh Patel 1:04:44 Ah.

Noam Shazeer 1:04:44 ... rather than, like every-

Jeff Dean 1:04:45 (laughs)

Noam Shazeer 1:04:45 ... like, your results depend on, like, whether there was, like, a web crawler running on the same machine-

Dwarkesh Patel 1:04:52 (laughs)

Noam Shazeer 1:04:52 (laughs) ... as, like, one of your computers. So I am so much happier running on, like-

Jeff Dean 1:04:58 Oh, sad.

Noam Shazeer 1:04:59 ... TPU pods versus, like-

Jeff Dean 1:04:59 I love asynchrony. It just lets you scale so much more.

Noam Shazeer 1:05:01 ... you know, two iPhones and an Xbox or whatever. (laughs)

Jeff Dean 1:05:05 Yeah, what if we could give you asynchronous but replicatable results?

Noam Shazeer 1:05:09 Ooh.

Jeff Dean 1:05:10 So one way to do that-

Dwarkesh Patel 1:05:11 Yeah, yeah, uh-

Jeff Dean 1:05:11 ... is you effectively-

Dwarkesh Patel 1:05:12 That's good. Yeah.

Jeff Dean 1:05:12 ... record the sequence of operations. So, like, which gradient, uh, update happened and when and on which batch of data. You don't-

Dwarkesh Patel 1:05:21 Mm-hmm.

Jeff Dean 1:05:21 ... necessarily record the actual gradient update, uh, in a log or something, but you could replay that log of operations, uh, so that you get repeatability.

Dwarkesh Patel 1:05:30 Mm-hmm.

Jeff Dean 1:05:31 Then I think you'd be happier.

Dwarkesh Patel 1:05:33 That's interesting. (laughs)

Jeff Dean 1:05:33 Than if you couldn't-

Noam Shazeer 1:05:33 Possibly. At, at, at least you could debug what happened.

Dwarkesh Patel 1:05:36 (laughs)

Jeff Dean 1:05:36 (laughs) Yeah.

Noam Shazeer 1:05:37 But then you, you wouldn't be able to, like, compare two, necessarily two training runs because-

Jeff Dean 1:05:42 Yeah, that's true.

Noam Shazeer 1:05:42 ... okay, I, I made one change in the hyperparameter-

Jeff Dean 1:05:45 (laughs)

Noam Shazeer 1:05:45 ... but also, like, I had, like, a-

Jeff Dean 1:05:48 Web crawler. (laughs)

Noam Shazeer 1:05:49 ... like, crawl, um, machine-

Jeff Dean 1:05:49 (laughs) Web crawler.

Noam Shazeer 1:05:49 (laughing) ... like, messing up. And there were, like, a lot of people streaming-

Jeff Dean 1:05:55 Yeah.

Noam Shazeer 1:05:55 ... the Super Bowl at the same time.

Jeff Dean 1:05:57 Yeah. (laughs)

Dwarkesh Patel 1:05:58 (laughs)

Noam Shazeer 1:05:58 (laughs)

Jeff Dean 1:05:58 I mean, the thing that let us-

Dwarkesh Patel 1:06:00 Non-Super Bowl running.

Jeff Dean 1:06:00 The thing that let us go from asynchronous training on CPUs to fully synchronous training is the fact that we have these super-fast TPU hardware chips and then pods which have incredible amounts of bandwidth-

Dwarkesh Patel 1:06:12 Yeah.

Jeff Dean 1:06:12 ... between the chips and a pod. And then scaling beyond that, we have, like, really good data center networks and even cross-metro area networks-

Dwarkesh Patel 1:06:20 Mm-hmm.

Jeff Dean 1:06:20 ... that enable us to scale to, you know, many, many pods in multiple metro areas for our largest training runs, and we can do that fully synchronously, as Noam said, as long as the gradient accumulation and communication of the parameters across metro areas happens, you know, fast enough relative to the step time, you're golden.

Dwarkesh Patel 1:06:38 Yeah.

Jeff Dean 1:06:39 You, you don't really care. Um, but, uh, I think as you scale up, there may be, uh, um, push to have a bit more asynchrony in our systems than we have now, 'cause, like, we can make it work. I've, uh, been... you know, our ML researchers have been really happy how far we've been able to push synchronous training 'cause it is easier mental model to understand.

Dwarkesh Patel 1:07:01 Yeah.

Jeff Dean 1:07:01 You know, you just have your algorithm sort of fighting you rather than- (laughs)

Dwarkesh Patel 1:07:05 (laughs)

Jeff Dean 1:07:05 ... (laughs) the asynchrony and the algorithm kind of like-

Noam Shazeer 1:07:07 Yeah.

Jeff Dean 1:07:07 ... battling it.

Noam Shazeer 1:07:08 As, as you scale up, there are more things fighting you, you know? Like, there's-

Jeff Dean 1:07:12 Yeah.

Noam Shazeer 1:07:12 I mean, that, the, the pro- right, that's the problem with the, you know, with scaling that you don't actually always know what it is that, that's fighting you. (laughs) Is it, you know, the, the fact that you've pushed, like, quantization a little too far in some place or another, or is it your data, or is it-

Jeff Dean 1:07:28 Maybe it's your adversarial machine MUQQ17 that is like- (laughs)

Noam Shazeer 1:07:32 Oh, no. (laughs)

Jeff Dean 1:07:34 ... setting the seventh bit of your exponent and all your gradients or something.

Noam Shazeer 1:07:37 Right.

Jeff Dean 1:07:37 (laughs)

Noam Shazeer 1:07:37 And all of these things just-... make the model slightly worse, so you don't even know-

Dwarkesh Patel 1:07:43 Mm-hmm.

Noam Shazeer 1:07:43 ... that the thing is going on, um, so, you know-

Dwarkesh Patel 1:07:46 That's actually a bit of a problem with neural nets is, they're so tolerant of noise-

Noam Shazeer 1:07:50 Yeah.

Dwarkesh Patel 1:07:50 ... you can have things set up kind of wrong in a lot of ways, and they just kind of f-figure out ways to work around that or learn and despite that.

Noam Shazeer 1:07:57 Yeah, you could have bugs in your code. Most of the time, that does nothing. Some of the time, it makes your model worse. Some of the time, it makes your model better-

Dwarkesh Patel 1:08:06 (laughs)

Noam Shazeer 1:08:06 ... and then you discover something new because you never tried this bug-

Dwarkesh Patel 1:08:10 Yeah.

Noam Shazeer 1:08:10 ... at scale before because you didn't have, uh, i- it didn't have the budget for it

Noam Shazeer 1:08:15 but, uh...

Dwarkesh Patel 1:08:15 What- what practically does it look like actually to debug or decode what the... Like, you've got these things, some of them which are making the model a lot better, some of which are making it worse. Uh, you, when you go into work tomorrow, you're like, "All right. What's going on here?" How do you figure out what the most salient inputs are?

Noam Shazeer 1:08:32 Right. I mean, well, at- at small scale, you do lots of experiments, so, so I mean-

Dwarkesh Patel 1:08:36 Mm-hmm.

Noam Shazeer 1:08:36 ... there's I think one part of, um, of the research that involves, okay, I want to, like, invent these improvements or breakthroughs kind of in the isolation, in which case, you want a nice simple code base that you can fork and hack and- and, like, some baselines, uh, and my dream is I wake up in the morning, come up with- come up with an idea, hack it up in a day, run some experiments, get- get some initial results in a day, like, "Okay, this look- this looks promising. These things worked, uh, these things worked and didn't work." And- and I think that- that is- that is very achievable because okay-

Dwarkesh Patel 1:09:16 At small scale.

Noam Shazeer 1:09:16 ... at small scale.

Dwarkesh Patel 1:09:17 Right.

Noam Shazeer 1:09:17 As long as you keep your- yeah, you know, keep a nice experimental code base and, uh-

Dwarkesh Patel 1:09:21 Maybe an experiment takes an hour to run or two hours-

Noam Shazeer 1:09:24 Yeah.

Dwarkesh Patel 1:09:24 ... or something.

Noam Shazeer 1:09:25 Yeah.

Dwarkesh Patel 1:09:25 Not- not- not two weeks.

Noam Shazeer 1:09:26 It's great. It's great.

Dwarkesh Patel 1:09:27 Yeah.

Noam Shazeer 1:09:27 So- so there's that part of the research, and then there's su- some amount of scaling up, and then you have the part which is like integrating, where you want to stack all the improvements-

Dwarkesh Patel 1:09:37 Mm-hmm.

Noam Shazeer 1:09:38 ... on top of each other and see if they work in- at large scale and see if they work all in conjunction-

Dwarkesh Patel 1:09:44 Right, how do they interact?

Noam Shazeer 1:09:45 ... with each other.

Dwarkesh Patel 1:09:45 Right. You think maybe they're independent, but actually, maybe there's some funny interaction between-

Noam Shazeer 1:09:50 Mm-hmm.

Dwarkesh Patel 1:09:50 ... you know, improving the- the way in which we handle video data input and the way in which we, you know, uh, update the- the model parameters or something. And- and, you know, that interacts more for video data than some other thing. You know, there's all kinds of interactions that can happen that you maybe, uh, don't anticipate, and so you want to run these experiments where you're then putting a bunch of things together and then periodically making sure that all the things you think are good are good together, uh, and if not, understanding why they're not playing nicely. Um, two questions. One, how often does it end up being the case that things don't stack up well together? Um, is it like a rare thing, or does it happen all the time?

Noam Shazeer 1:10:33 It- it happens, happens all the time.

Dwarkesh Patel 1:10:35 50%? Okay. (laughs)

Noam Shazeer 1:10:36 (laughs)

Dwarkesh Patel 1:10:36 Yeah, I mean, I- I think most things you don't even try to stack because they- they, you know, the initial experiment didn't work that well or showed results that aren't that promising relative to the baseline. And then you sort of take those things and you try to scale them up individually.

Noam Shazeer 1:10:52 Yeah.

Dwarkesh Patel 1:10:52 And then you're like, "Oh, yeah, these ones seem really promising, uh, so I'm gonna now include them in something that I'm gonna now bundle together and try to advance, you know, what the... and combined with other things that seem promising." And then you run the experiments, and then you're like, "Oh, well, they didn't really work that well. Like, let's try to debug why."

Noam Shazeer 1:11:09 And then there are trade-offs 'cause you- you want to keep your, like, integrated system, you know, as clean as you can because, you know, complexity-

Dwarkesh Patel 1:11:18 Code base-wise.

Noam Shazeer 1:11:18 ... code base, yeah, code base and algorithmically-

Dwarkesh Patel 1:11:20 Yeah.

Noam Shazeer 1:11:21 ... complexity, you know, complexity hurts, complexity makes things slower, introduces more risk, and then, you know, at the same time, you want them- you want it to be as- as good as possible and of course, uh, uh, every individual researcher, uh, wants- wants his, uh, inventions to go into it. Um, so there- there are- there are, uh, there- there are definitely, uh, uh, challenges there but, uh, but we've been working together quite well.

Dwarkesh Patel 1:11:45 (laughs) My sponsors Jane Street invented a card game called Figgi in order to teach their new traders the basics of markets and trading. I'm a poker fan, and I'd say that Figgi is like poker in the sense that there's hidden information, but it's much more intense and social. In poker, you're usually just sitting around waiting for your turn, whereas in Figgi, you spend the whole time just shouting bids and asking the other players. The game is set up such that there's a winner in the end, of course, but during each turn, you are incentivized to find mutually beneficial trades with the other players. And in fact, that's the main skill that the game rewards. Figgi simulates the most exciting parts of trading. Jane Streeters enjoy it so much that they hold an inter-office Figgi championship every single year. You can play it yourself by downloading it on the app store, or you can find it on desktop at F-I-G-G-I-E.com. All right, back to Jeff and Noam. Okay,

Dwarkesh Patel 1:12:41 so then going back to the whole dynamic of you find better and better algorithmic improvements, uh, and the models get better and better over time, even if you take the hardware part out of it, should the world be thinking more about... and should you guys be thinking more about this? There's one world where you're just like, AI is a thing that takes, like, two decades to slowly get better over time and you can sort of, like, refine that things are, you know, if, like, you've kind of messed something up, you fix it, uh, and it's, like, not that big a deal, right? It's, like, not that much better than the previous version you released. There's another world where you have this big feedback loop which means that the year... the- the- the two years between Gemini 4 and Gemini 5 are the most important years in human history because you go from, um, uh, a pretty good ML researcher to superhuman intelligence because of this feedback loop. To the extent that you think that second world is plausible, how does that change how you sort of approach these greater and greater levels of intelligence?

Noam Shazeer 1:13:42 I've stopped cleaning my garage 'cause I'm waiting for the robots, you know.

Dwarkesh Patel 1:13:46 (laughs)

Noam Shazeer 1:13:46 (laughs) So probably I'm- I'm- I'm more in- in the second camp of what we're gonna see a lot of acceleration.

Dwarkesh Patel 1:13:53 Yeah, I mean, I think it- it's- it's super important to understand what's going on and what the trends are, and I think right now, the trends are the models are getting substantially better generation over generation. Um, and I don't see that slowing down in the next few generations probably. So that means the models...

Jeff Dean 1:14:10 ... say, two to three generations from now are gonna be capable of, you know, let's go back to the example of breaking down a simple task into 10 sub-pieces and doing it 80% of the time to something that can break down a task, a very high-level task into a hundred or a thousand pieces and get that right 90% of the time, right? That's a major, major step up in what the models are capable of. So I think it's important for people to understand, you know, what's, what is happening in the progress in the field and then those models are gonna be applied in a bunch of different domains. And I think it's really good to make sure that we, as society, get the maximal benefits from what these models can do to improve things in, you know, I'm super excited about areas like education and healthcare, you know, making, uh, information accessible to all people. Um, but we also re- realize that they could be used for misinformation, they could be used for, you know, automated hacking of computer systems. And we wanna sort of put as many safeguards and mitigations and understand the capabilities of the models in, in places we can. And that's kind of, you know, I think Google as a whole has a really, you know, good view to how we should approach this. You know, our responsible AI principles actually are a pretty nice, uh, framework for how to think about trade-offs of making, you know, better and better AI systems available in different conte- contexts and settings, uh, while also sort of making sure that we're doing