Google's AI Infrastructure Chief, Amin Vahdat, on the Physics & Economics of Frontier AI

Amin Vahdat 0:00 In hardware, again, as you know, there is this opportunity where the more you specialize to a particular workload, the less flexible it is, the faster, the more power efficient the hardware's gonna be. So it is this art, and it's this projection of what are you designing to and how persistent is that workload? In other words, if it's gonna go away after a month or two months or three months, even if it's big for those three months, you got a really narrow window to intercept it. So it has to be somewhat durable, uh, and you have to be able to project ex- exactly what win you can get for specializing to it.

Sonya Huang 0:35 [upbeat music] Thrilled to welcome Amin Vahdat to the show. Amin, thank you for joining us for today.

Amin Vahdat 0:56 It's r- really exciting to be here. I'm really looking forward to it.

Sonya Huang 0:59 I am very excited for today's topic because we are in the middle of the biggest CapEx build-out in human history. You are in the middle of it. Google alone is expected to spend more than $200 billion on CapEx this year, most of which is going into building data centers. Uh, you're at the center of it all. You were named the head of Google's, uh, AI Infra at the end of last year. Um, so you are the man spearheading the efforts, uh, of one of the most capital-intensive build-outs in, in human history, and so I am really excited to get into it with you today.

Amin Vahdat 1:30 It's a huge year, uh, for sure. Of course, across the industry, including at Google. Uh, really have... I, I don't think we've ever seen anything like this, honestly. Certainly not at Google, but as you said, uh, I think in the history of humanity in terms of the build-outs and the pace of transformation, really unbelievable.

Sonya Huang 1:47 Before we get into it, maybe just, just crash course for the audience. What is an AI data center, and how, how, how is it different from a non-AI data center?

Amin Vahdat 1:55 It's a very good question, uh, in that, uh, AI data center and a non-AI data center have, do have a lot of similarities actually, and they're, they're not radically different. I mean, it consists of concrete, right? An enclosure. It consists of, uh, electrical yards, uh, mechanical yards, cooling. Uh, you have row after row of, uh, power being distributed. Um, there's a huge amount of network infrastructure there. In other words, we connect large amounts of compute to one another. Uh, a fair amount of storage infrastructure goes in there. I think that the big difference that we're seeing with AI infrastructure is specialization. In the past, when we're building a data center, it really is a 20, 25, 30-year building investments, and we're thinking about how it's going to be evolving over that 20, 25, 30-year period. You might have servers go into it. You might have, uh, s- networking storage. You might have some accelerators, GPUs, TPUs, whatever they might be going into it. But it's a 25, 30-year planning horizon. So we ha- and the lifetime of the hardware might be six years. So many generations that, uh, we have to plan for. An AI data center oftentimes, uh, is going to be more purpose-built. So in other words, we are oftentimes co-designing the building with the hardware that might go into it. We might be saying, "You know what? Actually, we're not gonna put a lot of storage into this building." Why? Because a storage rack might have 10, 20, 30, 40 kilowatts of power. You put that up next to, uh, a TPU rack or a GPU rack that easily is hitting hundreds of kilowatts today, might hit megawatts. People are talking about that in the next few years. Designing a building that might take 30 storage racks in a row versus one or two AI racks in that same row, very, very different design. Just think of it in terms of the size. Think of it in terms of the power and how that would be distributed across, uh, the building, et cetera. Networking would be the same. The amount of networking that you would need for a storage rack, tiny, I mean, if, especially if it's hard drives, compared to an AI rack. So if you're trying to make it fully fungible, you'll probably make it too big and too overbuilt, fungible over certainly a, a 30-year period. An AI data center likely is gonna be much more purpose-built, co-designed with the hardware, even to the point of cooling and power distribution, uh, et cetera. So, uh, a lot more co-optimization.

Sonya Huang 4:27 You all delivered a big Vera Rubin cluster to one of my portfolio companies, Ineffable Intelligence, and I saw the photographs of it as it went out. That is a thing of beauty.

Amin Vahdat 4:38 It, it is. It is.

Sonya Huang 4:39 It's, it's... I mean, th- th- that's one of those things where, you know, similar to looking at the biggest construction projects in human history, you look at that data center and it's like, "Wow, that is a monu- monument to what, what mankind can do."

Amin Vahdat 4:49 No, we got a, a nice picture of it, and it, um, and this is just, uh, you know, a, a couple of racks. You know, the, the cabling, the fiber, the distribution for it is, is really beaut- I mean, you got... Beauty's in the eye of the beholder, but for, uh, folks like me, and probably yourself, and others in the audience as well, um, it is a thing of beauty. We put this picture up on, um, social media, and, um, people love the fact that, uh, Ineffable, why they're a huge fan of, uh, what, what they're doing. Fantastic team. That got a lot of attention. Okay, great rack for Ineffable, but actually people just loved seeing the, the pictures of the, uh, fiber and the sort of-

Sonya Huang 5:20 Totally

Amin Vahdat 5:21 ... the fra- the fractal nature of the fiber. One of the most popular, um, posts that, uh, we, we've ever had actually, uh, was around that. So, so yeah, absolutely. We're really excited about that.

Sonya Huang 5:29 Yeah, I had

Sonya Huang 5:30 goosebumps seeing the picture.

Amin Vahdat 5:31 Mm-hmm.

Sonya Huang 5:32 How do you measure and hold yourselves accountable in this, in the middle of this big build-out? Um, we were talking right before the show about how FLOPS is a vanity metric, and you prefer an alternate metric. Can you, can you say more?

Amin Vahdat 5:42 Yeah. So I think, uh, one thing to, um, note is that, you know, whether it's FLOPS or pick your other favorite, uh, chip-centric, uh, uh, metric, these are in theory. In other words, under some conditions, for whatever chip you have, this is the maximum amount of FLOPS that, uh, you can deliver. What we really care about in the end is what's the performance delivered by workload? And that's, that's what we're looking at. So, and rarely it, it happens, but actually relatively rarely is that performance determined by a single chip. It's FLOPS, it's HBM, it's SRAM, uh, capacity, et cetera. Those are all super important, but it might be how two, four, eight, 16, 1,000, 10,000 more of these chips composed together with not just the accelerators, again, whether TPUs or GPUs, but then the CPUs that might feed them the data, the network that connects them all together. So what is the workload that you're running, and what is the performance of that workload? One measure that is interesting is what's your FLOPS utilization? So for example, for a particular workload, if you have in theory a teraFLOPS or a petaFLOPS capable for your workload, which fraction of that are you delivering? Now, that's a measure of goodput. Exactly.

Sonya Huang 7:00 What is, what is goodput?

Amin Vahdat 7:01 So you can think of throughput, which is a well-known term.

Sonya Huang 7:03 Yep.

Amin Vahdat 7:04 That's the throughput that is possible. But now let's consider some, uh, other considerations. One is what is the slowdown of the workload? Again, just n- uh, inherent to the workload, but other aspects of it that really hits our reliability. So in terms of how we hold ourselves accountable, if we have a chip failing for a synchronous workload, and many of these workloads, whether training or serving or agentic workloads, they're synchronous. There's many, many components that are working together simultaneously. Now, if you have, let's say 1,000, 10,000, 100,000 of these components working together simultaneously, and they're really needing to coordinate, uh, at microsecond or milli- millisecond granularity, one of them fails, it might actually bring the whole thing to a stop, right? Because everyone is counting on everyone else to do their part of the job in order to come up with the answer to a really tough question. One of them stops. Okay, now we, we have to figure out what happened, which one stopped, what's the checkpoint that we have of the computation at some previous state in time? How do we restore that checkpoint? How do we restart? Worst case, we have to restart from s- the beginning. That would be really bad, but that, that can happen in certain cases, especially on, um, on the inference side. The point here is that if you now then have to go back and redo a bunch of computation, if you then have to pause and wait for, to figure out what happened for, for the failure, all that work is work. It's not actually helping you get the answer, right? The... In terms of, let's say that you're working out a problem on paper, step one, step two, step three, step four. If you have to go back to step one because you've, you've made a mistake, yes, you're still doing work. That's throughput. You're doing work, but what is the goodput in delivering your answer? It's basically the total amount of time it took you to solve the problem. That's, that's what matters. And if you have failures, if you have failure recovery, if you have whatever it is that's interrupting your work, that's, that's part of the issue. So now how do we hold ourselves accountable? It's delivered goodput, not theoretical benchmark throughput or goodness in theory, but actually for a workload, for real failure conditions, what's happening? And the unfortunate reality is that at 100,000 accelerator scale-

Sonya Huang 9:16 I was gonna say, at that scale, something's failing all the time, right?

Amin Vahdat 9:18 Something all the time.

Sonya Huang 9:19 Okay. [chuckles]

Amin Vahdat 9:19 And, and these, each one of these, as, uh, you're aware, each one of these chips is a, uh, wonder of nature. I mean, the- they're at the very bleeding edge of what's possible to manufacture, right? And so again, it... To, to me, it's, uh, stunning. And now actually these chips aren't just one chip. They're packages made up of, in many cases, two, four, eight, maybe more chiplets, right, that are composing together. And then of course, HBM off to the side, network connectivity, maybe things like co-packaged optics. No criticism, lots of things can fail. And now if you got 100,000 of them, something is going to fail. You have to be prepared for that, detect that in near real time, recover from that in near real time. Doing all that, it's, it's really like finding... The, the telemetry problem is massive. It's like finding a needle in the haystack continuously across certainly seconds and minutes, but for some of these jobs, hours and days and even weeks, right? Just, uh, continuous and online. How we hold ourselves accountable is what's the goodput that we deliver for the workloads that actually matter in the data center?

Sonya Huang 10:24 Is goodput a Google term or is it an industry term?

Amin Vahdat 10:26 It's a Google term, but I think more and more people across the industry are starting to pick it up.

Sonya Huang 10:31 And then just to calibrate me, 100,000 accelerator scale, are we talking it's gonna fail once a minute, once a, once a day? Like how fr- how frequently are these-

Amin Vahdat 10:39 100,000 accelerators. So let me see. I think that, uh, it is definitely going to be at the scale of multiple times a day, and perhaps, uh, depending on the exact configuration, multiple times an hour, uh, something is gonna fail.

Sonya Huang 10:52 And what are the most common failure reasons?

Amin Vahdat 10:54 Th- this is the problem. Actually, it's a very good question. Uh, if there were a common failure reason, it... We'd be, we'd have figured it out and, and fixed it. It is a long tail of constant discovery. You know, whenever we have a new product that's being introduced, there's gonna be something that hits us. Um, in many cases, uh, frankly, because again, it's at the very bleeding edge, it might be network related. It might be how we connect these things together at, uh, super high speed. It might be something to do with the hardware. Those things we work through. Uh, but then a lot of the issues could be software. So this is the other aspect of how we hold ourselves accountable. Again, the chip might be capable of a certain level of FLOPS, but if you have a compiler bug, a runtime bug, a model issue, something else, operating system issue, it doesn't matter, that, that's going to impact the end-to-end system performance. So you, you might have perfect hardware, fully reliable, but then you might have software issues, uh, that hurt you.

Sonya Huang 11:52 Hmm. Is there a standard reference stack from the accelerator companies, from an NVIDIA or from the TPU team of, you know, here is the optimal system to build around our accelerators, and as long as you build to that system, you're, you're good? Or how much of your own kind of data center design do you have to do above and beyond what the, what the semiconductor companies give you?

Amin Vahdat 12:13 Uh, right. So there's a, there's a reference stack, and I mean, I would say that, uh, NVIDIA is a, uh, incredible, uh, whole systems, uh, company that I, I... I mean, they are obviously a semiconductor company, but they're not just a semiconductor com- company. They give you a really, really strong reference stack. But I would say that, uh, many... We find that, um- The way I'd put it is most of our c- uh, customers leverage that reference stack, but many also specialize it. So in other words, they might find that as, as natural for their particular use case, they have an optimization opportunity or something different that they need to do, and they're gonna do that. Similarly for the TPU side, we have a reference stack, but then, and, and most people will leverage that, many will specialize it as well, though.

Sonya Huang 12:53 Mm. Got it. And then I guess in terms of overall capacity, you've told your teams that Google has to roughly double the serving capacity every six months or so. Is that right?

Amin Vahdat 13:04 Well, so, and to be clear, uh, this is in terms of effective available capacity from... The way in the end that we look at it is from a serving perspective, it's token generation capability. So capacity, and this is where I would say it's a combination of software and hardware. The hardware might have some, quote unquote, "inherent level of FLOPS." What I'm saying here is not necessarily you have to double the number of FLOPS every six months. That's one, one path. You have to double the capability of that hardware to generate tokens every six months, and as much or more of that is gonna come from software as it is from, uh, hardware. So in other words, it could be a model optimization that delivers that. It could be some runtime optimization that, uh, uh, delivers that. It's really probably gonna be dozens or hundreds of individual optimizations that are just landing again and again and again to make it all possible. But yes, the r- the rate of capacity improvement is incredible.

Sonya Huang 14:03 I mean, I g- I guess we've had a few years of data center build-out now, and a few years of model progress and software progress. What has been the empirical breakdown of how much, um, capacity me- maybe as measured by intelligence per watt, how much capacity increase has come from the silicon itself, uh, versus the models, uh, versus f- so- other software and, and any other kind of big components?

Amin Vahdat 14:28 Yeah, so it's a really good question. Um, I don't have the exact breakdown, but I would say that, uh, in our experience, most of the benefits come from model-side improvements in terms of intelligence per watts. And by the way, the intelligence per watt is a fantastic metric. Uh, really focusing... We, we also, for us, it's, um, goodput per watts, and we can come back to why the watts needs to be the denominator. Goodput could well be a measure of the intelligence delivered per watt. In other words, goodput is a workload-specific metric. But I would say most, uh, of the gains often come, uh, come from the model side. Software system, um, quite a bit. Why? Because they're able to actually make sure that the hardware that you have is being used ef- effectively. And then the hardware, it, it is pretty stunning. In other words, we're living in a world right now where 2X or more year-over-year performance improvements is absolutely possible. And so the hardware does, does support it, and it, it is a, if you want, it's a free multiplier. Not free, uh, but, um, everyone else above it can count on it in terms of that being a multiplier that lifts everyone else up, uh, year

Amin Vahdat 15:32 over year.

Sonya Huang 15:33 I'd love to talk a bit about the TPU program and, and co-design. Uh, so Google started building custom silicon more than a decade ago. It was a contrarian call at the time. Um, how has the TPU program evolved?

Amin Vahdat 15:45 Quite a lot. I mean, when the program started 2013, it was really a contrarian call. Uh, and I mean, I think that at the time, it's hard to put yourself back into the moment 2013. Conventional wisdom, you know, all the smartest, wisest people would say, "You don't build a custom-built s- uh, accelerator for s- a single workload." Why? Because Moore's law is there, doubling of performance every, whatever it is, 18 or 24 months is there. Uh, you get to leverage standard programming models, all your C++ code, et cetera, Java, Python, whatever it might be is-

Sonya Huang 16:16 Like the bitter lesson of chips.

Amin Vahdat 16:17 Yes, exactly.

Sonya Huang 16:18 [laughs]

Amin Vahdat 16:18 It's like the bitter lesson of chips is that, um, specialization never wins. But in, in this particular case, it was we have this one application, uh, or, or a few, a small number of them, that would benefit tremendously and that would require an unimaginable amount of general purpose CPU to support. So really, in 2013, it w- it was a bet, and there were quite a few, few people, even within the company, I would say, who were, um, not sure that it would work out. So it was a bet. It turned out to be a massively successful bet. So the first chip was, uh, all, all about inference. Um, the second chip was, hey, actually, we can take the same idea and build a training chip. From, from there, it got picked up for, uh, more and more use cases. Around the time the second chip came out, transformers were invented, and this, this was a major, major moment, and that actually completely shifted the TPU program. Uh, we, of course, learned that recommender systems could run really, really well on the T- uh, TPUs, and so ads and re- related, uh, use cases came along. So I would say the change in evolution has been one of expansion in scope and impact, right? In other words, it started with a really impactful use case of a couple of applications for inference, serving, this was language translation, uh, primarily, and, uh, voice recognition, to then training, to then transformers, recommender systems, and continuing to generalize as, like, I guess the, um, gen AI moment took, uh, hold to ever larger, ever more scalable systems as well.

Sonya Huang 17:55 You mentioned the mo- moment the transformer came through being a ma- a major moment. Uh, I'm curious to explore... The TPU was, like, a specific architecture, but it wasn't, you know, specific to the transformer specific. Um, how do you kind of straddle the fine line of, you know, how specific you want your chip to be for the workload?

Amin Vahdat 18:13 This is a great question, and I think that it really comes down to the applicability, I mean, I think that, um, uh, of the, of the chip to a particular workload. So in other words, in the end, we've been thinking about this for, um, every generation, would we further specialize? The question that we were facing, let's say, um, two years ago or a little bit more, was do, in 2026, should we have two chips? Or one. We could have one chip that could do, uh, inference and training simultaneously and do both quite well, or we could have two chips, one that was further specialized for inference and the second one that was further specialized for training. This analysis and this work led to the release of two chips this year, 8i for inference, 8t for training. And in the end, what we realized is that it wasn't the case necessarily a couple years ago, but by '26, we, we saw inference and serving really taking off. And so having a chip that would be significantly faster for serving that we th- thought might be 30, 40, 50, 60% of the market in its lifetime started making a lot of sense for us. Relative to if, if inference... if we projected inference to be 2% or 5% of the market, even if that specialized chip is, let's say, 2x faster, it might not make sense, right? Because you'd actually go for the one general purpose chip that's not fully optimized, but that- that's okay because you now have the uniformity, et cetera. So really, uh, it does come down to a calculus question of, okay, would you further specialize? How big is that workload? How big is that workload projected to be in, uh, couple, three, four years? Is this a, um, sustained growth for a particular workload? In hardware, again, as you know, there is this opportunity where the more you specialize to a particular workload, the less flexible it is, the faster, the more power efficient the hardware's gonna be. So it is this art, and it's this projection of what are you designing to and how persistent is that workload. In other words, if it's gonna go away after a month or two months or three months, even if it's big for those three months, you got a really narrow window to intercept it. So it has to be somewhat durable, uh, and you have to be able to project exa- exactly what win you can get for specializing to it.

Sonya Huang 20:34 So the rough trade-off is there's a fi- a large fixed cost to support a new program.

Amin Vahdat 20:40 Yes.

Sonya Huang 20:40 And you basically have to think that there's gonna be enough demand for that specific program to justify the cost.

Amin Vahdat 20:46 Exactly. And, uh, to so- to some extent, you know, for example, for our 8i and 8t chips, both chips can do the other workload. This is also key. If 8i could only do inference and it couldn't do training at all, like in other words, it had zero performance for training, and then vice versa, if 8t was amazing at training and had zero performance for inference, that would've also been a key limitation. Why? Because we would have to predict ahead of time over a six-year period, the lifetime of the hardware, exactly how much we need of each. In this case, it was nice because both chips are better at what they're specialized for, but if you needed to, if you had leftover capacity in one place or the other, both can actually do the other, uh, other's job. Depending on the specialization you do, you might get so specialized that you actually limit yourself from being flexible, fungible.

Sonya Huang 21:35 And if the rough trade-off then is kind of size of program on the other end, like it seems to me that so much of the modern AI market is transformer-based, and so maybe just to ask a provocative question, why not just burn the transformer architecture into the chip?

Amin Vahdat 21:48 Yeah. So I think that, uh, it's transformer-based, but then, uh, the next level of question would be, um, transformers in the end are about the vector and m- matrix multiply operations, and, you know, a softmax. So there's a range of essentially linear algebra primitives. We have those, and others do as well, roughly baked into the hardware. It's all transformer-based, but then it's your model architecture is, okay, how many layers do you have? How do you go through the layers? How do you go across the layers? How... For each dimension, um, what exact shape of matrices and vectors are you applying? You, you could go further and specialize to not just the transformer, but your model. That's the next level of, uh, specialization. I think there are a number of companies out there that are, uh, thinking about that. I think it's a very, very interesting direction as well.

Sonya Huang 22:38 Okay. So at this point in time, do your customers view TPU and GPU as roughly fungible equivalents, or is there, is there a certain set of problems that is better suited for one or the other?

Amin Vahdat 22:48 There's f- for sure a better set of problems that are better suited for one or the other. I mean, gen- uh, GPUs are more general purpose than TPUs, uh, for, for one. That, that is, uh, clear. I mean, they're, we, uh, they're incredible products at, uh, Google and Google Cloud. Uh, we s- we sell a lot of, uh, GPUs. We use GPUs internally. But I think it really then comes down to the specifics of your problem. I... So I think that, uh, our customers, there, there is overlap between the two, but our customers then evaluate the workload and evaluate their options. What we like to do at, uh, Google is give our customers choice. In other words, we want to have the right solution for their needs and, of course, provide the solution that, that best meets their needs, uh, for them.

Sonya Huang 23:29 Mm-hmm.

Sonya Huang 23:30 What is the case for co-design from the chip to network to the software, and then what is the case against co-design?

Amin Vahdat 23:37 Yeah. So there's huge optimization opportunities with co-design. And so what you can imagine is that if you have a end layer stack and you wanna be able to pick and choose whatever component you want, let, let's say you're running across many clouds or you're running across many pieces of hardware, many pieces of software, you could design abstraction layers that would basically say, "I can run on any hardware, I can run on any software, I can run on any network topology, um, any amount of network that you give me, no problem, and my system is going to be fully adaptive." Very likely, so now you have a great, um, capability. You can move anywhere. Like, new capacity becomes available, overnight you're up and running because you've designed your system that way to actually be able to take advantage of anything. You have not hard-coded, um, or specialized at all to anyone's particular infrastructure. Downside is you probably leave a lot of performance on the table if you become fully fungible, if you become fully flexible to anybody's hardware, software, network storage, compute, et- et cetera stack. So the case for co-designing is, um, between each one of those layers, um, there's a big impedance mismatch if you try to get full generality. And so there might be 10%, 20%, 2X across each of these layers. You start multiplying those optimization opportunities through, and all of a sudden you're left with a big end-to-end opportunity in terms of whatever you wanna say, intelligence per watt or goodput per watt, that you can leverage, even again, all the way down to power delivery, uh, and power availability, software optimizations, et cetera, et cetera. So pros are, um, you can run anywhere, anytime. You have no lock-in. Cons are you're leaving, um, significant, in, in all likelihood, significant amounts of performance on the table.

Sonya Huang 25:34 Hmm. So my understanding is that if you take, uh, an OpenAI and Anthropic, um, OpenAI was kind of primarily building on a, a homogenous compute stack, and Anthropic was building a roughly more heterogeneous compute stack. Do you think that's part of the reason, that co-design is part of the reason why they converged on what is rumored to be very different architectures for their models?

Amin Vahdat 25:55 Um, I can't, I can't speculate. I can't s- I don't want to speculate in terms of what, uh-

Sonya Huang 25:58 Yeah. Fair enough

Amin Vahdat 25:59 ... OpenAI and, and Anthropic are doing. It's just one, one possibility. Uh, but I think that-

Sonya Huang 26:03 Yeah

Amin Vahdat 26:04 ... uh, uh, without knowing the details of what they're doing, I would imagine that there could be many reasons for why they wind up with, uh-

Sonya Huang 26:09 Fair enough

Amin Vahdat 26:10 ... different, uh, architectures.

Sonya Huang 26:11 Well, what about for Google, then? I'm curious what the working relationship looks like between your team and DeepMind, kind of who's in the room, at what stage of model development, and how are you making decisions together on co-design?

Amin Vahdat 26:23 It's one of the most fun, uh, and frankly gratifying parts of, uh, being at, uh, uh, Google, is the opportunity to work really shoulder to shoulder, uh, with the DeepMind team in terms of, uh, co-design of our hardware and models. Uh, there's a third element to it in, in terms of that we also get to extend that with the consumer services and cloud. I'll put that aside for a moment. I can come back to that. But with respect to DeepMind, it, it really is a deep partnership. So let me... I mean, from the past, I can give you examples where, uh, they have come up with model optimizations, um, let's say to transformers or to particular math that they might wanna do, and we might have a chip in progress, not quite done. But then they say, "Oh my gosh, if we had hardware support for this, our end-to-end training or serving might get significantly faster, more efficient. What would it take to actually now change our hardware definition that we might be in execution on to accommodate?" And this then leads to, uh, our engineers from, and, and researchers to getting together intensely in the same room, uh, for a few days, a week, two weeks, saying, "Okay, yes, we can do this. More likely, we couldn't quite do what you wanted, but we can do this other thing, and then maybe you can change your model architecture in this other direction that gives you 98%, that gives us 90% of what we were looking for. And yes, we can then go back and intercept the hardware." And/or we can say, "You know what? We're gonna delay the tape-out by a week or two weeks because, wow, for this level of benefit, it's, it's totally worth it."

Sonya Huang 28:06 Yeah.

Amin Vahdat 28:06 You know, similarly, when we're, um, projecting our roadmap out and it's time for, you know, we, we, uh, we have many generations of chips that are essentially in progress at any one point in time. We have the chips that are in production. That's one. We have the chips that we're working on getting into production. They're already back from the manufacturer, and we're debugging them, making them work. We have the chips that are in implementation, that are about to tape out and go to the manufacturer. We have the chips that are in design phase, and then we have the chips that are in concept phase. So it's really this five, six-stage, many, many-year pipeline from in production to in your mind, et cetera. The collaboration with DeepMind for in production is significant, because we get to work together in maximizing delivered intelligence or delivered goodput per watt, and we know exactly what's happening in the model, and we know exactly what's happening in the hardware, and everything in between. We also get to collaborate deeply on, though, the chips that are actually just about to tape out. Why? Because we can intercept, and we can make changes to the chip, literally the chip architecture in-flight, which would be somewhere between hard and impossible to do if we were working across company boundaries. Not, not impossible. It would be much harder, right, for us to say, "Oh my gosh, we're a few weeks or a few months away from getting this chip done, and now let's get in the same room, shoulder to shoulder, and figure out if we should disrupt the program." It's possible, but, but harder, is what I would say. And then, of course, for the chips that are in design, we have many architectures we can evaluate together. We can then ask our colleagues in DeepMind, "Where do you see model architectures going in two, three years' time? Here's the Pareto of things that we can do. Here's the Pareto of where model architecture is going." We have, uh, actually deep and significant simulation infrastructures that can predict how the workloads are gonna map to, uh, different hardware architectures. Deep, deep, and fast iteration. It really isn't the teams are working separately. It's, uh, in, in, uh, same building, same rooms many times.

Sonya Huang 30:13 Yeah.

Amin Vahdat 30:13 Deep daily interaction. I talk to, whether it's Koray or Demis, multiple times a week, uh, et cetera. So it, it really is a, a super fun aspect of the work.

Sonya Huang 30:23 That's awesome, and eventually their models will help with, with chip design.

Amin Vahdat 30:27 We're using Gemini to design, uh, hardware for future Geminis as well.

Sonya Huang 30:30 That's so cool.

Amin Vahdat 30:31 Yeah.

Sonya Huang 30:31 I wanna come back to that-

Amin Vahdat 30:32 Yeah

Sonya Huang 30:32 ... a little bit later. On the collaboration itself, I guess, is there an impedance mismatch still of just, I figure the cycle times are probably just slower when you, when you're dealing with hardware than, than, than, than your DeepMind folks get to deal with on the software side, and I figure your planning cycles are much further ahead.

Amin Vahdat 30:50 For sure.

Sonya Huang 30:50 Um, and so how much room do you actually have to adjust

Amin Vahdat 30:54 Yeah, it's, it's a really good question. I mean, we are planning hardware two, three, four, five years in advance. No, no question. In other words, we talked about TPU 8i and 8t, uh, just now that we, uh, announced, but you can imagine that, uh, 9, 10, and maybe some others are, are in, in, uh, concept to execution to, to something else. And so they-- and they might be, um, years out. If you're working with model architecture, by default, you're not gonna be thinking years out.

Sonya Huang 31:20 Yeah.

Amin Vahdat 31:20 But I think this is also the great thing about how the company has grown up together, and, you know, there was Google Research and DeepMind and Transformers being invented. All of this was happening in, in these overlapping rooms as well. So in other words, there's a whole generation of researchers who've grown accustomed to being able to influence the hardware, and knowing that the hardware is operating on multi-year cycles, and they also know that, look, if they have a small tweak that's gonna deliver, like, 1% or 0.5% or something like that for a chip that's about to tape out, they're probably not gonna come to us because they, they, they know enough to know that actually it's not like software where you can just do a change list and it's gonna ship to production in two weeks' time. Like, it actually, stopping a tape-out is a big deal. But they also know if they've got a big, like a, a really good opportunity, the, yes, we're obviously gonna work together to figure out if we can get it in there. So, um, as I said, it's multiplicative in terms of where the benefits come from. The hardware does lift all, all the tides, right? And so significant portions of the DeepMind team, and we're so grateful for it, it's, uh, an amazing team, but significant portions of it are thinking about, "How do I influence the roadmap?" Because it's actually a pipeline. Like, the idea I had two years ago, like, it's in production now, and it's helping all the workloads at Google go faster. Like, that's a good feeling.

Sonya Huang 32:41 Totally. It seems like an impossible task, though, to predict in five years what workloads will be most common and what algorithmic breakthroughs will have occurred. Uh, does it seem like an impossible task?

Amin Vahdat 32:54 I, uh, hear you, that it would seem like an impossible task, but here's the awesome thing, and actually we're, uh, working on, uh, writing this in great, great detail, and it's a lot of fun to do it. The stunning thing is that, um, the TPU architecture at a medium level of detail, not at a super high level of detail, at a medium level of detail, hasn't really changed since TPU v1. Like, one way to look at it is the instruction set architecture for a CPU. Like, you've got loads, and you've got stores, and you've got adds, and you've got subtracts and branches and, like... And yet, what has the software on top of it done, right, over, over that period of time? Same thing with TPUs. We have some fundamental instructions and so- fundamental primitives. Of course, we've extended it. It's not like the instruction set hasn't gotten changed at all, but the primitives there in terms of specializing the numerics to very big, very large matrix multiply units, um, a, a sparse core that manages, uh, vector operations and scatter gather operations, et cetera. There's five or six things that really define a load, remote load store. That's another one, actually. We can read and write, uh, remote memory through our ICI network. The fundamentals have been there, and they've extended just to many generations of models, many generations of even, uh, deep neural network al- al- algorithms and model structures.

Sonya Huang 34:16 Okay. Speaking of shifting workloads, it seems like one of the biggest changes in workloads over the last year or so, I think it really started at the beginning of this year, th- this calendar year, was kind of the rise of the long-horizon agents.

Amin Vahdat 34:27 Yes.

Sonya Huang 34:27 And I would guess that's a very different shape of workload than kind of the quick turn LLM conversations of, of years past. Um, what does that mean in terms of data center needs?

Amin Vahdat 34:38 Yeah. So I think th- one, uh, there's two huge aspects to this. One is it's no longer human to, um, whatever you wanna say, accelerate interaction. In other words, when you are typing out a, a prompt on a, on a web browser or on your phone or whatever, uh, of course, in response to your prompt, there's gonna be a bunch of work that happens. But then when the response comes back, you've gotta read it. You've gotta think about it, and then maybe you have a follow-up. That's gonna be multiple seconds of interaction time. Now, on deep horizon agents, it's not, there's no human in the loop that is going to naturally rate limits how quickly re- requests are gonna go to the model, right? So this w- went from, you know, seconds, maybe tens of seconds in terms of inter- interaction time, is now going into perhaps milliseconds, right? As soon as I get a, a response back, I can parse it, I can reason about it perhaps a bit, and then I can figure out what my next request is gonna be. So that's big change one. Big change two is all that reasoning and all that parsing is gonna probably happen on a CPU, and that CPU is then going to have to probably think about, "Okay, what other state do I need to go gather before making my next prompt back into the model?" Like, in other words, I've, I've learned something from this response. I'm gonna take another step, but I actually need to go grab some context, maybe from, uh, DRAM local to me, maybe from someone else's DRAM on another CPU, maybe on SSD or maybe on HDD somewhere else. So now, a huge amount of orchestration has to take place as well. So the design actually is changing pretty significantly, where the demand for accelerator compute is going up, but the demand for CPU and networking and storage, the, you know, traditional data center CPU, et cetera, is also going through the, through the roof, uh, at the same time.

Sonya Huang 36:24 So does that mean you're putting more CPUs alongside your G- your GP racks then?

Amin Vahdat 36:28 So this is, uh-

Sonya Huang 36:29 GPU racks, yeah

Amin Vahdat 36:29 ... yeah, GPU, TPU racks. This is the key question is, um, uh, going back to this, uh, question of optimization and, uh, specialization. If we start putting lots of CPU racks, and it is a question, uh, next to the GPUs and TPUs, that means that actually we can't fully specialize the density and network requirements, let's say, of a TPU rack relative to a CPU rack. TPU rack is gonna be more dense than a CPU rack. It's probably gonna need more networking than a CPU rack. So in other words, now our building design ... is gonna change. Another option is maintain your uniformity. Put your TPUs or your GPUs all in, uh, one building and the building next door, but maybe put your CPUs and your, maybe your, um, and maybe by the way, the hard drives have to be in another building on the other side because they have yet another set of requirements. Now you need networking between these buildings at a pretty significant level. So once you leave a building, the networking complexity goes up significantly from a reliability perspective, from a cost perspective. Latency goes up probably acceptably, but, uh, still now you might go to hundreds of microseconds, potentially more with queuing between the components. So the considerations do change in pretty interesting ways.

Sonya Huang 37:47 Hmm. Your job is hard.

Amin Vahdat 37:48 It's fun.

Sonya Huang 37:49 [laughs] It's fun.

Amin Vahdat 37:49 It's

Amin Vahdat 37:50 fun. Yeah.

Sonya Huang 37:51 Um, what's happening on the networking side? And, uh, I've heard that Google's always been the f- at the forefront of the newest in networking, including optical. Uh, can you say a word on what is like the s- the state of, of optical networking?

Amin Vahdat 38:04 You know, we at, um, Google, this was probably 15, 16 years ago, were among the first to bring, um, essentially, uh, what's called wave division multiplexing, um, where you could put, uh, multiple signals on a single fiber within the data center, and we actually leveraged that for all of our, uh, communication between racks. At the same time, we introduced the technology, uh, along with the wave division multiplexing, uh, called optical circuit switching. And essentially what optical circuit switching does is, in contrast with traditional electrical packet switching, what it does is it transmits and moves data entirely in the optical domain. The great thing about that is, and so let me describe the technology that underpins it. In a, in a packet switch, you would take a packet that has a header. You would look at it, um, in the electrical domain, figure out where the packet is headed. In its header, it might have an IP address that says, "Okay, where do I head it?" You look up, okay, for that, uh, destination, you look up in a table, what port do I forward it along? So basically billions, billions and billions, perhaps trillions of packets coming through at super high speed per second, you're forwarding them along. Optical circuit switching says, "I'm not touching these bits in the electrical domain. What I'm gonna do is I'm going to figure out for an input port which output ports to send the light to." Okay? And now I have, uh, there's multiple ways to do this. The one that we started with is called, um, MEMS, uh, switches, micro electrical motors that basically control mirrors in 3D. So we can now programmatically take a box that might have, you pick, 128 ports, 256 ports, some number like that, and we can configure every input port for a fiber that comes into it, map it to an output port, and then change the rotation of mirrors, where the light literally shines down on these mirrors and gets reflected to the right output port. Initially, the reason we did this, twofold, was to create locality between groups of racks. Let's say I had a compute cluster and a storage cluster, and both of them were in support of, again, making it up search. We knew that these two clusters would talk to each other a lot, so we would configure the mirrors to create shortcuts purely optically between those two clusters, clusters of racks. That was reason number one. Reason number two was we wanted to be able to expand the network and contract the network without actually moving any fiber. And without working into the details, I could draw this on a board, the optical circuit switch would allow you to actually reconfigure the spine of the network to expand it or to shrink it without a human being, um, having to do anything other than be, uh, literally a controller that would manage that. Now fast-forward to TPUs. Uh, a TPU, uh, has a torus topology that connects all the TPUs to one another directly. I, I talked about throughput and goodput earlier. One of the things that we can do is if we have a TPU rack that fails, we can replace it with another TPU rack without moving any fiber. Again, I'd have to wave my hands, uh, or draw at a whiteboard, uh, but essentially we can say we have a spare rack available at all times, and when a rack fails, we redirect the light to that new rack.

Sonya Huang 41:27 Huh.

Amin Vahdat 41:28 And that can be done in milliseconds.

Sonya Huang 41:31 Why do you have the fiber at all then?

Amin Vahdat 41:33 As opposed to f- a, uh, fully free space?

Sonya Huang 41:35 Yeah.

Amin Vahdat 41:35 Yeah, it's a very good question. Um, uh, attenuation loss and the bandwidth would drop, uh, dramatically. F- furthermore, across the range of a very large, uh, building, aiming everything, uh, without fi- without the benefit of fiber to connect it all together, uh, in 3D would be, would be challenging to possibly, uh, impossible. Um, we've talked about it. We've talked about it actually and, and there have been some really interesting discussions. Uh, but, but yes, it's mostly in fiber, but then when they hit the optical circuit switch, uh, essentially the light then literally shines down on these tiny chips. So, um, that's one big direction of, uh, networking. There's, but there's, there's a lot. I mean, I would say networking is, uh, exploding in, in terms of its, uh, uh, capabilities and need, frankly, in the data center.

Sonya Huang 42:21 So cool. We could have an entire conversation on that.

Amin Vahdat 42:23 Yes, it is very, very cool.

Sonya Huang 42:24 Even like the, going back to the picture from Ineffable, it was all the, the cables that, that kind of-

Amin Vahdat 42:29 Exactly

Sonya Huang 42:29 ... caught everybody's eyes.

Amin Vahdat 42:29 Exactly. Yeah, and those cables, eventually some of them wind up at optical circuit switch in our data centers.

Sonya Huang 42:35 Makes sense. Okay, I wanna flip to talking about power. You've been talking about goodput per watt and o- other units per watt. Makes me think per watt means power is kind of the, the binding constraints or the scarce constraints or the expensive constraints in some way.

Amin Vahdat 42:47 You know, so I, I, uh, I get asked this question, what is the biggest constraint that we face? And, um, the reality is there is no single biggest constraint that we face. They're all constraints. They're all super hard. It could be, uh, and, and they shift, uh, continuously. But if I had to answer fundamentally, uh, I would say that power is the single most fundamental constraint that we face. Like, everything else seems S- uh, like we know how to solve them, and it's a question of, um, solving them over some period of time. Power, yeah, yeah, I think the way that you put it is really nice. It's a, it's a binding long-term issue that we do not, uh, have a... I mean, the, you know, nuclear, uh, you know, perhaps nuclear is going to, uh, abundant, uh, clean energy, uh, which would solve a lot of problems, uh, for, for sure. When that happens and at, when it happens at scale, uh, still, still unknown.

Sonya Huang 43:37 And so how does it work in practice? You, you know, you're standing up a new data center. It needs a gigawatt of power. I imagine you can't just call PG&E and say, "Hey, please send a giga- watt of power." So how, uh, what does provisioning power actually look like? And are you having to actually vertically integrate all the way down to, like, you know, doing your own turbines? Or how do you solve the power bottleneck?

Amin Vahdat 43:58 Yeah. It's, again, it's a, it's a big question, uh, important question. We do, uh, our, our preferred model at Google is always to be, uh, utility connected, to be grid connected. So it's the equivalent of calling your, um, uh, fa-

Sonya Huang 44:11 Oh, so you can't just call PG&E.

Amin Vahdat 44:12 Are you f- Ex- exactly.

Sonya Huang 44:14 Okay.

Amin Vahdat 44:14 You call, uh, ver- very politely your favorite, uh, utility, wherever it is that your, uh, data center is, and of course, you give them m- many, many years of notice. So in other words, if, if we're talking about a, a gigawatt scale, it's not something that you can say, "Hey, I need a gigawatt, uh, tomorrow. When, when can you start billing me?" Uh, it's something that we co-plan, plan together. You know, for, uh, for us, it's something that we also take very seriously from the perspective of ensuring that when we work with these utilities, the, the costs of putting that infrastructure in place, the... And we could get into a long conversations, uh, a long conversation just on this topic as well, that we cover those costs because with the way that billing works, it actually could be that by the act of the utility building out capacity for us, let's say, other people's rates could go up. In theory, what we ensure is that actually, let's say, the transmission lines that have to be built, upgraded, uh, additional utility base stations, et cetera, that we pay for those as well. But so there's a long planning process. Uh, it, uh, can absolutely be the case that in, uh, let's say that we need a gigawatt in, I'll make up a date, uh, 2028, uh, and the utility can get us a gigawatt in 2029. They can get us, let's say, 700 megawatts in 2028. So now we might be left with a question of how do we cover those 300 megawatts? Well, one answer is just wait. Another answer is to say, "Okay, well, would we figure out how we generate some subset of that power ourselves?" And maybe that would be with solar cells or with, uh, with batteries as a backup, et cetera. Would it be for other sources? So then we again work with the utility. So it might also be a, uh, combination where we might maintain some power, uh, generation local and have the capability even when the utility, uh, comes online, uh, fully at, let's say, the gigawatt scale, where we could actually provide power back to the grid, so having that in when they need it as well, right? So in other words, there might be the... One way to look at it is there might be the two weeks of the year, maybe it's the hottest two weeks of the year, where there's huge amount of residential demand. If we have some local generation, we can then provide that back to the, uh, grid as well. So it really is working over multiple years with the utilities, uh, for us.

Sonya Huang 46:23 Why is your preference to do that versus to kind of go, like to work with the utilities as opposed to, to kind of vertically integrate yourselves?

Amin Vahdat 46:32 Yeah. So, um, the main, main reason is flexibility and uplift on both sides. So one, one way to look at it is, um, statistical multiplexing, or if you want, the law of, uh, large numbers. If we need to have a gigawatt of, uh, power, let's say we wanted to have that with 99.99%-plus reliability, probably means we have to build two gigawatts of power, right? With, at that level of 99.99 or 99.999, you have to have one plus one redundancy. That, that gets expensive. And then making sure that that's going to be ideally a clean energy source probably right next to our data center, that can also be challenging. Now, if we partner with the data center, again, maybe we have some, some amount of that that we can bring ourselves, that we can give to the grid when they need it less, that we can take their power. So in other words, by leveraging that statistical multiplexing over a much larger base, actually everybody wins.

Sonya Huang 47:26 Mm-hmm.

Amin Vahdat 47:26 We win, the, the grid wins, uh, residences, uh, win, et cetera. We prefer that. And in, in rare, rare cases, we, we will, uh, do it, uh, let's say, behind the meter, but even then, we're doing it under the assumption that we're gonna work with the utility where it might be a year later. We, we wanna be connected to the grid. So again, it's an uplift for us, it's an uplift for the grid.

Sonya Huang 47:49 Makes sense. How do you decide how big to make a data center?

Amin Vahdat 47:52 Yeah, that's, that's an art, and it's a, a source of significant debates. You know, at, at some point, you know, we had a debate, I remember even 10-plus years ago, 15 years ago, big debate at, uh, Google, should we just put everything into one data center? Like back then it was gonna be a gigawatt [laughs] which was-

Sonya Huang 48:08 Simpler times.

Amin Vahdat 48:09 Yeah, different times. Like 10, 15 years ago, we're like, "We're gonna have a gigawatt data center." So one obvious concern there is single point of failure, right? If you're looking at things from a 30-year perspective, uh, that's a, a huge concern. Uh, you know, on the other hand, if you're talking about, uh, training workloads today, bigger is better, right? In other words, having more... Because from a networking perspective, actually, you wanna have things concentrated in as small a distance as possible. But then again, uh, two issues, single point of failure, but then also now power availability, right? While a gigawatt was huge 10 or 15 years ago, but imaginable, um, there's no way anywhere in the country or the world we're gonna get the total requirements of Google built in, in one place. It's just, just not gonna be possible. So now what's the optimum size? Uh, again, we have models for this, simulators, uh, et cetera, but this, it, it also depends on, look, in some places we're gonna be at the edge of the network or we're gonna be in country, that might be tens of megawatts. We actually partner with ISPs, that might be a rack Like literally it could be a rack. This is a training cl- cluster. Okay, maybe that's gonna be closer to a gigawatt. Um, some other sites might be hundreds of megawatts, uh, et cetera.

Sonya Huang 49:23 Interesting. I'd love to understand how you think about the portfolio life cycle managements. Um, so I guess my guess would be that, you know, the biggest, newest clusters are used for training the latest frontier model, and then you kind of recycle the older stuff and run inference on it. Is that, is that, like, a fair framework? Are you, are you also standing up inference-specific clusters? How does that all work?

Amin Vahdat 49:46 Very reasonable framework, uh, that, that you're thinking about. That, uh, and I think makes a lot of sense, but I would say that the demand for inference is such that we can't just rely on, um, whatever older training clusters that are no longer being fully used just for training, uh, as the basis for inference. And also, if you think about it, we might centralize again, uh, in a particular year into a small number of large sites, our training clusters. And there is benefits to keeping the network distance between them, uh, small. So now, let's... Wherever they're located in the world, s- they're probably gonna be, let's say, on the same continent, or they might be even in the same portion of a continent in a particular year. So now you might be left with, um, not enough serving capacity on the other continents, et cetera. So then, okay, actually, we then have to go build this specialized inference, uh, elsewhere across the world. So, uh, I, I think your intuition is, uh, spot on, but not wholly. In other words, it really does have to be, okay, here are the training clusters. Yes, probably they're gonna be used for serving in some number of years. Uh, but then we're also having to build the, the serving clusters as, as well.

Sonya Huang 50:50 And your serving clusters, are they different than the training clusters? Like, are they smaller? Uh, is there... Are they, are they cheaper per, per, per megawatt?

Amin Vahdat 50:58 Uh, not necessarily cheaper, actually, because, uh, for serving, there is more need to co-locate, uh, uh, compute networking and storage. And so that then we get to the inability to specialize. So for training, you actually have this big density, uniform deployment, et cetera. Uh, whereas for serving, you're gonna have to have the mix of storage, compute, and accelerators. Furthermore, for, um... It, it's a v- very interesting aspect of this that we could go into more detail on as well, but for serving, you actually don't wanna have too much in one place. You wanna be serving your workloads from, uh, all over the, the planet. But now, uh, we wind up having, um, individual model endpoints, and there actually can be variations of models. So now we have to spread these models out across the world, also accounting for locality. So they will be smaller. They'll actually be less vertically integrated, probably, on the inference side.

Sonya Huang 51:55 Super interesting. So if you made a data, data center five years ago, you know, state-of-the-art accelerator then five years ago was very different.

Amin Vahdat 52:03 Mm-hmm.

Sonya Huang 52:03 Vastly less efficient than the ones that are being made today.

Amin Vahdat 52:05 Yes.

Sonya Huang 52:06 Do you actually go back and, like, swap out the, the chips in those, in those old data centers? And I know this kind of relates to, there's been an ongoing debate, I think, of like what is the actual useful life of a chip.

Amin Vahdat 52:16 Yeah. Yeah. So you know, I've been, uh, uh, on, on record as saying this in one, one of my more, uh... I was surprised by how, how much pickup this, uh, had. But, but, uh, you know, I said that our, uh, seven- and eight-year-old TPUs are still at 100% utilization.

Sonya Huang 52:29 Wow.

Amin Vahdat 52:30 Uh, and, and-

Sonya Huang 52:31 I think I saw this. I think that was this muted.

Amin Vahdat 52:32 Yeah, which-

Sonya Huang 52:33 Okay. Yeah, yeah

Amin Vahdat 52:33 ... yeah, I, I didn't mean for it to be a, a-

Sonya Huang 52:35 [laughs]

Amin Vahdat 52:35 ... a, a meaningful statement, but it was a meaningful statement, apparently. Yeah, so our, um, older TPUs, uh, uh, GPUs, but our older TPUs are seeing significant utilization. We do, uh, replace them. In the end, there's a question of, okay, have they, uh, uh... Depreciation, uh, lifetime is approximately six years. Uh, so once they're fully depreciated, uh, and given the power efficiency of newer generations, et cetera, it does make sense to, um, replace them and upgrade them. It's not that we... I mean, we do replace the chips, but it's really the, the, the systems. So in other words, we think in terms of pods. So a TPU, let's say, um, 8t pod might be 9,600 chips, and it might be, uh, 140 something, 152 racks, et cetera. So then we would say, "Okay, we're gonna pull out that pod, and we're gonna replace it with a, whatever it is, TPU 12 or 13 or 14 pod." It might not be a perfect fit. Uh, which then we have to... In other words, the new pod's footprint might not be a perfect fit for the old pod's vacancy. We, we then just have to account for that, and we have to figure out how we would retrofit. We can't plan that because we don't know what that many generations of TPUs are gonna be. Uh, so then it's, uh, again, a bit of an art to figure out how we would, and it's actually, uh, hard work, figure out how we would decomm and then replace as quickly as possible with, uh, with new TPUs.

Sonya Huang 53:53 You've written publicly about, uh, open standards. Um, can you say a word on that?

Amin Vahdat 53:58 Yeah. So I think that's, uh, the... We t- we talked about this, um, interoperability question, and so for us, while we support vertical integration and we make it possible to, you know, extract just as much performance as you want, for us, it's really important to, um, not force lock-in and not force a sort of walled garden in terms of how the system works end-to-end. So, you know, for... I mean, I'll give one example. At, uh, Google, we've developed a, um, model development framework called JAX. Uh, we like it a lot. We think it's really, really good, and we use it, uh, extensively internally. Many of our customers like PyTorch, right? And so one, one thing we could say is, "Hey, if you wanna run on TPUs, um, you gotta use JAX. It's the best." I'm, I'm, I'm being, I'm facetious a little bit, uh, but may- maybe it is, maybe it isn't. We think it's the best, and that, that's your only choice because it's, it's so great. Or we could say, "Look, if you like JAX, we like JAX. If you like JAX, you can use it, but if you like PyTorch, we have Torch TPU, where your unmodified models, et cetera, can, can run." We've... And we, you know, we've seen this, uh, throughout history, uh, actually with many, many examples. I've used the example of, um, IP. Why did the Internet Protocol win? There were actually many competing protocols to IP. This is ancient history, uh, in the '70s and early '80s. Why did IP win? It was because it was, uh, open standard, and it was this narrow waist to the hourglass where anything could run on top in terms of software, and anything could run on underneath it in terms of hardware. So open standard, interoperable, anyone who brought IP could plug into a router port and become part of the internet. Like, that was... It was beautiful, and that's what allowed the internet to explode in its, uh, growth and reach across the world. So we really do believe in those, uh, open standards in terms of plug-in points. Now, if you want to plug in something highly specialized, you can, right? If you, if you think that you've got a better thing to plug into our, our framework, you absolutely can, but we wanna really support open standards, ideally open source around it as well.

Sonya Huang 56:08 It's too big and too important of a build-out to be, you know, any one vendor's closed proprietary stack.

Amin Vahdat 56:14 Yeah. We really-

Sonya Huang 56:14 Yeah

Amin Vahdat 56:14 ... we really believe that, and it's gotta be one of choice. And that's also why, for example, we fully support and have, uh, TPUs, GPUs, other accelerators, et cetera.

Sonya Huang 56:23 Yeah. How has day-to-day life for, for your team changed with AI? Where is it changing your, your function the most?

Amin Vahdat 56:31 So I think the, the... whatever you wanna say, the easiest answer is on the software engineering side, where, I mean, it's been documented, uh, externally quite a bit. Uh, so I think that we are, at Google and my team, using it to, uh, great effect in, in terms of our software development's, uh, uh, capabilities, but also, uh, frankly, uh, tests, roll-outs, even, uh, helping with design, et cetera. But there have been, s- on the hardware side, which has maybe gotten less, uh, external coverage, also significant change. In other words, my, my hardware engineers, I w- I was just looking at the numbers earlier, uh, today in fact, are using AI, using whatever the, the token counts, if, if you want. It's not the best metric, but it's still a metric. Uh, our hardware engineers are using AI as much as the software engineers are, and productivity has gone up. Time to, you know, time from design kickoff to tape-out is shrinking.

Sonya Huang 57:28 Mm.

Amin Vahdat 57:28 Time for bring-up is, uh, shrinking, so in other words, productivity is, uh, going up, uh, significantly. But then other, maybe less expected, uh, changes, how we do data center design, uh, has, uh, changed significantly. In other words, how we... and how we evaluate... You, you mentioned, hey, do we do a gigawatt building or 100 megawatts, um, campus, or a 200-megawatt campus. I- in the past, this would be very detailed, very sort of, um, spreadsheet-driven, human-driven processes, and it still is to some extent, but there's a lot of AI now involved that really streamlines the process in terms of planning and, uh, development as well.

Sonya Huang 58:09 Interesting.

Amin Vahdat 58:10 So, well-

Sonya Huang 58:10 Does, like, reasoning models, or...?

Amin Vahdat 58:12 Um, I wouldn't say quite yet reasoning models. It's not, it's not replacing human judgment, but it's, um, making it much easier to bring the necessary information together, like, uh, in, in, in one place, and, uh, basically h- h- uh, put the right information in front of the humans making the decisions.

Sonya Huang 58:29 Okay, I'm gonna bring us home with two kind of out there, kind of fun questions.

Amin Vahdat 58:33 Yeah.

Sonya Huang 58:33 Um, question number one, uh, orbital data centers.

Amin Vahdat 58:37 Mm.

Sonya Huang 58:37 Um, I've seen that, you know, Google is thinking about this quite seriously, and, uh, you don't need to comment on that, but I mean, does the fact that people are seriously running the numbers on orbital computes mean that there are kind of serious b- binding constraints on Earth, and, and what do you think of, of orbital data centers?

Amin Vahdat 58:53 Yeah, it's, it's an exciting direction. We are, uh, actually, uh, pursuing it, and we've, uh, referred to it, uh, more with no tongue in cheek, as a, as a moonshot, as one of our big, um, big efforts that, uh, we're excited about investing in. Uh, it goes back to the earlier part of the conversation where you raised, I think correctly, that in terms of fundamental constraints, um, energy and, uh, energy production is a key, key challenge. And so really, um, bottom line is from a fundamentals perspective, um, in space, um, you have something like 40% more power available because of lack of attenuation in the atmosphere, et cetera. In other words, there's just more solar capacity, 1.4x. So that's one part of it. But, but in a, um, sun synchronous orbit, you have 98 to 100% coverage of sunlight on your solar cells relative to 28, 30, maybe 35%, um, on land. Right? So in other words, just this enormous amount of power, obviously the sun. You, you take the 1.4x and you take the 3 to 4x, uh, in terms of number of hours per day. You remove batteries more or less from the equation, and now you have the potential for something that can really... And, and of course, carbon-free. Lots of benefits, lots of challenges.

Sonya Huang 1:00:14 Yep.

Amin Vahdat 1:00:14 Right? Lots and lots of challenges. So in other words, okay, now what about cooling? You, uh, you know, naively you might think cooling in space is easier. It's actually harder. Uh, reliability, we talked about how these things fail sometimes, so repairs, uh, is becomes harder. Not, not impossible, but it becomes harder in space. Networking-

Sonya Huang 1:00:31 Send a little robot up with the, with the cluster.

Amin Vahdat 1:00:33 Um, and exactly.

Sonya Huang 1:00:33 Yeah.

Amin Vahdat 1:00:33 So perhaps there'll be, uh, that this model, uh, is a, is a promising, uh, direction. Uh, redundancy will probably be your friend. Uh, your, uh, uh, prescient question on free space optics is now going to become reality. We're probably not gonna be stringing fiber, uh, between these components, so it will actually be free space, and we're gonna have the lasers pointing at the, um, receivers and calibrating in, in real time. These are, uh... There are no showstoppers here, no fundamental showstoppers.

Sonya Huang 1:01:00 Last question. I've had this image of the ineffable, uh, big supercomputer that you built them-

Amin Vahdat 1:01:06 Mm-hmm

Sonya Huang 1:01:06 ... um, in my head this whole time, and so here's the question: 10 years from now, what will the most frontier supercomputer look like?

Amin Vahdat 1:01:17 Oh, my gosh, yeah. So 10 years is right at that edge where it's, um, uh, I would say In all sincerity, probably some people are thinking about what the 2036 computer looks like. We might have a few thinking that far ahead, but wow, I, um, the cone of uncertainty there is, is, uh, way, way too wide. Like, in other words, if you look at the trends, um, the level of integration is going to be tremendous. So, you know, we saw the beautiful fiber and the, um, VR200, uh, racks that we put together for Ineffable. My, my guess is it's going to be more integrated, less fiber to... I won't say no fiber-

Sonya Huang 1:01:58 Okay

Amin Vahdat 1:01:58 ... in 2036, but I think that probably from a rack perspective, um, the rack will look, um, much more tightly integrated, and the amount of fiber, there'll be maybe a small fiber bundle coming out of the rack. And then, uh, so in other words, one, one view you might have from a modular manufacturing perspective is these racks, likely in 2036, are gonna be manufactured centrally. And so whether they have 72 or 144 or 288 or probably more, but, you know, 576, 1152, pick your favorite multiple of GPUs, since we're talking about the I- in- Ineffable case, but then it'd be the same for the TPUs, deeply integrated into a rack. Would a rack be multiple megawatts? You know, we don't know exactly how, but 2036, we get to, we get to imagine things. Could be multiple megawatts in a single rack, and now you bring water, and you bring power, and you bring fiber, right? You, you, you wheel that, uh, rack in, you plug those three things in, and you're off, uh, off to the, off to the races, uh, as well. Uh-

Sonya Huang 1:02:59 Okay, so it's not gonna be like a big alien orb then? [laughs]

Amin Vahdat 1:03:02 Up, up in space. By 2036, I wouldn't say that it's not possible-

Sonya Huang 1:03:06 Yeah

Amin Vahdat 1:03:06 ... uh, for 2036-

Sonya Huang 1:03:07 Yeah

Amin Vahdat 1:03:07 ... for it to be, uh, uh, uh, then directly launched into space, uh, and then maybe, um, uh, uh, taken by a robotic arm at some space station and then plugged into the right, uh, module. Yeah, may- maybe that for 2036.

Sonya Huang 1:03:19 Fun.

Amin Vahdat 1:03:19 Fun, fun to think about.

Sonya Huang 1:03:20 Um, I really enjoyed this conversation.

Amin Vahdat 1:03:22 Yes, it was a ton of fun.

Sonya Huang 1:03:23 You are... This is the most intense, biggest CapEx build-out in history, but I also think it's, it's a technology-

Amin Vahdat 1:03:32 Revolution

Sonya Huang 1:03:33 ... revolution-

Amin Vahdat 1:03:33 Yeah

Sonya Huang 1:03:33 ... that is a thing of beauty.

Amin Vahdat 1:03:35 Yeah.

Sonya Huang 1:03:35 And I think you have such a grasp over, you know, the beauty of the technology and, and the binding constraints and, and how to balance this, this all. Uh, Google is in good hands.

Amin Vahdat 1:03:46 For sure.

Sonya Huang 1:03:46 Thank you for taking the time to share what you're doing.

Amin Vahdat 1:03:48 Yeah, it's a, ton of fun. And we get to live, live-- we get to live through this, and we get to define it. So, so thank you very much. This was a great conversation.

Sonya Huang 1:03:53 Thank you.

Amin Vahdat 1:03:54 Yeah, thanks. [outro music]