The $7B Startup Building AI Chips to Challenge Nvidia | Fractile Founder & CEO, Walter Goodwin

Walter Goodwin 0:00 We're so traumatized by the dot-com bubble that we've priced in exuberance right from the outset. We will be spending far more on running models in 10 years' time than we are today.

Harry Stebbings 0:11 Joining me in the hot seat today is one of the leading minds when it comes to frontier chips, Walter Goodwin, founder and CEO of Fractal. The business has just raised a $720M Series C co-led by Thrive, Founders Fund, Lightspeed, and Redpoint, valuing the company at a $7B price.

Walter Goodwin 0:30 It feels like we need thousands of times more compute than we have today. Frontier intelligence is not yet commoditizing. I think we'll have a couple of multi $100BN companies built in that space. One neo cloud that is competent is much like another neo cloud that is competent.

Harry Stebbings 0:45 Ready to go? [upbeat music] Walter, I am so excited for this, dude. It's lovely to be able to do it in person, and I'm so thrilled that you could join me in the studio, man.

Walter Goodwin 1:04 It's very good to be here with you, Harry.

Harry Stebbings 1:06 Now, I wanna start, for those that don't know, 60 seconds, like what is Fractal? What do we need to know?

Walter Goodwin 1:12 Fractal, we're a frontier AI company of sorts. Um, but we make chips, we make, uh, these big rack-scale systems, um, that are, you know, fully focused on advancing the frontier of AI capabilities. And so from a team composition point of view, this means that we have, you know, ML researchers down to, uh, process experts that fly out to Taiwan every few weeks, down to people that actually look at the thermals and the mechanicals of putting together one of these rack-scale systems.

Harry Stebbings 1:43 Totally get you. I was listening to you on No Prize. We were just talking about Sarah. And you were saying unequivocally that you were a chip company, and I was listening to this thinking, "Great." But then I'm looking at it going, everyone's moving full stack now. We're seeing OpenAI have Jalapeno. We're seeing DeepSeek have their own. We're seeing Anthropic build their own. Obviously Meta and Mark has his

Harry Stebbings 2:03 own. My question is, is the future won by vertical integration, and can you be a, a chip company alone these days?

Walter Goodwin 2:10 It's a very good question. I think there is... There's a verticalization in that as all of these different components come together, um, and especially as you kind of look into the future and you work out, you know, what are these models gonna look like, how do the workloads evolve, how does the silicon evolve, there is this absolute necessity to kind of have a vision on every part of that. Um, so Fractal is a chip company. Um, but in a sense, we're also a frontier AI company. And I think what that means for us is, you know, and this will continue to be the case, you know, we monetize by selling chips. We monetize by selling racks of compute to our customers. But it's completely essential that we have a bunch of folks internally who are ML researchers, who are looking at how model architectures might evolve, who are maybe even running, you know, smaller scale experiments than the ones that you would see in the leading labs. And so I think there is a distinction here between the expertise that is needed internally to win and then the kind of nature of the business, what the business model is. You know, we, we never intend to compete with our customers. We will never monetize internal models because that's how we've chosen to build this company. We've chosen to compete in the chip layer. Um, but you can't sort of step back from challenging yourself to have expertise in all of these other places because, you know, in our case, that would mean you'd, you'd place the wrong bets. You have to synthesize everything that you're learning across the ecosystem, and you have to bring that back then to be obsessive about the part of this space that, that you're building. And if you do this right, you've ideally found yourself an enormous sort of piece of green space where you're incredibly helpful to your partners, to your customers, and to your suppliers upstream. Um, and you've kind of got to know parts of their business as well. But of course, you, you know, you never seek to, to conflict

Walter Goodwin 4:04 with them.

Harry Stebbings 4:04 Would you argue that everyone's moving to vertical integration because everyone's fighting against commoditization of one layer? If you think about what, you know, OpenAI did with Jalapeno in nine months, um, it suggests some form of commoditization of the layer. Do you think everyone is desperately expanding to f- run against commoditization?

Walter Goodwin 4:23 I think there's definitely a fight to keep your margins expanded that do mean that, let's say, if you are serving frontier models, you know, it is important that you're not squeezed all the way down by very high margins on incumbent chip players. And that's where it's very useful to have, for example, a first-party play or another third-party play that can be a counterpoint to that. Um, and so I think in a sense there is this, there is this kind of confluence of forces that do mean that, you know, if you're a frontier lab, if you're a hyperscaler, it's very important to kind of have your own retort to various parts of this. Um, I do also think that there's this pretty crucial distinction today still between what is perhaps becoming commoditized and, you know, what we still see sort of at the frontier.

Walter Goodwin 5:13 Uh-

Harry Stebbings 5:13 Wh- what, what is commoditized then? Help me. I'm an investor, okay? I, I, I [laughs] look for value in amazing-

Walter Goodwin 5:19 Mm

Harry Stebbings 5:19 ... entrepreneurs and businesses. What is commoditized that most people think isn't, and what isn't that maybe people think is?

Walter Goodwin 5:25 There's this sort of sometimes concern, sometimes excitement about the idea of intelligence saturation, you know, that as models get smarter and smarter, there will become a kind of waterline that they exceed, where we don't really need greater intelligence for a whole set of tasks. Um, and I think it is true that there are certain tasks for which we see this kind of intelligence saturation, and this is where you then see, you know, an open model is good enough, uh, and these models become commoditized at that layer. Um, I think where that becomes more nuanced though, and where actually models Even at that level, sort of still fight against commoditization is that there is always a kind of economic pressure to, for example, reduce the cost of intelligence. Uh, and perhaps counterintuitively, you know, this is actually why I think the frontier labs and the frontier efforts remain so vital is, of course, frontier intelligence is not yet commoditizing. We see, you know, large margins there. We see big deltas against, you know, against sort of everybody else. Um, and also these sort of learnings from the frontier, they kind of trickle down also into building better, you know, better smaller models as well. And so actually, you know, if you look at that sort of fight across the entire gamut of frontier LLM capability as well as kind of behind the frontier LLM capability, there's still this sort of ferocious battle. Um, and I think, you know, I would distinguish or, or I would characterize that as, as sort of not commoditized insofar as if you are a market leader in something, uh, you can still extract high margins. So a good example of this actually is, uh, DeepSeek released a few days ago their entire like inference stack, their entire software stack, and it had really remarkable... You know, people very quickly characterized it and realized that DeepSeek had done so well in building an inference stack to run their models on GPUs, that they were getting, you know, much higher throughputs, much better interactivity, better speeds than, than everybody else who had built a DeepSeek inference server. And so, you know, something that you might have thought would've been by this point commoditized, which is the serving of an open model. You know, in their case, by going full stack, uh, they built a model which was more efficient, and they built an inference server for that model that was more efficient, and they'd carved themselves out the right to have, you know, really healthy margins on that. So I think even in the sort of open source space, we don't yet see this kind of complete race to the bottom, at least for those folks that sort of, again, try to own a lot of vertical understanding of the space. I think what is commoditizing, and I, I don't think anybody has ever argued it won't be, is kind of, you know, the whole space of, of cloud compute. You know, one neo cloud that is competent is much like another neo cloud that is competent. Uh, the reason it's so exciting today is we don't yet know which ones are competent. So I think there's still a lot of space there for there to be huge winners and, and maybe sort of huge losers as well.

Harry Stebbings 8:29 But I think that's why you see all the neo clouds desperately trying to vertically integrate too, just because I think they're very aware of the commoditization of their own layer. You know, we had the founder of Nebius on the show who was talking about the movement into model routing and inference for them as a core business and a core strategic, just 'cause I think they're aware of that.

Walter Goodwin 8:46 Yeah. I think that's true, and you definitely see, you know, every neo cloud is trying to build an inference server. Uh, a lot of these folks who built an advantage in sort of inference serving, you know, which is a software advantage ultimately, are also thinking about how they can have more control of their destiny on the compute side. Um, obviously I'm biased, but I think this is where the kind of neo chip story and the, you know, the differentiated silicon can actually become a really core differentiator for those players as well.

Harry Stebbings 9:15 Can you unpack that? And what

Harry Stebbings 9:16 is that?

Walter Goodwin 9:17 So we're still in the very early innings of the kind of rollout of silicon that is not the mainstream conventional, you know, GPUs and TPUs and other XPUs. Um, and it is still much bolder, especially if you look at the kind of way that these build-outs get financed to lean into, you know, a new chip platform, um, a Fractile, a Cerebras, and so on.

Harry Stebbings 9:42 And we need these new... Sorry, just to bring everyone along with us. We need these new chip platforms because workloads are becoming more and more complex and intense, and s- the specialization of these chips will allow for the execution of these workloads to be much more efficient.

Walter Goodwin 9:54 I think these new platforms, you know, obviously we're one of these, you know, have a belief that in certain, certain workloads you have this kind of more than 10X advantage, whether it's on cost or in our case it's cost and speed. Um, and so, you know, in a sense they are contrarian bets still. You know, we still don't see many gigawatts of new silicon built out. Um, but they do kind of have this potential to fundamentally alter the economics. And so if you imagine you are a neo cloud and you're the first neo cloud to work out how to break out of, you know, a particular bond with a particular GPU vendor, uh, potentially you get to reap this enormous benefit. And we've seen this with, uh, you know, Fluid Stack is a great example, right? They were, uh, in some ways the first neo cloud to kind of, uh, decide that they were going to step, uh, bodily away from Nvidia and they deployed a bunch of Google TPUs, and they've had an extraordinary year, um, as they sort of crossed that Rubicon and earned the right to start deploying, you know, other chips. Um, and so I think there are opportunities to, uh, you know, as you kind of go more vertical, also take some of these sort of slightly, uh, you know, higher risk bets, I guess, and, and, and, and reap huge benefits

Walter Goodwin 11:10 from that.

Harry Stebbings 11:11 So if we think about that, what does that do to the Nvidia business? If we think about kind of Nvidia as a horizontal provider and you as a kind of specialized provider, is the future of chips specialized according to workloads, and how do we think about, like, that market composition horizontal versus specialist?

Walter Goodwin 11:28 Yeah. I think the way that I think about this, you know, Nvidia is an extraordinary business and they build by many metrics i- in a sense the best AI chips in the world, right? And the best systems, and they have extraordinary work on data center scale inference, for example. So in the inference space I think they've done some, some brilliant work as well. What I think is different about this moment in time and, and, and sort of core to Fractile's identity is there is this idea of kind of an evolving frontier. And, you know, the frontier necessitates slightly different sort of bets and a slightly different way of thinking because you are... Necessarily setting up a business that is able to kind of look around corners, is able to make bold predictions, and is then able to steer the entire tech product, for example, towards those bold predictions. And I think if you look at Nvidia's business, what they excel at is delivering high-quality products at extraordinary volumes that are, in a sense, unopinionated, right? These are powerful general purpose processes that are, ah, very good at various parts of, for example, you know, the AI compute world. If you look at where Fractile's bet is on inference, for example, which is building chips that are orders of magnitude faster than these GPUs, you know, that's a particular focus. And for us architecturally, it's meant this focus on finding paths to huge memory bandwidth, for example. Uh, and it's potentially taken us away in some ways from some of that complete generality. Um, I think what's interesting though today, and the reason of course that we're so excited about, about this space, is the frontier is kind of also an overwhelming portion of the total market. You know, if you look at what's happened just over the last few years, we have this enormous build-out of specifically inference compute to serve LLMs at the frontier. Um, and so, you know, what is in some sense specialist is also by no means a niche. Um, and I think that's this sort of real transformation. That if you look back, you know, 10 years, even 5 years, uh, it would've been slightly crazy to build a kind of specialist chip, right? We didn't have any single workload that made up sort of more than, you know, let's say 10% of the total compute burden. Suddenly we do, and I think that demands slightly different company structures that can have that kind of predictive power built in that try to also orchestrate the entire org around what it means to, to chase after that.

Harry Stebbings 14:08 There's so many elements there that I, I wanna unpack. We, you mentioned memory and HBM there. Why is the memory wall such a prominent problem still today? Can you help me understand that?

Walter Goodwin 14:18 Definitely. Yeah, it's a, it's one of these sort of interesting facts about kinda chip physics, right? If you look at, um, the two things-

Harry Stebbings 14:27 All great days start with that line [laughs] .

Walter Goodwin 14:30 I know [laughs] . Uh, so if you look at, if you look across the last, like, 20 years in, in the silicon world, um, there's sort of two things that really make up the kind of top-level specs, the kinda top trumps of your chip, right? And like, one of them certainly for AI chips is the flop count, how many floating point operations can you do per second. This is like a multiply or an add. Um, and so that's your kinda mathematical power on that chip. And then the other piece is, is kind of the properties of the memory on that chip, the capacity and in particular the bandwidth to that memory. Uh, and that's how quickly you can, you know, shuffle data back and forth from, if you're looking at a GPU, you actually have that silicon in the middle of it that is running the maths. It's got that high flop count. And then there's other pieces of silicon around it, the HBM, uh, that are kinda coming in on this, on this sorta highway bringing data in. And we've been extraordinarily good with Moore's Law at scaling flop count. We've scaled at about a millionfold in the last 25 years on these chips. Um, bandwidth by contrast is kind of much more intrinsically physics limited. We've scaled at about 40-fold in the same time. So, you know, it's lagged behind the flop scaling about 25,000 times. And so we've naturally found ourselves more and more kind of bandwidth limited in a lot of the operations that we do. And for something like LLM inference, this has become a kind of critical bottleneck because of this interplay between the generation of text, which I think we all sort of have this, uh, sense that it's this sort of ultra-aggressive, you know, sequential thing, one word after another is generated out of these models. Every time I go and generate one of those words, I'm loading from that memory over that limited bandwidth all of the weights of my model, and today also all of the context that I'm operating on as well, so all of the past states of things that I've seen. And it's that that is then my bottleneck on generation speed. You know, the reason that if you take a very large frontier model, it might generate at 80 tokens per second, 80 words per second, is going to be not because of the limited maths capability on the chip, but because you don't have nearly enough bandwidth to that memory. I cannot load the weights of the model from memory into that processor more than, let's say, 80 times per second. Um, and the reason I started with, you know, the, the sort of sexy line about physics here is it is, if you zoom in at the interface, there's a bunch of essentially copper wires that are going from this DRAM memory into the processor, and we've been pretty good, the world's been pretty good at making them smaller and smaller and smaller, but there's only, there's sort of so much you can scale that. And so you have, let's say, a few thousand wires-

Harry Stebbings 17:12 And is that why, sorry, you have SRAM? Am I right in saying that?

Walter Goodwin 17:15 Mm. So-

Harry Stebbings 17:16 And m- moving memory closer to it.

Walter Goodwin 17:18 So SRAM, yeah, so there's this sort of, um, you know, DRAM is, like, often a different building basically, and you're, you're carrying these boxes down the stairs and then up the stairs again into the, into the processor building where the maths is done, and you have to carry them back again unfortunately 'cause there's not enough space in that building, let's say, to, to store all the boxes. So actually maybe it's a warehouse, and then you've got the office where you're doing, you're doing the work. Um, SRAM is like having a bunch of boxes that are permanently in, in, in, in the, in the processor, right? So it's a small amount of memory that you cram onto the same silicon, um, where you're doing your processing. Um, but that's the, that's the snag as well. The reason that over the last 30 years we've externalized memory into this, this thing called DRAM and put it on different silicon, these are built in completely different foundries. They've been specialized for very, very different things, so DRAM is completely focused on density. The DRAM bit cell, as it's called, is an incredibly simple structure, and you can cram, uh, you know, many billions of these into a very small area of silicon. SRAM, by contrast, is built Conveniently on the same exact process nodes, you know, it's built on the silicon that you're building your processor by TSMC, for example. Um, but it isn't optimized for density. So you might have, you know, less than 10 times as much kind of SRAM capacity per unit area as you can get sort of gigabytes of DRAM on that same area, and by extension, per dollar as well, because your silicon wafer costs you a certain

Walter Goodwin 18:50 amount.

Harry Stebbings 18:50 When you look at inference and the future of inference, which is then best suited? Is it DRAM or SRAM? 'Cause latency's important.

Walter Goodwin 18:58 That's right. So latency and, and bandwidth as well.

Harry Stebbings 19:00 Yeah.

Walter Goodwin 19:00 Like, how much can I just, uh, cram through this? So this is one of the things that we've been working on over the past couple of years at Fractal is, I guess, the rejection of this trade-off. Um, so SRAM as a memory can, uh, is the only one today that can be built on the same processor. It's the only one that can be incredibly close and have incredibly high bandwidth. DRAM, much cheaper, much denser, much more capacity, but fundamentally limited in the way that we access it. But if you look at that, that sort of physics, the fact that it's coming in over wires, it's because we've kinda got it in the wrong place, right? We've got the DRAM sort of sitting in plane with, uh, with the chip, and we've got this, like, limited, it's called a beachfront, right? It's, like, literally the limited kind of real estate on the edge of the chip. Um, what we've been working on at Fractal over the past couple of years is, uh, essentially stacking that memory on top of the logic. So looking for ways to, rather than have this kind of limited beachfront, you know, everybody gets to have a, a seaside property, basically. It's like the entire memory layer sits on top of that logic die, and suddenly your interface-

Harry Stebbings 20:09 How is that possible?

Walter Goodwin 20:11 So th-3D in, in the chip world, it is a dimension that people have been scaling for a long time. Uh, but it's got all of these complexities. So actually, if you look at an HBM, a high bandwidth memory, it is a stack of DRAM chips. Um, but it's a stack that has certain properties that, you know, don't make it amenable to just being taken and, and plonked on top of, uh, your logic die. And so we don't do that today. But there are these process technologies, and this is something that sort of, you know, Fractal's team has a long hinterland in around, uh, 3D stacking, um-

Harry Stebbings 20:47 So you're building condos on the beach instead of beach houses to allow for more capacity.

Walter Goodwin 20:53 I guess we're building one of those kind of, uh, you know, new-built archipelagos where everybody has a bit of beachfront, right? It's, uh, it's the, uh, you know, uh, strips of land. Everybody's right next to a piece of coast. When you put this on top, suddenly you have this whole two-dimensional area where you can have essentially those copper wires are now not going laterally. You have millions of them coming straight down.

Harry Stebbings 21:18 When you look at differentiation between other providers, is that your differentiating factor? And how do you analyze the, the specialized providers that also compete with you in this realm, like an Etched? Like, like what is the differentiation?

Walter Goodwin 21:31 Yeah, I think there's a few ways to, to look at this. You know, one is, one is sort of philosophical, and then the other is technology today. So I think philosophically, um, I do think that Fractal is, you know, we have a mission, right? And the mission is to advance frontier AI capability through building better hardware. Um, today that's taken us over the past, you know, past couple of years to this particular solution, and I think this, uh, extraordinary work that the team has done in unlocking a future where you can have the bandwidth of SRAM, but with the capacity and the scalability of DRAM, it is a world-changing kind of innovation. Um, and it's going to have longevity as well. You know, we're, we're very enthused about taking this forwards over multiple generations and kind of compounding a, a technology advantage there. Um, but at the same time, you know, I think with a, with a mission, if the workload changes, you also want to be able to, to change your response. And so, you know, I think one thing which we're cautious not to become is the kind of, you know, the company that cracked 3D DRAM. Um, we want to be challenged to continue to innovate on, you know, whatever comes next. If you look at mechanistically what it does mean for us though over the next few years, there is this sort of challenge today if you look at the chip space where on the one hand you have all of these HBM-based chips, um, and at some level it's very hard for them to diversify. You know, they are-- Whether it's, you know, the GPUs, TPUs, the Jalapeno chip from OpenAI, um, Etched, Matix, you know, they have, uh, often really brilliant kind of architectures on the chip which allow them to do things slightly more efficiently than one another and so on, but they are fundamentally kind of, uh, boxed in by the, like, Top Trump specs, right? You can get only so many FLOPS and you can get only so much bandwidth. The, uh, HBM that sits off to the side of them is exactly the same piece of silicon. It's the same module from the same vendors for each and every one of those chips. Um, and then on the other side you've got these SRAM-based chips, uh, that have kind of high memory bandwidth, but also have this huge capacity constraint, right? And so if you look at, you know, if you look at Groq or you look at Cerebras, what they can do today which is extraordinary is allow you to run powerful models much, much

Walter Goodwin 23:52 faster. One of the things that I think certainly I've always been excited about with Fractal, it was sort of w- what I set the company up to do four years ago, is build chips that are better at LLM inference, but in particular build chips that allow us to run That inference much, much faster because we always believed that we would get to this space where we had what we actually do have today, what, what's called test time compute scaling, right? Reasoning models, long-running agents that run for longer and longer to get better and better and better at what they're doing. A, a, you know, a better answer for you. Um, and this is the kind of profile of all of the frontier workload that's sort of changing the world today. You know, if you look at what happens when you turn a frontier LLM to the problem of solving a Millennium Prize, uh, problem, you know, it will go off and generate many millions of tokens sequentially, and you scale up this swarm of them as well, so there's a sort of parallel element to this. But the many, many millions of sequential token generation means you're waiting days and days and days to get one of these responses. Um, and it's a, it's a space which somewhat tantalizingly hasn't been cracked at all by any of the silicon, any of the chips that we have today, because there's this sort of tantalizing mismatch, I guess, between the fast inference chips, the SRAM-based chips, um, that have the ability to, you know, access memory quickly, generate tokens quickly, and the needs of this new workload, the kind of long-running agent, which needs a huge amount of capacity because you have these really long contexts that are emerging. And so the world urgently needs this kind of, uh, up and to the right thing, where you have simultaneously really high bandwidth and high speed, but also high capacity, so the ability to scale to longer contexts and larger models.

Harry Stebbings 25:45 Is this one of those where it always takes longer than you think, or is this one where you see dramatic inflection points in the acceleration of technology?

Walter Goodwin 25:55 I think what's exciting about this one is, and this is true, I think, often of, of sort of, you know, technological breakthroughs, you are always standing on the shoulders of giants. Um, in our case, you know, some of the technology that we're leveraging to do this 3D integration, for instance, has actually been around for quite a long time. It's just not been turned to this exact problem, to the, the memory stacking challenge. And so it's a space where if you're willing to do the, kind of both the architectural effort and redesign your chips to make use of this, and then you're also willing to do the kind of, uh, manufacturing operations legwork, and you're able to persuade your partners that you're sort of not crazy along the way, there is actually the possibility here for these kind of overnight, overnight transformations.

Harry Stebbings 26:43 Can I be blunt? I was talking to a founder of a, a brilliant chip company last night, and he said there's a huge amount of hot air and, and BS in the industry, and there's a lot of marketing fluff, but there's not much actually in production. When you look at your etched or your Fractiles, are we actually in production? Like, how do we actually think about on the ground

Harry Stebbings 27:06 being in production, and is he right with that statement?

Walter Goodwin 27:10 There's two parts to this. You know, yes, Fractile has silicon that's gone through, uh, the foundries, has come back to us. Um, yes, we're running further trial runs, and we'll be doing sort of full product tape outs, uh, in a few months as well, which is quite exciting. Um, I think the scale of production is something which is very much worth challenging, right? There is a, uh, real challenge, especially for kind of new chip entrants at the frontier, um, in accessing, for instance, TSMC capacity, HBM capacity. Um, and I think in particular, you know, for some of these efforts, there is a race for exactly the same resource. You know, a lot of these chips today are built with the same backend partner, for example. You know, whether it's the TPU or Jalapeno, MTIA from Meta, uh, Matix or Etched, um, you know, all of those-

Harry Stebbings 28:03 And that backend partner is who?

Walter Goodwin 28:04 ... are rumored to be using Broadcom, for instance. Um, and, uh, same or very similar TSMC process node, same HBM technology, and so you can see why there might be, like, a, a real kinda gold rush for that particular finite resource. And I think that means that it can be challenging to get to really meaningful scales in a short space of time. And one of the things that we've sort of, I guess, always been quite focused on at Fractile is because we've had this sort of almost unfair reach into the stack, right? The, the, the point of a backend partner is they do the kind of last mile, the physical design. They interact with the foundries, with the, with the manufacturing side of the business. Fractile's always had that in-house, and we have this extraordinary team that has kind of always worked that way, you know, even before Fractile, for, for decades in how they've delivered their chips. Um, and it's become a really core strategic advantage for us because it means we've both always had direct communion with our manufacturing partners. Um, w- we've had to do that legwork from the outset.

Harry Stebbings 29:06 And just so I understand, the alternative is you go through Broadcom, who then have the manufacturing partner relationships.

Walter Goodwin 29:11 That's

Walter Goodwin 29:12 right, yeah.

Harry Stebbings 29:12 Got you.

Walter Goodwin 29:12 And-

Harry Stebbings 29:12 So Broadcom is the intermediary layer that most people work with that you don't.

Walter Goodwin 29:17 And that we don't.

Harry Stebbings 29:18 Yeah.

Walter Goodwin 29:18 And so I think there's, there's two things-

Harry Stebbings 29:19 What does that enable you to do then?

Walter Goodwin 29:21 So one thing is build completely different kinda building blocks for these chips, right? I think if you look at a Broadcom, uh, you know, I was at a, a conference this summer where, uh, Charlie Kawwas, who's, who's, uh, you know, one of the leaders there, held up a kinda LEGO model of the next gen platform. And he said, "This is gonna be in production with six major customers in 2028," which is a sort of strange thing to say, 'cause what he means is something that looks identical to this is gonna be made with six of the major players and given different names. It'll be the TPU, MTIA, Jalapeno, you know, 3, um-

Harry Stebbings 29:59 Dorito 2.

Walter Goodwin 30:00 Exactly. Um, and, uh, and so it sort of revealed something about the structure of this market, right? Where each of those chips will have a different name. They will also be different. There are incredible chip teams inside all of those companies that come up with, uh, insanely clever kind of architectural optimizations, but in the end it's gonna be exactly that same platform. It's gonna be the same stack of HBMs, the same number of chips. A lot of the sort of fundamental building blocks are dictated by the backend partner, because it's a platform technology play. You know, it's a kind of you get to change the color of the paint, basically.

Harry Stebbings 30:37 [laughs]

Walter Goodwin 30:37 Um, and so for us, what it's allowed us to do is, like, maneuver and build these new building blocks. So the memory stacking work, for instance, is I- incredibly complex and painful and requires convening multiple, you know, much larger supply chain partners to get them to do something that is kind of highly custom. But once you crack through that, as we have, you know, you then have an asset that nobody else has. And this was a slightly long-winded way of getting all the way to your question about, like, volume production that also, more likely than not, has meant that you've gone and found a way to use resources that others haven't. So in our case, a different process node, maybe a slightly different DRAM node or DRAM generation as well that might be going into frontier HBM. Um, and so you sort of find, you know, the opportunity to build gigawatts of capacity sort of down the back of the sofa with things that other people in this sort of, you know, set of frontier platforms have already moved away from.

Harry Stebbings 31:40 Can I ask you, when we think about the kind of production line, so to speak, there, a lot of people told me that I need to ask about tape-out. What, what is tape-out? What do we need to know and how, how do you think about it?

Walter Goodwin 31:51 So tape-out is the moment where, uh, the final file from your chip design process, which is actually basically a, it's like an enormous image. We call it a GDS2 file, and the image dictates, you know, where are all of the copper traces, where are the transistors, where's the logic. That gets sent to, to TSMC or to your, to your logic foundry partner, and they commence the process of building the mask sets, the kind of stencils that will ultimately produce the chip, and then you start doing your, your first kind of trial runs, your first production runs as well, potentially. Um, so it is a moment in time that suddenly if you're a chip designer, is a bit of a kind of seminal handoff, right? Because at the end of this long process of architecture, front-end design, verification, physical design, layout, you send the file, and if there's a bug in it at this point, you know, it's too late. The mask is getting printed. Uh, and so that's why it's kind of a, a, you know, it's a major date on the calendar sort of thing. It's also, though, just sort of today, step, step one in, in quite a long process, right? Because that silicon then goes through the fab and you wait months for that to come back, and you have to be poised and ready then to receive that and run, you know, your bring-up process. You know, this package silicon returns to you in the lab. You then sprint to get, you know, a workload running on a single chip, on a tray, eventually on a rack as well. Um, and so that's sort of part two, I guess, is that, um, you know, at Fractile, uh, we track progress. You know, there's tape-out dates in the middle of it, but the thing that we're continually shooting for, uh, which we call task zero, uh, is, you know, the day that a particular reference workload runs on an entire rack of Fractile chips.

Harry Stebbings 33:40 What does no one see about this process that is most prominent to you? If we think about tape-out being like a seminal moment-

Walter Goodwin 33:47 Mm.

Harry Stebbings 33:48 -handoff, where is the, "God, no one quite understands the difficulty, the momentousness of the post tape-out date"?

Walter Goodwin 33:56 Mm. I think, well, I think that's it, actually, is it's what comes next because there is an awful lot that, uh, you can fail to prepare yourself for if you're ultra fixated on that one thing, the kinda handoff of, of the first chip to the, to the fab. Um, and so I think, you know, for us, it's continually tracking, you know, what is, what is it that we're actually trying to build here? We're trying to build systems that yield well, so the underlying economics are great because, you know, we've, we've designed silicon that has the right levels of repair and redundancy. It's, you know, highly manufacturable. It makes a lot of sense as a product and then fits into a system that, again, yields well, is reliable. Um, and so the sort of building of a product that is manufacturable in volume, where you can then find a path to ramping as quickly as possible, you know, that's, that's the goal in some ways. Um, and I think tape-out is a, um, uh, I almost feel like maybe over the last few years there's been this sort of, uh, accidental, uh, shared obsession between sort of chip startup founders and ven- the venture capitalists that are backing them on this phrase, 'cause it's such a tangible moment. It's such a tangible date. Um, but a lot of the work that we do at Fractile, you know, all of the work we do on building software, all of the work that we do on systems and a lot of the silicon team as well is focused on those parts of the chip that in a sense are kind of like intangible, ineffable, but are about what it looks like to be able to get this stuff up and running and reliable and to volume as

Walter Goodwin 35:39 quickly as possible.

Harry Stebbings 35:40 When we were downstairs, you said that actually silicon may be, may be no longer the bottleneck in the same way that it was before, and maybe data centers were or data center capacity was. Can you talk to me about that relationship and the, the transition of shortage versus not?

Walter Goodwin 35:55 Yeah. Yeah, it's been-- I mean, this is why the last four years have been so exhilarating because it feels like every quarter, uh, the thing that is most constraining, let's say, our ability to run more front-end models at scale in the data center has shifted. Um, there's these underlying properties that we urgently need, like the ability to run inference much, much faster, the ability to run at longer context with better economics. Um, but then we've had this sort of, you know, if you think back three years ago, there was no question of other chips. It was an Nvidia GPU shortage. Uh, and then I think we found ourselves in a underlying silicon and supply chain shortage. You know, acute memory shortages today for HBM and so on, which stretch years into the future. Uh, logic silicon shortages with, with TSMC as well. Um, and I think, you know, if you look at the space today Land and power is definitely another limitation. You know, we've had all of the kind of low-hanging fruit for industrial sites that can be retrofitted to be data centers, um, places where, you know, in the US you can get permitting to bring gas turbines close to spin these things up in, you know, 90 days. Uh, these kind of often one-time tricks have kind of all been exhausted, and so there's sort of... We're now exploring the kind of longer tail of, you know, harder, harder ways to find land and power to install these chips. And so on the, you know, one hand, actually it is data center capacity that is our kind of, uh, fundamental bottleneck on how much compute we can build out. And so this is the kind of, uh, you know, if you want to be completely agnostic and invest in a, a, you know, your long AI, you know, buy, buy energy basically, because that is obviously the kinda most fundamental. Um, but at the same time, it's precisely that kind of constraining limit that is why it is so vital that we have the best possible chips for any given job. If you look at, um, uh, frontier labs and hyperscalers today that have like a fast mode on their models, uh, you know, if you're running that on an HBM-based chip, um, you're doing so at this kind of extraordinarily inefficient point on a kind of trade-off curve. On these chips, you can either be sort of slow and efficient in generating tokens or, uh, fast but, but deeply inefficient, sort of far fewer tokens per second per megawatt of power, for instance. Um, so what happens if I want to take a multi-trillion parameter model and run it at thousands of tokens per second, uh, but I have a 500-megawatt data center and I don't have enough aggregate throughput, you know, I can't serve my user base on those chips. There's a real pressure to now install a set of chips that do retain, let's say, really good aggregate throughputs even as you squeeze them to, to much, much higher speeds. You know, Fractile would be an example of that. And so I think the, uh, although it's land and power that may be the kinda most acute bottleneck, even ahead of kind of supply chain constraints, it's actually exactly that that kind of incentivizes, uh, chips that are much, much more fit for purpose.

Harry Stebbings 39:09 How much of an energy deficit do we have? I know it's an incredibly macro and bold question.

Walter Goodwin 39:13 Mm.

Harry Stebbings 39:14 But when you look at energy requirements versus energy supply, what do you think the deficit looks like?

Walter Goodwin 39:19 I think if you look at backlogs in the US and the UK, it's this extraordinary, um... You know, the queues for grid connection will be like an order of magnitude more than there is energy in the grid. Um, so viewed in that way, you know, and the extent to which all of those are actually legitimate projects that would get off the ground is obviously something that we can, we can question. Um, but it does look like there is this, this staggering level of demand. Um, and I think energy is one of these real

Walter Goodwin 39:50 fundamentals.

Harry Stebbings 39:51 Going back to what we said earlier about DeepSeek and their incredible innovation on their inference stack, when you look at China's policy and regulatory environment towards energy, I mean, it's green light anything. [chuckles]

Walter Goodwin 40:03 Mm.

Harry Stebbings 40:04 It's green light anything, including moving millions of people out of their homes because they need the sites for energy provisioning.

Walter Goodwin 40:09 Mm.

Harry Stebbings 40:10 And in the US you have NIMBYism, and in the UK you have, you know, um, bats and butterflies preventing sites of scientific or natural beauty. Um, are we not just inherently disadvantaged in a world of energy requirements versus China?

Walter Goodwin 40:24 I think this is one of the places where China has, you know, has somewhat stolen the march on the West, right? Uh, and in particular, you know, these mega projects, you know, huge hydro projects, for example, they're ideal for AI compute build-outs because they have this particular kind of signature where, you know, it might've once been kind of inconvenient to have 15 gigawatts of power all arriving at a single site that then needs to be distributed. You know, an AI data center doesn't really care where it's located. Um, and so why not go put 15 gigawatts of AI compute right next to this, you know, hydro plant in the middle of nowhere? Um, and I think that there is, you know, in the US and in particular in the UK and in Europe, a lot that we can do to look a little more like that. And I think that the good news here, and this is something which, you know, uh, I, I think the government in particular, you know, here and, and, and y- European governments should be doing more to, to play up is it's actually exactly that fact. It's that a lot of the power sources that could be brought to bear on this are, um... You know, it... Renewables can be a huge part of this story, right? You can locate a data center, you know, in, in, uh, in Scotland next to an enormous wind farm, where previously maybe there just would not have been a story as to why you wanted to have energy generation there.

Harry Stebbings 41:46 Do you know what's really interesting for me is we, we often, we do a lot of Instagram now in short form-

Walter Goodwin 41:50 Mm

Harry Stebbings 41:51 ... and the minute we talk about data centers and them not happening, we did it the other day with Microsoft pulling out of the Enscale project-

Walter Goodwin 41:56 Mm

Harry Stebbings 41:56 ... um, everyone goes-

Walter Goodwin 41:59 Good

Harry Stebbings 41:59 ... "Good riddance."

Walter Goodwin 42:00 Yeah.

Harry Stebbings 42:00 "Good riddance."

Walter Goodwin 42:01 Yeah.

Harry Stebbings 42:01 "You tech bros, good riddance."

Walter Goodwin 42:03 Yeah.

Harry Stebbings 42:03 And it was astonishing to see the universal backlash-

Walter Goodwin 42:07 Mm

Harry Stebbings 42:07 ... from bluntly moronic civilians.

Walter Goodwin 42:09 Mm.

Harry Stebbings 42:10 Can I ask you, you, you saw inference as a market before so many other people. I wish I'd listened years ago. The thing that I know now that I wish I'd known 12 months ago is just the infinite demand for inference.

Walter Goodwin 42:22 Mm.

Harry Stebbings 42:23 What do you know now about the market for inference that

Harry Stebbings 42:26 you think isn't that obvious, what you hear other people say?

Walter Goodwin 42:28 Mm. I think the thing that remains, uh, in some ways it's a s- it's a even more pronounced version of, of sort of my original bet. So, you know, I think four years ago the insight was sort of twofold. One is as you start to get these kind of foundation models that have been trained on everything and then generalized to anything you might turn them to- Of course, you have this kinda hard logic where training becomes a fixed cost, and the deployment, you know, inference is just the use of these models to do useful things. Um, it's the only part that matters. If you were spending more on training in the long run, like, why are we even building these things? You know, you really could kinda pack up and go home. Uh, you-

Harry Stebbings 43:12 I always describe it to my mother like kind of the Olympics and Usain Bolt. You know, he trains in the gym, great, and then decides, "Ah, I'm not gonna actually do the race."

Walter Goodwin 43:19 Right. Right.

Harry Stebbings 43:20 What's the point, dude?

Walter Goodwin 43:21 Exactly.

Harry Stebbings 43:22 This is why we're here.

Walter Goodwin 43:23 Or, or it's like imagine you spent far more money building a factory than you ever actually recoup from generating something out of it, right? It's, it... The, the hard logic of, uh, building something with a high fixed cost is that you must believe you're gonna be doing enough deployment to justify that. And so, uh, you know, I think that was sort of, was, was part one, and then this bet that we would have, you know, what we call kind of test-time compute scaling. The advent of reasoning models, they didn't arrive until two years later, for which I'm, you know, very grateful to, uh, Noam Brown and the team at OpenAI that was the first to kind of, you know, publish one of these. Um, uh, I think what's happening today is, uh, we continue to see this sort of vector of these, uh, test-time compute kind of scaling laws, right? It's this stack of different ways that we pursue greater capability. So one is, you know, 10X-ing and 10X-ing the amount of compute we use to train these models. Um, but the other today is how do we, how do we pour more compute into deploying these models into the inference side? And it's a huge part of how you get to greater capability. You know, if you look at, again, the, uh, uh examples of solving a Millennium Prize problem, that Navier-Stokes', uh, uh, solution was, I think, 80 hours of execution from 10,000 agents in a swarm. And so you have there these kind of two axes that trace out this enormous kind of volume of token generation, right? Where I've both got 10,000 parallel agents all generating their own chains of thought, and then they're running for 80 hours. Um, and so it is so different to the kind of intuition that we might have about an LLM, where we sort of think we ask a question, and the single-threaded thing will go and generate almost immediately an answer. When we bring these models to bear on the hardest problems, and I think generally the hardest problems, and this is a bit of good news, you know, are also the ones that have by far the best, you know, the most positive societal impact. It don't- generally, it is solving a science problem or, uh, you know, something in medicine that we maybe haven't been able to solve before. Um, to go and do this, where you have this incredibly economically and societally beneficial thing to go solve, which is beyond the reach of kind of just, you know, applying the model, uh, and asking it one shot, you know, "What do you think? What's the answers to this?" Inference compute is the axis that we have, and the Navier-Stokes example is, is, is kind of evidence of that. And so, I think the thing that is-

Harry Stebbings 46:02 What do you mean inference compute is the axis that we have? It's the axis that we have to-

Walter Goodwin 46:07 It's the, it's the-

Harry Stebbings 46:07 ... do the work

Walter Goodwin 46:07 ... it's the knob that we have to turn to get greater capability. So if you imagine that at, at any moment in time, we have in the world the best frontier model. It will come from one or other lab at, at any point in time. And the best frontier model is extraordinarily capable. You talk to it, and it's smarter than any you've spoken to before. But it turns out with... when you ask it, you know, "Hi, I'm working on this, you know, I'm working on this challenge. We're really trying to find, um, you know, a, a cure for this rare disease," we're not at a place, and of course we're not at a place, where it's just gonna say, "Oh, here's... Here it is." Like, "I've thought of this. It's in, you know, in, in my neurons, this is, uh, fired out immediately." But the thing that we also discover is if you do throw enough compute at it, if you say, "Hey, how about rather than just trying to give me an answer, you propose, you know, many hundreds of candidate expert, you know, s- potential solutions that you could go and explore. And you create multiple copies of yourself, and you go and traverse the internet, and you read the literature, and you debate with copies of yourself, and you exchange information and come back to me in, you know, basically a week's time." Once you've generated maybe billions of tokens in pursuit of finding an answer, you know, then a little bit like if we give a bunch of smart people a lot of time to solve a problem, you do have a chance that you've come back with a solution. And, and we really see this. This is the key. It's what every single application of a frontier model to solve a hard problem is going to look like.

Harry Stebbings 47:44 M- my question to you is, I, I am a, a venture capitalist, I am a investor, and I see the crunch, the debt cycles, the credit, the problems that we are hitting now and about to hit. There is not enough money in the world for the requirements of this cycle and, and everything that you just said there, which I agree with, and then I look and go, "And then we haven't addressed the physical world and physical data and robotics and everything in between that's required there." And so I'm holding these two seemingly opposing thoughts in my mind going, "What the fuck happens

Harry Stebbings 48:21 now?"

Walter Goodwin 48:21 Right. Yeah. Well, this is why we need continuous innovation in every part of the stack that leads to serving intelligence, right? And the good news is, you know, for all that obviously we're focused over here on chips, you know, if you just look at the algorithmic side, like model efficiency, the cost, assuming no hardware change to serve a given level of intelligence, is coming down precipitously quickly.

Harry Stebbings 48:44 Deep sea ATB-scale improvements in efficiency with no hardware change.

Walter Goodwin 48:47 Yeah. And as we see, you know, longer context becoming a challenge, we see incredible innovations on how you can compress context and do sparse attention and other ways of handling this. So there is sustained innovation on what you might call kind of, uh, the compute efficiency of the models, you know, at With the same chip at ISO compute, you know, how much more intelligence, how many more solutions can I, can I arrive at? And then of course, we do have the same on hardware, and we're working very hard to make sure that we have that hardware effect as well. And so I think what's, what's really thrilling, and it's the reason that I think on the whole, you know, this build-out is entirely justified and we won't see, you know, a, a collapse in this market, is that, you know, far from the, uh, fiber build-out of 2000, 2001, you know, we have complete full utilization of the GPUs that are powered on in data centers. We don't have dark GPU fleets because as you've seen, you know, we have this enormous number of problems as a species that we still have to solve. We now have tools that we can throw at those problems. Um, and we have many orders of magnitude more thinking that we could be doing at sort of inference time to solve the hardest challenges if we had the compute. And so, you know, I agree with you on the one hand, it feels like we need thousands of times more compute than we have today. We're not going to get thousands of times more, uh, data centers, and I think that's probably a good thing for the world. What we will be able to do is take, you know, roughly that kind of land and power and set of data center sites that exist today, and over the next few years, through both model innovations and hardware innovations coming together, squeeze more and more and more, dramatically more intelligence per megawatt, per rack in one of those data centers.

Harry Stebbings 50:41 So I had Mike Cannon-Brookes on the show from Atlassian, and, um, he said to me, uh, "You know, I think the cost of inference is gonna come down dramatically over the next three to five years." And that's why I'm-

Walter Goodwin 50:54 And I'm pretty good at you

Harry Stebbings 50:54 ... that's why I'm nodding at, um, uh-

Walter Goodwin 50:56 Yeah

Harry Stebbings 50:56 ... you should-

Walter Goodwin 50:56 You can do me in the next one.

Harry Stebbings 50:57 Uh, well, I mean, you in the next one is relatively close to just me. [laughs]

Walter Goodwin 51:01 [laughs]

Harry Stebbings 51:02 Uh, you, you want me doing Russian. That really-

Walter Goodwin 51:04 Mm

Harry Stebbings 51:04 ... is a good one. But with the Mike Cannon-Brookes theme in mind, um, what happens to the cost of inference?

Walter Goodwin 51:12 So I think there is... You know, if you look at the cost, what, what will a token cost in 10 years' time? Um, it's one of those sort of questions that, you know, begs another question, right? Like, what is the model that we're running? It's probably orders of magnitude larger than the model that we have today. And so that token is also delivering in some ways, you know, substantially more capability. Um, I think the margins in inference is something that will, you know, will fluctuate and they will accrue in different places. We will be spending far more on running models in 10 years' time than we are today. You know, that part I think is, is clear. Um, but who is gonna win in this space? You know, you've got today, I guess, you've got chips, you've got clouds, you've got frontier labs building frontier models. You've got folks that are building and serving open models. Um, and I think one of the reasons why, you know, I am very long on chips in terms of where that value accrues, or being one big place where that value accrues, is actually because of the nature of the challenge of placing the right chip bet. You know, it's, it's got this intrinsic kind of lag. That lag to design the chip, to get the chip out, to have it built, and then the degree to which it has to have a, a l- useful lifetime in a data center, uh, means that you don't have quite that same kind of fully efficient race, you know, potentially a race to the bottom in some ways, uh, in the chip space as you do elsewhere. If you have a better idea, if you can build some sustained advantage in some way, um, I think you see a lot of the value accrue there. Um, I think you also see a lot of it accrue still at this frontier. You know, the construction of frontier models which have this sort of sustained advantage over some of the others. It's a costly process. It requires incredibly talented teams working with huge amounts of compute. Um, but it is something which then delivers, you know, who doesn't want the greatest intelligence on Earth? Of course people will continue to, to pay for that. And so I think the overall kind of prize pool expands. Uh, there will be kind of relative winners and losers within that, but it is a, it's a positive sum game in inference.

Harry Stebbings 53:27 We just had Satya on the show, um, the other day, and he said that we would see a future where every company has their own specialized model with their own unique and proprietary data. And actually a, a cool kind of value segment would be evals and benchmarks for themselves-

Walter Goodwin 53:41 Mm

Harry Stebbings 53:42 ... and for their company and for their own objectives. Do you agree with that world of many specialized models on a per company basis? Do you think it's, it's relatively techno-optimist and, and wrong?

Walter Goodwin 53:52 Mm.

Harry Stebbings 53:52 How do you think about that?

Walter Goodwin 53:55 I think there's absolutely space today, and just if you look at the technology, there is positive impact in taking a model and fine-tuning it for your use cases. But it's a very, it's a transient thing, right? Like, generally the advantage of having something that is, you know, internally fine-tuned will disappear as soon as the next release or the next open model arrives. I think that can still become a bit of a kind of yes, and, right? Like, as long as you have a model where this, as in a, you know, a model of construction these, where this stacks on top of the best frontier capability, um, then I can absolutely believe that there is, you know, a, an advantage to having a kind of an internal, a specialized variant of something that is otherwise constructed at the frontier. Um, but I do think this is one of the, the hardest questions to answer, um, because over time it may well be that, you know, you can only do this with the open models, for example. Uh, and if that's the case, um, you know, what does it mean for, uh, you know, this sort of trade-off where is it better for me to specialize an open model or just to go all in on the thing that's been released, you know, at the frontier that is still pursuing these better compute efficiencies and by extension might actually be very cost effective for my

Walter Goodwin 55:11 challenge?

Harry Stebbings 55:12 Totally get you. So when you look at that value stack, you said obviously chips, no shit, where you think is the most valuable. Where is the least valuable segment?

Walter Goodwin 55:24 I think there is a danger for anybody that is working, you know, purely in the software layer. Um, I think, you know, I've seen a few, uh, plays recently on kind of orchestration. You know, there's a coming wave of heterogeneity in the chips that we're gonna have in, in the world. You know, GPUs, other XPUs, these new chip efforts. Um, and you know, some clouds and some software providers have set themselves up to be the kind of orchestration layer over, uh, over this sorta coming wave. And I think there's a world in which that kind of heterogeneity is, is a slightly transient thing. You know, that, um, yes, today, you know, we have, for example, uh, a world where Nvidia having acquired Groq will deploy two chips alongside one another. You know, Rubin GPUs and Groq chips, and they'll do what's called prefill decode disaggregation and attention MOE disaggregation, where these pieces of silicon are doing different parts of the LLM inference workload. And so there's this sort of sophisticated orchestration between those two.

Harry Stebbings 56:32 At the same time, it's kinda like model routing, but chip routing.

Walter Goodwin 56:34 Yeah, exactly. Chip routing and, you know, I think there's a kind of, um, kind of, uh, t- technologist view that maybe, hey, wh- why stop there? Why not have, like, seven different pieces of silicon? And maybe they're from different vendors, and they're each specialized in different parts of this. Um, and I think there's a belief that this might be a, a, a huge kind of wave of value creation will be in being specialist in how you orchestrate over these. I think a lot of that could collapse back down over time. You know, the, the, there isn't a fundamental existential reason why we desperately need to have seven different platforms to run a workload. Um, and there isn't really a strong economic case for kind of why we would be able to sustain that as well. I mean, if you imagine even just the, the Nvidia Groq, uh, data center deployment, how many Nvidia GPU chips do I put in there versus Groq chips? You know, the, the bet today would be, so this attention feed forward disaggregation, uh, if you'll give me a minute on the tech here-

Harry Stebbings 57:36 Oh, I will

Walter Goodwin 57:36 ... is a, uh, you know, I put the weights of my model on the Groq chip because I access them really fast on that high bandwidth, and because every user that I'm cramming through that data center, uh, they all want the same model weights, right? So I can kind of, um, uh, amortize, I can share the cost of that much more expensive memory. I can recoup it, make it up on volume. Attention, which is the part that operates over the context, uh, isn't shared across users. My context is different to your context. And so for that piece, I'm incredibly exposed to the cost of my memory, and we talked earlier about how SRAM is far too expensive. DRAM's much cheaper, so I need to fall back to the GPU and the GPU's memory to run that piece. And so that's the logic of, you know, an Nvidia data center with these two chips in today. But as I spec that data center, how many Groq chips do I put? You know, what if my context lengths get longer over time? Suddenly I probably wanted to have more Nvidia GPUs than I did last year, and the context lengths have got longer. How much longer will they get? Maybe the model gets larger suddenly, and actually I wish I had more Groq chips. Um, and so you're very likely to end up in a place where some portion of that capacity in your data center is stranded. You're just not filling it because you've got the wrong mix. Um-

Harry Stebbings 58:49 That's unacceptable.

Walter Goodwin 58:51 And it's completely unacceptable, right? Because you have a certain number of megawatts. You're trying to juice as much revenue as you possibly can out of them. And so this idea that you're p- placing a, a fixed blend, and you're not gonna be able to change that data center for, you know, the next few years, right? That's fixed. It's CapEx. It's being amortized over time. Heterogeneity is, is actually a really challenging thing, and I think it's a transient response to the fact that you don't today have a chip that can kind of do it all, right? These SRAM-based chips solve this fast inference challenge a bit, right? But they don't have the memory capacity to do attention, and so we still need to bring that GPU in. They also don't have the, uh, FLOPs density, back to the Top Trumps, to run the compute-intensive part of LLM inference, the prefill portion, the prompt processing, because, uh, you've got this one piece of silicon which cost you a lot, and you've had to share the area that you might put maths in with the area that you put memory in. So you definitely haven't got as much maths as your counterparty that used HBM, who could use 100% of that chip area for the tensor cores that run matrix multipliers. If you look at, let's say, what Fractal is doing, you suck the SRAM upstairs basically, and you have a, you have a DRAM layer stacked on top. All of that compute die on the base is now free to put tensor cores down. You can have as many FLOPs as anybody. You can be a Top Trumps maxer on that. But you also get the SRAM-level bandwidth from the 3D access, and you get the capacity from the DRAM. And so suddenly you have this kind of, you know, holy trinity where you've, you've got all of these things in one, and it becomes clear that you don't need heterogeneity and all of this complex orchestration and running the risk of stranded capacity.

Harry Stebbings 1:00:30 Was Nvidia's acquisition of Groq an insurance buy?

Walter Goodwin 1:00:33 I think there was a... There's so many fun rumors about this, right? Uh, like, I think the Groq team is an incredibly smart team, and Jensen is a smart guy that wants smart people on his team. Um, uh, they also did build a platform that I think answers or answered a, a big p- challenge that, you know, Nvidia faced at the time, which is, how do we move into the fast inference space? Um, I think, you know, I think Jensen is quite positive some in this space. Um, he said some nice things about Fractal last week when he was in London, um, you know, and, and is in some ways a bit of an ecosystem player. Um, the way that Nvidia, you know, captures the kind of bulk of this market is that, you know, they will sometimes be deployed next to other platforms, other platforms that augment parts of their capability. Um, there is another rumor that obviously Cerebras got a deal with OpenAI. Um- I think that can be an alarming moment if you're running a multi-trillion dollar GPU company and one of your largest customers suddenly signs on for something that you perhaps hadn't previously thought was existentially necessary, which is a fast inference platform, and what's your response? And I think both deals were signed on the 24th of December, uh, which must have been a fun time for, to be a lawyer as well.

Harry Stebbings 1:01:49 And so it's in- or being a lawyer is a brilliant time right now. Uh, and so you think it's a knee-jerk reaction to that diversification from OpenAI?

Walter Goodwin 1:01:57 I think it's a acknowledgement that some parts of this space at the time required a different solution.

Harry Stebbings 1:02:03 Mm.

Walter Goodwin 1:02:03 And this SRAM play from Groq was the right solution at that moment to be able to run LLMs faster. Today, though, we've already seen ourselves, I think, move on from this. You know, you cannot solve LLM inference with these SRAM-based chips. There isn't enough capacity. Uh, the cost to achieve speed is too high. The constraint on the context lengths you can operate on is very, very stark. And so, you know, I'd sort of characterize that solution as, uh, you know, it gives you sort of snappier chatbots, but it doesn't necessarily give you this thing of being able to take kind of overwhelming frontier intelligence, these agents that run for days on end, and truncating that to be orders of magnitude faster. I think that really is where the huge prize is. Um, and today that's not something that that

Walter Goodwin 1:02:52 provides an answer to.

Harry Stebbings 1:02:53 So, um, the different players in the inference space, you know, that people know are your Modals, your Fireworks, your Basetens, your Togethers. You would bet on one. Which one would you bet on?

Walter Goodwin 1:03:05 I'm friends with founders of multiple of those companies, so, uh, I relish them all.

Harry Stebbings 1:03:08 But it's like children, you always have a favorite.

Walter Goodwin 1:03:10 Yeah.

Harry Stebbings 1:03:10 Yeah.

Walter Goodwin 1:03:11 You know what I, I like about all of these guys is they still have the capacity to dodge and weave, right? They, they haven't yet locked up huge amounts of their own capacity. They're not a Neocloud. Uh, they have the potential to bet on a new chip as it comes up. They have the potential to carve out new portions of the market. I think there's a world in which this piece doesn't fully commoditize, right? You can be specialist in serving multimodal models, text models, fast inference. Um, so I won't place a specific bet. I will say that I would rather be an inference provider today, you know, a specialist in running these models, getting them up, um, than I would a Neocloud, actually. I think there is more mileage in being able to create the sort of business flywheel of, of, of signing up these enterprise customers that consume these tokens.

Harry Stebbings 1:04:07 Will they be multi-hundred billion dollar companies? And is it an Uber/Lyft market where winner and significantly less valuable second and third place, or is it a GCP, Azure, AWS market?

Walter Goodwin 1:04:20 Even that's only three players, I guess. Um, so-

Harry Stebbings 1:04:23 Sure

Walter Goodwin 1:04:23 ... yeah, I think you're not gonna have 10. Um, and today you probably do have a long tail that takes you towards that. Uh, and if you add the Neoclouds into that, you definitely have, you know, tens of players, all of whom are maneuvering around billions of dollars to-

Harry Stebbings 1:04:38 Are they multi-hundred billion dollar companies?

Walter Goodwin 1:04:39 ... win in this space. I think we'll have a couple of multi-hundred billion dollar companies built in that space.

Harry Stebbings 1:04:44 Can you talk to me about the new funding round? How much did we raise?

Walter Goodwin 1:04:48 Uh, it was a $720 million Series C.

Harry Stebbings 1:04:52 How was the process? Was it fun? 'Cause now you're like the belle of the ball.

Walter Goodwin 1:04:56 It's, um, you know, if you go back four years, actually, go back two years, you know, uh, we were still teaching people about the word inference, and it's- it makes for a, it makes for a kind of trickier investor pitch, right? Because, um-

Harry Stebbings 1:05:10 But when you educate an investor, it's just, it's kind of like leave then enormously

Walter Goodwin 1:05:14 You've got 30 minutes, right?

Harry Stebbings 1:05:15 Yeah.

Walter Goodwin 1:05:15 And, uh, you have to spend 20 minutes, uh, and four years ago you had to spend 10 minutes explaining what an LLM was, and then you had to spend another 10 explaining why you felt that training these things wasn't gonna be the be all and end all, and, and you'd deploy. I'd then spend five saying, "And here's why I think speed is gonna matter." We hadn't yet seen reasoning models and so on. Uh, so compared to then, you know, it's been, it's been very fun, I think, to, to be able to go straight in and talk about how we have a completely unique and extraordinary team that can do things differently to everybody else, how we have a very unique position in the supply chain and can maneuver differently. I think a different set of philosophies on how you build to win in this space. You know, that's what I get to spend my time over coffee talking about. Um, and so I think there is a real advantage to now being in the eye of the storm, in a way, in a space where I think there is an appreciation of why this is such a crucial problem to go and solve.

Harry Stebbings 1:06:14 $720 million. Talk to me about is that enough? And suddenly you have that in your account.

Walter Goodwin 1:06:23 Mm.

Harry Stebbings 1:06:24 What changes? Like, what do you do that you couldn't do before?

Walter Goodwin 1:06:27 So suddenly there's a, there's a weight of responsibility, right? Like, there's a number of folks who have, uh, you know, uh, staked a lot of capital on us, and I think one of our responsibilities there is to ensure that we are always have, you know, putting multiple shots on goal, as it were. So one thing that capital allows us to do is kind of distribute ourselves to look at multiple things at the same time. You know, we're now able to start looking more generations ahead. We're able to explore more parts of the stack than we maybe, maybe previously would have. You know, more innovation at the systems layer, at the networking layer. Um, uh, and sort of building this entire roadmap of, of future systems is one thing that, you know, capital allows us to do. Um, this is an expensive space, but I think it's a space where, you know, the underlying capital that is needed to, to kinda get that chip out, get those first systems back You know, it's- that's been a kinda fixed cost for the last few years, and in some sense, um, you know, that's not what most of this capital is used for. I think what a lot of this is now important for is, is that ability to maneuver faster in the rest of the world, in the supply chain, uh, you know, the placing of long-term agreements over certain components that we need to have years into the future.

Harry Stebbings 1:07:46 Mm-hmm.

Walter Goodwin 1:07:47 Um, uh, and the ability, I think, to build, you know, more of that muscle of predictive power as well. You know, we can now start to think about how we run more of our own experiments, work out what might be important in the future. And as I said, Fractal will always be a chip company, um, but we do need to have strong insights so that we can be more helpful to our customers about what scales well, where model architectures might head next, how you can handle very long contexts, what it looks like to run an agent swarm versus one very long serial agent. You know, all of these almost ML research challenges that we also get to now go and tackle. Um, and we're also expanding geographically, so obviously we've got folks in Taiwan, uh, we've got a team in the States. Um, I'm very excited that we were joined just a few weeks ago by Dan Rabinovich from Meta, uh, who worked on a lot of their AI compute systems there, driving our kind of systems build-out. So everything from the chip upwards, you know, the rack scale systems, the networking, the integration with the data center. Um, so it, it is this very long stack, and we get to go and build parts of the business, you know, where it makes sense to, um, and pull in the world's best talent as we, as we

Walter Goodwin 1:09:06 go after that as well.

Harry Stebbings 1:09:07 Can I ask you, speaking of investors, if you could invest in any one of your suppliers, who would it be and why?

Walter Goodwin 1:09:13 So TSMC is, uh, kind of an extraordinary business in that if you look at everything that is happening today, everything traces back to them, right? And it's sort of... I mean, this is not a, this is not a scoop for your podcast, right?

Harry Stebbings 1:09:29 Sure.

Walter Goodwin 1:09:29 I think it's pretty well known that TSMC underpins all of the world's chip production. Uh, it is cheap at the price. I mean, I, I w- I would buy that stock every day of the week. Uh, all of the world's chips that are at the frontier, or the majority of the world's chips at the frontier, are built there. Um, they are an extraordinary business, an extraordinary team. Um, so I think we'd, we'd go in there.

Harry Stebbings 1:09:51 I'll take that. If you could invest in any of your customers, who would it be and why?

Walter Goodwin 1:09:57 So I really like some of these scrappy neo labs. Again, I'm thinking like an investor here.

Harry Stebbings 1:10:02 Sure, yeah.

Walter Goodwin 1:10:03 You know?

Harry Stebbings 1:10:03 That's what I want.

Walter Goodwin 1:10:04 They're a little cheaper.

Harry Stebbings 1:10:05 Go on.

Walter Goodwin 1:10:06 Um, and I think we are gonna see some-

Harry Stebbings 1:10:07 Which, which one?

Walter Goodwin 1:10:09 Uh, so, uh, well, friends of mine have started Inherent in London. I don't know if you've come across them. They're looking at, uh, uh, both, I guess, the sort of the RSI challenge, but also in particular what it takes to build systems that automate science, uh, automate research. Um, and I think there is a lot of mileage here in, you know, it's not quite the same as pursuing the best frontier model that goes and tackles everything, but I think there are so many interesting questions in what it looks like to orchestrate and deploy models, build systems that are better at, at solving these challenges.

Harry Stebbings 1:10:46 If you were forced to invest in one of your competitors, final one before we do a quick fire, who would it be and why?

Walter Goodwin 1:10:52 So I'm a big believer in 3D DRAM, uh, and one other company called dMatrix is pursuing a different architectural direction in the same space. Um, uh, I think both companies are gonna be very successful off the back of that.

Harry Stebbings 1:11:03 What was their last price?

Walter Goodwin 1:11:05 I don't know.

Harry Stebbings 1:11:06 Um, okay. Uh, are you ready for a quick fire?

Walter Goodwin 1:11:08 Mm.

Harry Stebbings 1:11:09 So what have you changed your mind on in the last 12 months?

Walter Goodwin 1:11:13 Maybe I'm not quite ready for a quick fire. Do you wanna read me the quick fire, sorry, and I'll just, uh-

Harry Stebbings 1:11:18 [laughs]

Walter Goodwin 1:11:19 [laughs] Is that not, is that not an honest quick fire?

Harry Stebbings 1:11:21 Please can we keep that in? Um, a- a- no one's ever ready for the quick fire. But, um, so one is what have you changed your mind on in the last 12 months?

Walter Goodwin 1:11:28 Mm. I think the underlying pace of capability advance in models has been extraordinary. So these maths results that we saw this summer, uh, in terms of kind of, you know, th- people talk about the truncation of their timelines, right? I think what has become clear is anything that is kind of verifiable, you know, maths and coding tends to fall into this, uh, will be solved in the next 12 months. We will be able to tackle any of these kind of hard, verifiable challenges with frontier models.

Harry Stebbings 1:11:57 Who do you not have on your board that you would love to have on your board?

Walter Goodwin 1:12:00 This is an easy one. I think Demis Hassabis is both, uh, an academic hero of mine and a bit of a British hero. Uh, and I think he should, uh, he should come aboard.

Harry Stebbings 1:12:10 Have you asked him?

Walter Goodwin 1:12:12 Not yet.

Harry Stebbings 1:12:12 Well, this is a good way to do it. Um-

Walter Goodwin 1:12:14 Demis? [laughs]

Harry Stebbings 1:12:16 What's the biggest misnomer about the chip industry today?

Walter Goodwin 1:12:20 I guess that it's a, you know, misnomers about technology businesses in general are that they're very technological, and it's a very, very, very human space. So much of it is about relationships. So much of it, in our case, is about having this extraordinary team that can collaborate across the entire stack. Um, so it's that it's powered by

Walter Goodwin 1:12:38 humans.

Harry Stebbings 1:12:38 Will Nvidia be a $10 trillion business in five years?

Walter Goodwin 1:12:41 Yes.

Harry Stebbings 1:12:42 You said that pretty confidently.

Walter Goodwin 1:12:45 It's a very positive sum game. Nvidia today trades, uh, this is a soundbite, so it may be unscored, but, uh, I believe it trades at a lower forward multiple than Cisco after the crash, uh, you know, after the dot com bubble burst. Um, I'm very long on this space. We've talked about this bubble before it even started booming, right? We- we're so traumatized by the dot-com bubble, um, that we've priced in exuberance right from the outset. Um, and so, you know, I think we're not yet appreciating how sustained this build-out will be. And you don't have to have incremental data centers added necessarily. You know, we talked about this advance in compute efficiency. If I have finite numbers of megawatts, I need to replace my chips every few years with better chips. And so that's a huge advantage for Fractal as a market entrant, because every chip that you see in a data center today will definitely be out of those data centers, you know, in six or seven years tops. Many of them won't last more than a couple of years. Um, and so that's an enormous market opportunity for the market leader. They have extraordinary supply chain relationships. They're going to do very well. There is then this enormous, and I would say, new opportunity in being the chip vendor that serves the needs of the frontier, and so I'm very long on Fractal's chances over that horizon as well.

Harry Stebbings 1:14:13 You are in charge of the UK. Congratulations. Um, what do you change to make the UK a superpower in technology?

Walter Goodwin 1:14:21 I think, you know, sadly, so much of what unlocks growth is unlocking energy. It is doing everything that we can to bring online more renewables, more nuclear, more sources of power that allow us to have build-out of industries that matter here.

Harry Stebbings 1:14:40 Final one: What are you most excited for about the next 10 years?

Walter Goodwin 1:14:44 I think it's, again, we're using these models now. What are we using them for? We're using them to solve problems we couldn't solve before. And I think we're only just starting to see that pick up and take off in terms of what it means for the real world. And so it is going to be all of the ways in which life gets a bit better. Uh, we find, you know, cures for diseases we couldn't previously tackle. We have education that is high quality for, for everybody. These things are unequivocally going to come. We have the technology to make it happen now.

Harry Stebbings 1:15:17 I think it's so awesome. You look at, like, class sizes traditionally, like 40 children in a class. With AI, you have, like, personalized education for some that like multiple choice, some that like visual learning at their own speed. I was shit at maths, probably unlike you. I was always too scared to put my hand up and say I don't understand 'cause I'd look stupid. And, like, having personalized learning t- to that point is, like, one tiny example.

Walter Goodwin 1:15:38 Yeah.

Harry Stebbings 1:15:38 I so agree with you. Listen, Walter, I love doing this show 'cause I, I really get to learn from amazing people like you. So thank you for doing it with me, and this has been so much fun.

Walter Goodwin 1:15:46 Thanks very much, Harry. It's been very fun.