No Priors Ep. 78 | With AWS CEO Matt Garman

1. 0:00 – 0:23

   Introduction

   1. MGMatt Garman0:00

      I don't know how many times in the first couple of years I had to explain why a bookseller was offering compute services and storage services. There's not a lot of business opportunities that are as big as cloud computing and, and as potentially transformational. Original AWS thesis was we'd take care of the muck so you don't have to. I want my customers to want to run on us. (electronic music)

   2. SGSarah Guo0:22

      Hi

2. 0:23 – 2:53

   Matt’s early days at Amazon

   1. SGSarah Guo0:23

      listeners, and welcome back to No Priors. Today we're talking to Matt Garman, who took over as CEO of AWS in May. Matt has been with AWS since it was a $0 billion business to today's $100 billion run rate business. Welcome, Matt.

   2. EGElad Gil0:37

      Matt, thanks so much for joining us today. It's a real pleasure to have you. One thing that I think is really fascinating is you actually started on AWS in the very early days it was just getting started, and you did that as an intern while you were getting your MBA. Could you tell us a little bit about the, both the origins of AWS as well as your own involvement with it?

   3. MGMatt Garman0:52

      Sure. So it was in 2005. Uh, I did my business school internship at Amazon and, uh, as we were looking around for projects, uh, I talked actually at the time to Andy Jassy, uh, and he was, um, telling me that he was starting a new business inside of Amazon that was technology focused and he couldn't tell me about it, uh, before I started, but I thought it sounded interesting so I, I joined him and worked on that. And it was, uh, it was, it was AWS pre-launch. Uh, I got to work on that as an intern. It was a, it was a super cool opportunity. Um, I came back full time, uh, effectively as the first product manager for AWS, and I'm now on year 18, uh, or just, you know ... So, uh, been working on the, the, uh, the business the entire time and, uh, you know, it's a, it's a fascinating space. Even back then, uh, we saw the potential of what, what AWS could be. Um, obviously that was, you know, it was a startup, right, uh, in- inside of Amazon. And so it was, you know, we, we had visions of what we thought it could be, but, um, you know, there's a lot of hard work and some good luck and some things that have gone right for us, uh, and a great team over the last, uh, couple of decades to, to go build AWS. And, you know, the fascinating thing today is it's still in the very early stages of what the business can be. There's, there's not a lot of business opportunities that are as big as cloud computing and, and as potentially transformational, um, to, to every, uh, industry out there. And so, um, it is an exciting place to work, uh, just as much as it was, uh, in 2005 when I was an intern kind of writing the original business plan of, of who, uh, when we first launched our services, which companies might possibly be interested in using these things.

   4. EGElad Gil2:23

      What were the, um, other projects that you were offered at the time?

   5. MGMatt Garman2:26

      I'd worked at startups before going to business school and, um, part of what I was looking for is I wanted to see how larger companies did new projects and kind of entrepreneurship, if you will, inside. So there's a comp- a couple of few technology companies that I looked at and, and was excited about Amazon and there was a couple of other kind of retail businesses that, uh ... I mean, there was, the internships were mostly in, in retail. Um, and, uh, you know, there's some new categories that they were starting

3. 2:53 – 6:36

   Early conception of AWS

   1. MGMatt Garman2:53

      up and things like that, that could have been interesting, but, um, I always also knew I wanted to go back to technology. Um, so this was, uh, far and away ... So it convinced me to come to Amazon because it, it seemed so exciting.

   2. EGElad Gil3:02

      And how, how well fleshed out were the original plans? Because I ended up, um, using AWS for my first startup in 2007 or 2008 and it was pretty new and it was a small subset of services at the time.

   3. MGMatt Garman3:12

      Yeah.

   4. EGElad Gil3:13

      And to be honest, at the time, uh, Amazon wasn't thought of as a d- as deep of a technology company as it both was and is. Uh, and AWS was really kind of like, wow, Amazon is doing this thing that, you know, at the time you thought maybe it would be natural for Google or somebody else to have started with. And so I'm a little bit curious about, um, about the, that sort of early conception and roadmap and what was in place and what you added over time.

   5. MGMatt Garman3:35

      I don't know how many times in the first couple of years I had to explain why a bookseller was, uh, was offering compute services and storage services. I think the approach that we took worked out to be one of the key pieces to our early success. I think if you look at, at some of those others like Google and, and, and later Microsoft, um, they kind of went at this space ... F- first we were the first ones out there that had anything like this, but even soon after that, I think they went after this space like they were gonna force the developers to change how they build applications and kind of build them in a, in a new way. And we went after it as in we're going to build building blocks and let developers and, and builders go build interesting things. And so our view and, and this, this started really from internal at Amazon. If you, if you scroll back all the way to 2003 or so, Jeff Bezos basically mandated across the company that in order to move from a big monolithic stack that wasn't going to scale anymore for Amazon, we had to move everything to services. Uh, and that was really a lot of the impetus for us looking at after that and we saw the success of that. We said, "Well, maybe this would work for other people who are likely going to have the same struggle that we're going to have, uh, as, as Amazon." And so we thought starting from first principles, what are the things that we would need to go build to help people build a company? And we knew people needed compute, we knew people needed storage, we knew people needed databases. And, um, so we built those things, right? And we, we didn't force people to change how they architected, right? When, when we gave you a, a virtual Linux server, when you logged in, it was just a Linux server. Like it wasn't magic. You know, it auto scaled and you got it in 30 seconds, which was pretty awesome at the time where normally it was going to be six months or something for you to go get a server. Uh, and this is when everybody was like, when you had a startup, you had to go to Exadata and buy racks of servers in order to get your startup off the ground. Um, and so, uh, so that was obviously transformational. But once you actually got the infrastructure, it kind of operated the same way that you were used to. S3 was a little bit different, right? The put, get, delete was a little bit different, but the, the storage concept wasn't too different.

   6. EGElad Gil5:30

      Mm-hmm.

   7. MGMatt Garman5:30

      Um, and so I think that was one of the things that really helped us move quickly and helped people not have to have a total paradigm shift of how they develop. It was more how they get the infrastructure to develop on. And then over time we could add things like Lambda or we could add things like, you know, Bedrock and AI services and other things like that that are more different that people weren't used to. But, but kind of starting out with some of those things people were familiar with and they could instantly grab on and, and build quickly without having to learn a new concept I think was a, a really big, um, accelerator for us at the beginning.

   8. EGElad Gil5:59

      Yeah, it was huge. I remember, um, when it, when it first came out, I thought there was a few things that were done really well. One, to your point, is just, you know, providing really basic building blocks. Second was just the iterative nature of it.... where you launch with a small number of services and then you kept adding stuff that were kind of the obvious next steps and are, you know, and early on we, people would wonder, at least I and the startup community wondered, will they end up with everything I need? And you very quickly did. Um, but to your point, you also built it in a way where before that I think, um, most people building today have no idea, you, to your point, you literally would have to set up your own Server Rocks or-

   9. MGMatt Garman6:30

      Yeah.

   10. EGElad Gil6:31

       ... find somebody who would do it for you. It was hugely painful. You had a whole team that was doing that for every company and in some

4. 6:36 – 12:21

   Understanding the full opportunity of cloud compute

   1. EGElad Gil6:36

      cases it caused real problems. My company eventually got bought by Twitter, my first startup. And then at Twitter we ran into real issues in terms of data center capacity and planning and all these things that you can now just get on Amazon for. So it's, it's, uh, pretty radical in terms of what's been enabled for startups and the decrease in headcount per company that's needed to associate with that.

   2. SGSarah Guo6:53

      I remember as late as 2010, 2015, maybe you guys are still having this conversation with the largest customers, that, um, you know, if you're, let's say, a large financial, you had, like, this whole platform team, several different iterations of it, and the line I would get from people, uh, would be like, "We're never gonna do that public cloud thing from a security perspective. You can't compete with us on cost. Our platform is better." Like, you know, "It's not reliable." And- and just, like, there's a very strong orientation-

   3. MGMatt Garman7:25

      Mm-hmm.

   4. SGSarah Guo7:25

      ... toward skepticism of this, you know, even in 2010, like upstart company versus like, "Uh, we are ex large financial customer." I think, I think almost everybody has seen the light at this point, but, you know, you- you- you described Amazon AW- AWS in particular as still being quite, quite early in that journey, and I think one thing that people acr- even those in the tech industry who experience more exponential growth than most, like, thinking about how large these markets become is really challenging. Um, and we might be, you know, I think we're at one of the, at the beginning of one of those cycles again with AI. We'll, we'll come to that. Um, uh, like, at what point did AWS internally, did you guys, like, know that this was gonna be that large? And how'd you talk about size and opportunity early on?

   5. EGElad Gil8:15

      And just to give scale real quick on this, I think you guys went from something like 500 million or so in 2010 to about 90 billion last year in terms of revenue for AWS.

   6. MGMatt Garman8:23

      That's right.

   7. SGSarah Guo8:23

      (laughs)

   8. EGElad Gil8:23

      And so it's just, it's this insane-

   9. MGMatt Garman8:26

      Yeah.

   10. EGElad Gil8:26

       ... ramp of 89 and a half billion dollars in incremental revenue over, you know, 14 years or whatever it is. So it's, that's amazing.

   11. MGMatt Garman8:32

       Yeah. It's, it's, it's easy to get caught up in those big numbers and that the fact that it's so early. And so, so, you know, back to your, your question on when we kinda knew we were, um, we were kind of on that track, I, I, I remember it was probably 2008 or 2009, I- I can't remember exactly, uh, but I definitely remember the trip. We went on a trip to New York and actually there's a bunch of financial services com- companies, the Goldman Sachses and JP Morgans and, and they, they wanted to learn about what is this cloud computing thing. And- and they mostly were just fact-finding. They were trying to get information from us on how they could more efficiently run their internal IT systems, I'm pretty sure. Um, but we went in there, we're like, "Well, it's worth a shot." And- and, you know, they were, they were like, "Okay, look. Our w- workloads are never gonna run you. Like, maybe someday our website will run you or something like that, but, but never any of our internal workloads." And, uh, you know, and we listened to them and- and we said why? And- and- and we- we said, "Awesome. Tell us why." And they were like, "Okay, we have to have this compliance, you have to meet this rule, you have to meet this thing, we have people to do these audits," blah blah blah blah. And we spent the next decade just checking those things off the list and- and we basically never said, you know, or part of what we did is we said we want to know what are the most difficult workloads to run? What are the hardest things to do? And let's go solve those. 'cause if I can solve the, you know, JPMC running in AWS, if I can run, solve the US Intelligence Agency running in AWS, like, the- the reasons for a regular other companies are diminishingly small. And- and so that was kind of our mentality. I was excited and I love getting the startups running it to us, but the big enterprises can kind of easily dismiss. You're like, "Well, it's a startup, you know? They don't have all these things." Uh, and so what we did is we just did both. We said, "Look, we're gonna go in as much bus- business with startups as possible as we can and we're gonna check off the list all of the things that are gonna help JPMC or the US government or, um, Pfizer or whoever it is run on us in a, in a secure, safe way." And- and that's what we did. And- and, you know, today those are all huge customers of ours. After I was in AWS for about a year, I remember sitting down with a friend of mine, uh, who was a business school f- uh, classmate of mine who was also working in a different part of Amazon, and he was like, "Oh, how's that AWS thing going?" And I was like, "You know what? I think this thing could be a billion dollar business."

   12. EGElad Gil10:42

       (laughs)

   13. MGMatt Garman10:42

       And he looked at me and he's like, he's like-

   14. SGSarah Guo10:44

       (laughs)

   15. MGMatt Garman10:44

       ... "Dude, do you know how big a billion dollars is?"

   16. EGElad Gil10:46

       (laughs)

   17. MGMatt Garman10:46

       Like that seems unlikely. I was like, "No, no, seriously. I think we could get to be a billion dollar business." Um, (laughs) so, uh, you know, I'm, I, we- we knew it was gonna be successful and we didn't know, you know, quite how successful or when, I would say. Um, and so, you know, now that we're at 100 billion run rate, uh, you look at, you- you still go out there and I think 85% of workloads are still running on prem today by most estimations, somewhere in that range, you know, pick your number, whether it's 80 to 90, whatever it is. Like, that's enormous. Like, if I have, I still, there's still 10X growth of just existing workloads. Forget all the new gen AI workloads that are being created every day. These are just existing workloads to move. There's a, there's a 10X number in there. Um, and so that- that business is massive and I think there's a couple of, you know, one of the big inflection points we saw is we went after the- the intelligence agencies for the US government and we won that contract and it was secret. Um, and it, you know, we- we- we pushed really hard to go in that and it was against all the incumbents, the HPs and IBMs and Oracles and whatever, and we won the contract for, to do this cloud workload and, uh, but it was, um, confidential and we couldn't share with anybody that we were doing it and, uh, IBM sued-

   18. SGSarah Guo11:59

       Yeah, I remember this.

   19. MGMatt Garman11:59

       ... to, uh, because they- they said it's not fair or it wasn't, and- and so then that became, so then it became public that we won this deal and then the intelligence agencies went out to public and said, "No. AWS is the most technically sophisticated, they have the most capabilities, they're the mo- operationally strongest and that's who we're gonna go with." And so we had the government out there now saying that we are the best and most technically capable to run these-... high, uh,

5. 12:21 – 14:19

   Blockers to cloud migration

   1. MGMatt Garman12:21

      highly important workloads. And, and that was a huge, um, kind of stamp of approval for us, if you will. And, and I do think that that's one of those moments that in some ways we kind of got lucky that they sued otherwise it might still be secret that, uh-

   2. EGElad Gil12:34

      Yeah. (laughs)

   3. MGMatt Garman12:34

      ... that we were doing that. And I do think that that helped gain a lot of credibility in the enterprises.

   4. EGElad Gil12:38

      What do you think is, um... You mentioned that, you know, 80% of workloads still haven't migrated over, um, what, what do you think are the main blockers to that today? Is it just momentum? Are there specific features? Are there big things still to build?

   5. MGMatt Garman12:50

      There's some technologies that, you know, I think... And look, if I had a easy button, and, and by the way, we're trying to build an easy button, but, uh, but if I had a easy button that would just migrate mainframes to a modern cloud architecture today, almost everyone will push that button, but it doesn't quite exist today. And it's not as simple as like, "Great, I'll go run your mainframe in the cloud," like that's not what customers want. They wanna actually modernize those workloads and have them into, you know, micro-services and, and containerized workloads and other things like that. So, so yes, that's one, is there's just a bunch of workloads like that that are old and, and their customer's running a big SAP thing and they wanna move it to the cloud but it just takes time 'cause it's tied into a bunch of other things like that. There's also a bunch of workloads that as you get out of core IT workloads that are in line of business that are the next set of things. And whether that's, um, you know, say telco workloads, right? That are, that are running kind of the, the 5G infrastructure around the world. Um, we've slowly been moving those to the cloud and helping those customers get that flexibility and, uh, and that agility of, of running those in the cloud as well, but they're slower to move. Um, if you think about all the compute that runs, uh, factories out there today, on factory floors, most of those have not been modernized, most of those are, are thinking... Uh, and, and there's a huge opportunity, by the way, for AI to, to totally revolutionize how you think about factory workflows and, and efficiency there. But a lot of that hasn't moved, um, and, and so some of this is, you know, there's, uh, on-prem infrastructure that people are still amortizing, uh, there's people who's, there's

6. 14:19 – 18:04

   AWS reaction to Gen AI

   1. MGMatt Garman14:19

      still people whose job it is to, to run on-prem data centers, and so they're kinda resistant to moving things. So, you know, there's, there's a bunch of factors in there and so some of it is just, uh, takes time, some of it is technology pieces, um, some of that is we still have stuff to go build, and, and innovate and help make it easier for customers to do that.

   2. SGSarah Guo14:36

      I'd love to hear about just the initial, um, investigation of, like, generative AI as a technology change and, like, how AWS began to react to it and invest in it. Because to some degree it puts us all back in the, like, on-prem colo era of the world where to get one of these, you know, if you're doing any sort of real pre-training, uh, to, to get your startup off the ground, you're back to, "I guess I'll buy a bunch of DGX boxes somewhere and," it's like, "I need to think about the cost and management of that."

   3. MGMatt Garman15:07

      Well, I, and I, well actually most people are still buying those but in the cloud. But it is kind of a, it's not a serverless type of a thing, it's, you know, mo- most people are still not buying, uh, you know, H100s and hosting them in a colo or anything like that. Um, and increasingly I think that's gonna get harder and harder as you move to liquid cooling and, and, and larger clusters. But, um, you know what? It is a, it's a super interesting space. I think we, we've been working on this space for how many years now? Um, and, and look, we, we've been investing in AI broadly for the last 10 years, and that's why we started five or six years ago investing at the infrastructure layer and building our own processors because we, we knew this was coming, we saw this path coming and we knew that that's also not a short term investment so it's one of those things you gotta invest way ahead. And then we were investing in, in building generative AI models, um, and then, you know, OpenAI kind of made a, a generational leap forward with what they were able to do and what's possible. And then many people have talked about this but it, it really in some ways was a discovery as much as anything about just what was possible and, and kind of unleashed a new set of, of capabilities. And so we actually as a business took a half a step back and said, "Okay, these are gonna be transformational abilities and, and assuming that this technology gets better and better and better over time, how do we make it so that every company out there can go build using those technologies?" And so different than how can I go build a consumer application that people are gonna be interested in, we kind of took it from the point of view of, of AWS, right? Like, just what, what are the building blocks that I can help all of our customers, whether they're startups, whether they're enterprises, et cetera, go build interesting generative AI applications? And so we started from first principles. Customers are gonna care a ton about security. They, they're, that's not gonna change. They're not gonna all of a sudden not care about securing their infrastructure. We also had this hypothesis, two more hypotheses. One that the idea that there wasn't just gonna be one model. We thought that there was gonna be a lot of models for a lot of different purposes and there'd be big models and small models and people would want to combine them in new and interesting ways. Uh, and I think the last two years have probably pa- played that out but I think when OpenAI first launched that wasn't as obvious, but that was kind of one of the bets that we made. Um, and then the third one is that we view that every enterprise that was building on us, the interesting IP that they were gonna bring to the table was mostly gonna be their data, and they were gonna care that their data didn't leak back into a model or, or escape from their environment. And so we built a bunch of what we did starting from those principles of, how do we make sure that these things are secure, that their data is secure, that they can have access to every piece of technology that, that customers need to go build interesting applications, and that they can do it in a cost-effective way? Uh, and so that's how we approached the space and I think we now have a platform in Bedrock, in Trainium chips and Inferentia chips, in, um, and then a bunch of the other capabilities around as well as the suite of models that we offer, both, um, proprietary as well as open source ones, uh, or Openweights ones. Um,

7. 18:04 – 20:18

   First-party models at hyperscalers

   1. MGMatt Garman18:04

      that, that I think we're, we're starting to see that really that momentum pick up and we're seeing more and more customers really like that story, they like that platform to build from, and we're seeing, uh, enterprises really lean in and want to build in that, in that space because it, it gives them a lot of that control that they want as they go and build applications.

   2. SGSarah Guo18:21

      How much do you think it matters that, um...... AWS has, let's say, like first-party models it offers its customers 'cause that's clearly a strategy for, um, some of the other hyperscalers.

   3. MGMatt Garman18:35

      Google, uh, that's obviously their strategy. Um, uh, they're, they're really the only one, uh, that has a fir- a first-party model today of the other hyperscalers. Uh, Microsoft's done a good job of co-opting, uh, uh, OpenAI's innovation. Although in their last, uh, in their last... I saw recently, they listed OpenAI as one of their biggest competitors now so it'll be interesting to see how that all plays out. But, uh, you know, for us, I think it's important and, uh, which we do. So we are building our own first-party models. We have our first-party models today. In fact, the, the titan embeddings model is by far the most popular embeddings model that we have inside of Bedrock today for people that are building search indices and, and thinking about things like that. Um, and we are building larger and larger models as well, first party. I think it'll be important, but not critical. I mean, I think people love using, uh, Anthropic's Claude models. Those are fantastic and right now, those are the best p- performing models in the world, um, which is fantastic. Uh, we just launched, uh, LLaMA 3.1. Uh, on the day it launched and we have a really tight partnership with Meta and their open weights model is fantastic. And, uh, and I, and I think increasingly we're seeing customers really love that open weights model because they can go and, particularly enterprises, customize it. They can do fine-tuning to it, they can add their own data to it, and really customize and distill and do some interesting things. And so I think that is, is super critical. You know, we're seeing kind of specialized models, if you will, where we see folks like Adobe building Firefly as, uh, all built on top of AWS purpose-built for their own th- their own thing that they're building. Um, you know, whether the, the Amazon purpose-built models are... I, I think they're an important part of that. Mostly it's, partially it's for us for learning, some of it's for powering our own applications, and some of that may be for end customers, but, um, it's all kind of that diversity of option, honestly. Like, we, we want there to be the best

8. 20:18 – 22:46

   AWS point of view on open source

   1. MGMatt Garman20:18

      set of options and we want them all to run at AWS, and so we want those workloads to run there. And so to the extent that we can do something novel or interesting with our own first-party models, we'll do that. And, um, but we're also delighted for our partners to run as well. So I think it's a little bit of both.

   2. SGSarah Guo20:35

      Yeah, it's really interesting. Um, uh, Alyssa Henry, you know, longtime, um, AWS leader as well, is a friend. And, uh, over the years, I would like ask her-

   3. MGMatt Garman20:44

      Yeah.

   4. SGSarah Guo20:44

      ... about some interesting new open source project. And she... You know, it was the most terrifying thing, honestly, because she would always be like, "AWS loves open source. We make more money on open source than any open source company does," which is, uh, you know, i- if you think about all the advantages that AWS has, even if you're very friendly in the ecosystem, it, you know, can, can turn out that way. And so I think the sea change that's happened in terms of availability of open weights models and multiple players here, Mistral, LLaMA, et cetera, that are very competitive, uh, is like, I think, you know, a, a, a huge, um, huge boon to AWS's, uh, general open ecosystem model.

   5. MGMatt Garman21:24

      We've always leaned into open source. We're huge contributors to many open source projects and, and we lead many open source projects and, uh, and I think we do a good job, um, uh, creating and turning those into businesses for our customers and helping run managed open source projects. And so it's a big area for us. One of the reasons, frankly, is that, you know, we've long said we don't want customers tied to AWS because they're locked into some proprietary licensing, uh, piece. We want them to be able to... We want... I want my customers to want to run on us as opposed to kind of locked into a Microsoft license where you're, you're held to some different license that you can't get off of easily or, or old school kind of like Oracle database that you can't get off of. Um, you know, we want people to be able to run. And so even where we have something like Aurora, which is our kind of managed database, it is 100% PostgreSQL compatible and if you take that code and go run it in a pro- PostgreSQL database somewhere else, it won't operate as well, you know, 'cause we do a great job at that, but it runs. Like and, and, and in theory, if you run it as well, it will, uh, operate as well. Um, and so, you know, that's, that's how a lot of our services are built and how we think about things. Um, we'll, we'll support pri- proprietary things as well and, and, you know, at some level there are some services where you gotta take advantage of the cloud where, um, you know, there are some proprietary things that customers can use, but, um, uh, things like Dynamo and other, other technologies like that. But we

9. 22:46 – 26:07

   Grounding and knowledge bases

   1. MGMatt Garman22:46

      really embrace open source and, uh, and I think it's been, uh, it's beneficial to the whole industry frankly, and it's, it's, uh, it's a way to get better security, better visibility, and kind of that license portability I think is, uh, um, is a key aspect as well.

   2. EGElad Gil22:59

      You mentioned, um, that, you know, the model side of the AI world and there's other main AI building blocks. There's RAG and, uh, you know, there's certain aspects of fine-tuning and other things that people are increasingly doing over time. There's other parts of it like eval suites and... What, what are the main building blocks that, uh, you can talk about in terms of things that are either coming to AWS or how you think about that-

   3. MGMatt Garman23:19

      Yeah.

   4. EGElad Gil23:20

      ... more fragmented world of all these different components and how they fit together relative to AI workloads today?

   5. MGMatt Garman23:24

      That is kind of the idea of Bedrock, is that we want to make it easy to do. And I, I do think actually in many ways today, the models is the front and center thing that everybody pays attention to, but I think increasingly it'll become a smaller percentage of the thing that people pay attention to 'cause people are gonna care about whether it's RAG or some other sort of knowledge base and we call it knowledge bases because it's, you know, the technology may change over time under the covers, but that, like, how do you have a grounding set of truth that you use? I also think grounding data is an interesting thing for, um, for like real-time information that you want as part of your AI systems. Um, we have things like guardrails, which I, is our, our customers find incredibly important 'cause, you know, if you're building a chatbot on the financial services website, you can actually get fined a lot of money if that thing starts giving out financial advice and you d- and so you really want to be able to control, let alone, you know, going down and talking about politics or something else that you definitely don't want it to talk about. Um, and so those guardrails are super important as people think about what they want their AI systems to do and interact with and where they want them to stay away. This is not controversial. I'm sure you, you both hear a lot about this. But again, the next generation of, uh, and the next step forward in what we can get out of AI systems is gonna depend a lot on...... how well we can integrate, uh, agentic workflows and actually get these AI systems to do things, not just kind of summarize and tell us information. And so, um, uh, building in the ability to have agents as part of that workflow, uh, is a big area of investment for us. We want that to be easy for you to build as part of that bedrock capability. Um, I do think that pre-training and fine-tuning, uh, is gonna be something that more and more customers are gonna wanna do, uh, as well as distilling over time, 'cause I think... I was just talking to a couple of customers, uh, earlier today that are very focused on, "How do I get this model down to a much smaller thing so I can put it on an industrial edge or somewhere like that?" And so, how do you think about distilling down so I ƒ- get the value of what I want? I don't need the whole kind of, um, reasoning engine behind that. And I think there's a, a long roadmap of, uh, like you said, kind of model evaluation and other things like that, and some of that is us and also some of that is partners, by the way. Um, and so we're... You know, AWS has been a place where I think part of the, the thing that has made us successful is really embracing the ecosystem to go build around there. And so, thinking about labeling data as an example, we have a deep partnership with Scale AI to come in and help you label your data if you're gonna be doing any, any, uh, uh, fine-tuning or pre-training or things like that with your data. Um, we partner with folks like LangChain to help put together some of those agent workflows and other things like that. And not to mention, of course, the model providers who are, are super important partners of ours as well. So, I, I think it's all of those things, and our job is to, how can we make it easier and easier for you to go build those applications in a, uh, a tightly coupled way so that it's easy to go use those different components,

10. 26:07 – 31:15

    Semiconductors and data center capacity for AI workloads

    1. MGMatt Garman26:07

       easier to innovate rapidly, and, um, and easier to, to build the proprietary data that you have as part of your AWS data lake so that you can kind of pull that in? 'Cause frankly, most of these, uh, generative AI systems aren't gonna be super useful if you don't have interesting data to go pull from.

    2. EGElad Gil26:23

       The other place that a lot of people are spending time right now in terms of bottlenecks to utilization or usage or future-proofing is actually more on the chip side or semiconductor or system side and then, um, in terms of DC capacity, and obviously you all have been building Tanium chips and other things, which I think is really exciting to see that evolution. How do you think about future, uh, GPU shortages? Does that go away? When? I'm sort of curious about how you think about forward-looking capacity, and is the industry actually ready in terms of building out data centers, building out semiconductors, all the rest of it, packaging, you know, (laughs) the whole, uh...

    3. MGMatt Garman26:56

       (laughs) Um, look, I, I think we're probably gonna be in a constrained world for the next little bit of time. Just, you know, that some of these things are... They take time. Like, look, uh, look how long it takes to build a, a semiconductor fab. Like, it is... It's not a short lead time, and that's several years, and, and TSMC is running fast to try to ramp up capacity, but it's not just them. It's the, the memory providers and, and the... and, and frankly data centers that we're building, right? And so as we think about, um... There's a lot of pieces in that, in that value chain that I think, as you look at the demand for AI, which has been, um, I don't know, exponential, might be undershooting it. Some of those components that support that, I think, are, are catching up, and I think AWS is, is well-positioned to, uh, to try to do that better than others are. You know, we've, we've spent a long time thinking about, uh, the last 18 years learning, how do we think about smart investing? How do we think about capital allocation? We've h- We've spent a bunch of time thinking about, how do we acquire our own power? How do we ensure that it's green and carbon-neutral power? Um, all super important things, and we're the, the largest purchaser of, of renewable energy, um, over the last, uh... new, new contracts, right? So actually going out and adding and, and supporting new renewable energy projects. We're the largest provider, I think each of the last four or five years. Um, and so, so we've been leaning into that for a while to, to ramp up this, and, and this is just a step up. And so I think we're thinking about, you know, how are we acquiring enough power? Our own chips is a way to support, um, the growth of N- NVIDIA chips. And so I think the more diversity there, the, the better off we are. We're, um... We're a huge partner of NVIDIA's. We, uh... You know, NVIDIA actually runs their AI training clusters in AWS because we actually have the, the most stable infrastructure of anyone else, and so they, they actually get the best performance from us, and, uh, and we love that partnership, and, and we have a great and growing relationship with them. And, you know, we think things like Trainium are a, a good diversification, and I think there'll be some workloads that, that run better on Trainium and, and are cheaper on Trainium over time, and, uh... As well as Inferentia. I think inference is, is one of those, uh, workloads that... Today it's, you know, 50/50 maybe of training and inference, but, uh, in order for the math to work out, inference workloads have to dominate. Otherwise, all this investment in, in these big models isn't really gonna pay off (laughs) . So hopefully, uh, for the industry that, that, that all happens. Um, but I think we're probably gonna be tight for the next little bit of time. And so, um, you know... 'Cause the, the demand is, is almost infinite. I mean, it seems infinite right now.

    4. SGSarah Guo29:31

       How does AWS think about making investments in, um, data centers of this scale to train the next set of foundation models, right? Uh, and because I think you could take a... You know, AWS is very educated player. You could take a proactive approach. You could take a customer-driven approach. But the idea that there are individual players who want tens of thousands of nodes at a time and interconnected GPUs is like a, sort of a new demand, um, vector.

    5. MGMatt Garman29:57

       Some of the demand for some of these really large models is, uh, is very large, right? I mean, it's the... They're, they're talking about needing gigawatts of capacity, um, which is, uh, it's (laughs) kind of a, a mind-boggling number that, that some of these models need. We're doing both proactive as well as customer driven, right? We try to balance 'cause, 'cause there's real capital outlays that are required as part of this, of course, and, and we're talking tens if not hundreds of billions of dollars of, of capital investment. And so, you know, we think about it as how do you, how do you make the right investments in things like land and power and other things that are, are fungible and, and could potentially be used for other things if, if eventually demand changes or the, the s- the slope changes, um, as well as then having...... um, visibility into the supply chain for more near-term things that you'll need, like servers and chips and, and, um, memory and other pieces like that. And so we balance a bunch of those things, managing the, the financial implications of, of what we need to go buy, uh, as well as the, the long-term customer demand. And we try to map out, how do we meet some of those match? And, uh, and some of our customers give us long-term commitments to help with some of those things, and we give better rates for customers that give very large long-term commitments for, for some of that capacity that requires a lot of, um, investment. But, um, but, you know, it's, uh, there's a lot of error bars

11. 31:15 – 33:18

    Infrastructure investment for AI startups

    1. MGMatt Garman31:15

       on that too 'cause at, at anything that's growing at multiple hundreds of percent year over year, um, you're, you're not gonna nail that number appropriately. And so we try to have enough buffer in there that we can support upsides when they happen and, and manage if it's a little bit less than we thought.

    2. SGSarah Guo31:30

       As someone who's seen, um, just many generations of startups decide, like, what investment they want to make in infrastructure, that is suddenly a much more important question to a generation of AI companies than, um, than it has been in recent history. What, like, advice would you have for them as the, the man holding the data center, I suppose?

    3. MGMatt Garman31:50

       We've had to go on this before, right? We started from $100 million of revenue to $100 billion, or, well, actually we started, I started when we were at zero dollars of revenue to $100 billion of revenue. And so, you know, we've, we've had this kind of rapid growth before where we think about, how do you balance some of those pieces? And, and I think part of that is, is how do you make sure that... For me, I think as you're, as a startup thinking about this is, how are you thinking about investments with a real plan of how do you have monetization and not, um, assuming that there's always more VC funding to come and, and bail you out. And, and so kind of having a plan there to have flexibility of, you know, what, how, how can I start monetizing sooner if I need to? And, and where can I keep investing if that's the part that I'm in that, that, that, that traction makes sense? Um, 'cause I think, look, the only, the only reason that any startup goes out of business is 'cause they ran out of money. That's the only, that, it's as simple as that. (laughs) As long as you don't run out of money, you're not gonna go out of business. Obviously easier said than done, but, uh, but I honestly think some startups kind of forget that. They're always like, "That's no problem. I'll just go raise more." And, uh, and kind of remembering that just because there's like a hype cycle doesn't mean that someone's gonna give you a bil- another billion dollars six months later. And I actually earned, learned that early in my, uh, career. My, my very first startup, uh, we raised, I think at the time was a lot of money, it was $27 million. We ran out of money in, like, 18 months, and then, you know,

12. 33:18 – 36:22

    Value creation in the AI ecosystem

    1. MGMatt Garman33:18

       the 2000s came around, and there wasn't any more funding and went out of business, (laughs) we, we assumed that we could just go raise more money. And, uh, and so I think that was a good early lesson that you can't always do that.

    2. SGSarah Guo33:28

       If I look just at, um, you know, my own portfolio or our friends' companies, what, what is interesting is if y- y- like, of course, you have a few examples that everybody is looking at, like OpenAI, um, where, um, value creation and dominance or at least, like, lead in a market is highly correlated with the amount of money they're spending.

    3. MGMatt Garman33:50

       Mm-hmm.

    4. SGSarah Guo33:50

       It's not true across the portfolio. And I'm, like, only investing in really AI companies, uh, in that, uh, yes, e- everybody absolutely needs computers, and I'm talking about companies that are doing their own, um, training or fine-tuning and, and... But some of our companies that are making the most progress and, like, you know, we're still talking very early days, you know, zero to tens of millions in their first year or two.

    5. MGMatt Garman34:14

       Right.

    6. SGSarah Guo34:15

       Um, i- I think one of the other open questions that people have wondered about is, um, does all of the value creation in the ecosystem go to, um, your compute vendor and eventually a big piece of it over to Jensen at NVIDIA or to the model vendor? And I, I think the, the answer, at least to, like, right now is clearly not, right? I think there's different, there's c- capture at different levels.

    7. MGMatt Garman34:38

       There's probably enough for everybody. Uh, today most of it does go to NVIDIA, I think. That's (laughs) a lot of it. But I just think that's 'cause it's where it is early in the cycle. Uh, you know, I think, um, uh, and they've built some incredible technology that's enabling some really cool stuff, so I think that that's, it's, it's, um, it's fine. And at, at some point it's gonna be the, the companies that find out how you actually go solve real problems and deliver real value to enterprises and to customers and other things like that, and that's gonna be that, you know, I, I see a lot of... If, if I, if I take a step back and see who's implementing AI out there, it's a lot of enterprises that are doing proof of concepts and, and a lot of... And sometimes they'll find one that really works well and it'll go to production, and I think if you can have a startup that can make that part easier that says, "Look, this is a real value," right? It's not a chatbot on your website, but it's something that helps you go faster, make sales better, innovate more rapidly, um, you know, do something you were never able to do before, uh, improve manufacturing efficiency, whatever it is the startup is focused on or the company is focused on for that matter, it's, it's that, it's gonna be an application level, right? It's most, most people don't, um, build a CRM from scratch. They go use a Salesforce or something like that, and, and most companies don't build software from scratch. And so I think most companies are not gonna build their own models over time. They might tweak them a little bit, but I think a lot of companies are gonna build, are gonna use the applications that use the software and the models underneath, and, um, so it's not surprising to me that you don't necessarily have to spend the billions of dollars to go build your own model. You can build a small model.

    8. EGElad Gil36:17

       It seems like there's a lot of precedent for this just in terms of prior waves of both software and internet where

13. 36:22 – 38:48

    Enterprise adoption

    1. EGElad Gil36:22

       I think Co2 or somebody had some very good slides where they broke down the relative value accrued by each layer, and to your point, each layer sort of ends up benefiting over time. So-

    2. MGMatt Garman36:29

       Yeah.

    3. EGElad Gil36:30

       ... it, it seems like that question is almost overstated in terms of-

    4. MGMatt Garman36:33

       ... the importance of it, yeah.

    5. SGSarah Guo36:34

       Do you have a prediction, if you just look at what happened with public cloud, if, um, what happens with AI platforms, uh, evolves differently? And, and the, the reason being, you know, I just spent a little bit of time talking to large enterprise customers again and exactly as you said, there's a lot of activity coming from, um, different POCs, there's investment in the area, there's top-down interest and I, I do think there's a real conviction that the value will be real. There's also, like, I feel a little bit of deja vu in-

    6. MGMatt Garman37:05

       Yep.

    7. SGSarah Guo37:05

       ... a bunch of large organizations saying, like, "Nobody meets our needs and we're gonna have to build our own platform and it's gonna be everything from, like, data management to, you know, um, GPU in, management for training an inference to, like, eval suite to, you know, compliance and audit." And so I kinda wanna say I've seen this story again, uh, before, but, you know, h- how would you predict it plays out?

    8. MGMatt Garman37:28

       It's exactly that. I think every time there's a new space and some of those things don't exist, people are like, "Well, I've gotta go build my own." Well, it's like, uh, or, you know, likely there's a bunch of other people that are actually building that too, like us, that once it exists, and I, I don't know how many times customers have told me like, "Oh, if I'd, you know, if you had already had this, I wouldn't have gone and built my own and now I can stop investing in it 'cause it's actually not the thing that gives me value." And so managing GPUs, I, I bet very few of those enterprises, that's the actual thing that their stockholders are like, "Yeah, that's what this enterprise is super important and good at, is managing GPUs." Like, outside of, like, you know, hyperscalers or somebody like that. They don't really want to, right? If they could use something like SageMaker and it has all the capabilities that they can go and build the, the services that they want, and it, it doesn't today. We know that, but we're iterating incredibly rapidly and launching new stuff every month, uh, every week really. And so, um, you know, I feel pretty convicted that that for, particularly for that level of the stack, like, it just doesn't make sense for people to do that part and it's not surprising that they do it today because some of those things don't exist and they wanna go deliver on something, and, you know, their priority was slightly different than when we go deliver it or something like that. But having it integrated as part of your platform, having it integrated as part of where you have your data lake, having all of those things tied together kind of as a, as an infrastructure piece likely makes sense. And, and it doesn't mean

14. 38:48 – 41:25

    Near-future predictions for AWS usage

    1. MGMatt Garman38:48

       that we'll do all of those things, by the way. Some of them will be partners that are built on top of us, um, which is great too. But, um, but I do think that it's the, the people that are specializing in those places are probably the ones that are gonna eventually kind of support that space, not individual enterprises. If we were to abstract out a, a level and, uh, you know, ask what you, what your vision is or how you're thinking about the next three to five years of AWS more generally as a business, wha- what are the key things or, um, areas of focus for you? Well, this is one. I think, I mean, I'm, I'm just as excited about generative AI and, and, and AI broadly as, as you all are. Um, I do think that it's an enormous opportunity for us and for our customers to... And, and I think it actually, in many ways, it has a positive flywheel effect and is, can be a tailwind to some of that first stuff that we were talking about a little while ago about helping customers move to the cloud. You know, I think if we think about where can generative AI help, some of that can be like, how do you make that go faster, right? How do you take some of that more, you know, our, our original AWS thesis was we'd take care of the muck so you don't have to. There's still a lot of that that customers have to do today that I think generative AI can help with. And so over the next three to five years, there is a big investment for us in both building that tool set, building that whole platform that we're talking about so that customers don't feel like they have to go manage a bunch of these pieces. They don't, and they don't have to think about, you know, GPUs or they don't have to think about how do they think about kind of tying these clusters together or whatever. All of that can be abstracted away. If you think about the start of what Bedrock is, if you go use Bedrock models today, you never interact with a GPU, right? You just, you send it tokens, you get tokens back. And you can, eventually you're gonna be doing things like fine-tuning and, and pre-training where you're sending in information and training the models under the covers, but you're still just then sending it tokens and getting tokens back. And so the more we can abstract that to whether it's serverless or, or an application platform that people can go build, and that's just, you know, again, we're at the early stages of what that is. And so I think that as you move forward, generative AI honestly becomes one of the compute building blocks that you think about. You're gonna need storage, you need compute, you need databases, you need inference, if you will, for your application largely. And I think that's just gonna be, and, and networking and a bunch of the other things, and inference is just gonna be one of those building blocks that people come to expect. And just like with compute where you may want a Intel processor or a Graviton processor or you may want block storage or you may want object storage, you, you may want different models behind the inference and you may think of that and, and it'll have slightly, you know, different databases. Um, inference is gonna have different flavors to it and it'll be big models and small models

15. 41:25 – 42:58

    AWS’s role for startups

    1. MGMatt Garman41:25

       and you'll trade off cost and latency and capabilities and things like that. But I do think it's part of the applications. And so we're trying to build it as part of that platform, when you're just building your application it, it comes with it. And then there's a lot to get there between now and then, but I, I imagine that's how most applications are gonna be built going forward.

    2. SGSarah Guo41:40

       So we started this discussion talking about, like, the early days of AWS and sort of how you were discovering the, the sort of, uh, you know, unthinkably large requirements from, um, large financials and yet, you know, $100 billion of revenue later, you are still, uh, a huge partner to startups. Like, th- that goes against some of the conventional wisdom of, like, choose one audience and just, you know, slowly move upmarket as something that, you know, many startups themselves choose to do. Like, why continue to work so much with startups? And I, I know you personally still think about this a lot.

    3. MGMatt Garman42:12

       It's super, uh, important for us and, and startups are the lifeblood of what, what helps us grow and we get so much benefit from the learning from startups. And, and so they will continue to be incredibly important for us and, and we're gonna, if anything, lean more into startups and supporting startups, um, as part of what we do. Thanks so much for providing your perspective over the last, oh, you know, 20 years or so of AWS. It's been really fascinating talking to you, so thanks so much for the time today.

    4. SGSarah Guo42:37

       Thank both of you for having me. Appreciate it. It's been fun.

    5. MGMatt Garman42:40

       (instrumental music)

    6. SGSarah Guo42:41

       Find us on Twitter @nopriorspod. Subscribe to our YouTube channel if you wanna see our faces. Follow the show on Apple Podcasts, Spotify, or wherever you listen, that way you get a new episode every week. And sign up for emails or find transcripts for every episode at no-priors.com.

Episode duration: 42:58