Building the Cloud for AI Agents | AWS CEO Matt Garman

1. 0:00 – 0:55

   Intro

   1. MGMatt Garman0:00

      The agentic workflows tend to perform better on AWS than anywhere else. Compute sandboxes, gateways, agent permissions versus people, a lot of those are things that we have built and are building and thinking actively about.

   2. RRRaghu Raghuram0:11

      The top frontier labs gobble up every available GPUs. At the same time-

   3. MGMatt Garman0:15

      Yeah

   4. RRRaghu Raghuram0:15

      ... you wanna promote newer companies that are gonna become the enterprises of tomorrow. How are you thinking about balancing?

   5. MGMatt Garman0:20

      From the very beginning of when we launched AWS, startups have been the lifeblood of the core of what we do. These are the innovators that are at the edge of technology, understanding what's possible. We're very intentional about keeping capacity available for the startups. We recently announced we're gonna be buying two million NVIDIA GPUs over the next couple of years. We-

   6. RRRaghu Raghuram0:37

      Your CapEx this year is what, 200 or-

   7. MGMatt Garman0:39

      $220 billion dollars, uh, for '26, and we don't anticipate slowing down any time soon because the demand is just massive.

   8. RRRaghu Raghuram0:46

      With all the debate around AI extension risks and the Hugging Face attack, what are CEOs asking you about all these things?

2. 0:55 – 2:51

   $169B, growing 37%

   1. RRRaghu Raghuram0:55

      Welcome to the pod, Matt. What a time we are living in. So I have a lot of topics to talk to you about.

   2. MGMatt Garman1:01

      Awesome.

   3. RRRaghu Raghuram1:01

      So-

   4. MGMatt Garman1:01

      Thanks for having me. I'm excited.

   5. RRRaghu Raghuram1:03

      Yeah, yeah, absolutely. So let's start actually right from the beginning, right? So you were the first GM for EC2-

   6. MGMatt Garman1:13

      Mm-hmm

   7. RRRaghu Raghuram1:13

      ... and that was 2006, right? And today you guys are, what, one sixty, one seventy billion in revenue?

   8. MGMatt Garman1:20

      Yeah, about a hundred and sixty-nine, hundred seventy billion, yep.

   9. RRRaghu Raghuram1:23

      Yeah, growing thirty-five-

   10. MGMatt Garman1:25

       Thirty-seven, 37%, yeah

   11. RRRaghu Raghuram1:26

       ... thirty-- That's 37% at 169.

   12. MGMatt Garman1:29

       Yeah, it's, uh-

   13. RRRaghu Raghuram1:30

       That's crazy

   14. MGMatt Garman1:30

       ... there's a ton of op-- And it, it's, yeah, it's, it's, uh, it's interesting to think about it from, you know, we're there in day, day, day one when-

   15. RRRaghu Raghuram1:36

       Yeah

   16. MGMatt Garman1:36

       ... we had like, you know, the first-

   17. RRRaghu Raghuram1:37

       Yeah

   18. MGMatt Garman1:37

       ... dollar of revenue. But-

   19. RRRaghu Raghuram1:38

       Yeah

   20. MGMatt Garman1:39

       ... um, yeah. And it's, you know, the interesting thing is we're still at the early stages of what the business can be and what the opportunity is for customers. Um, most workloads still, you know, there's a huge amount of workloads still that live on-prem today, and the amount of compute, um, that people are doing every single day is, is more than it was the day before. And so you see the tailwind from AI, you see the tailwind from migration into the cloud, and, um, you know, the, the, the business has grown really fast, and it's, it's been a super fun part to be a, um, thing to be a part of.

   21. RRRaghu Raghuram2:09

       Yeah, it is. It is. I mean, they'll be writing, uh, history books about this and business books about this for a long time to come. I wanna touch on the on-prem. That's, that sort of boggles my mind. I obviously did my best to keep them there for a long time.

   22. MGMatt Garman2:23

       You did. You built a lot of stuff-

   23. RRRaghu Raghuram2:24

       [laughs]

   24. MGMatt Garman2:24

       ... on-prem back in the day of-

   25. RRRaghu Raghuram2:25

       Yeah, yeah. I know, but, uh-

   26. MGMatt Garman2:26

       That's true. We're trying to now get all of that to move into AWS.

   27. RRRaghu Raghuram2:28

       Yeah. So we can talk about that later. But, uh, so if you think about EC2 in the early days, and you got your start, uh, with obviously selling to startups and, and today the AWS cloud service, I mean, as you reflect on the evolution, what's, what stands out?

   28. MGMatt Garman2:45

       Yeah.

   29. RRRaghu Raghuram2:46

       Part one, and then part two we'll talk about how this nature of how you serve startups has changed.

3. 2:51 – 12:26

   Why startups are the lifeblood

   1. MGMatt Garman2:51

      Sure. Well, l- like you said, actually, it's, uh, it's funny. So I actually interned for, um, AWS in 2005 when we were first-- kind of was an internal project. It was my business school internship, and my project was actually to come up with, uh, an analysis of who we thought AWS would be most interesting to, and, uh, the, the answer was startups, like probably-

   2. RRRaghu Raghuram3:11

      Yeah

   3. MGMatt Garman3:11

      ... not surprisingly. And, uh, and so, you know, from the very beginning of when we launched AWS, startups have been the lifeblood of, of the core of what we do for, for a number of reasons. One is the value proposition is just so attractive-

   4. RRRaghu Raghuram3:23

      Yep

   5. MGMatt Garman3:23

      ... what AWS provides to startups. Um, and, and we, we spend a lot of time and effort making sure that we're great partners to the startups, helping them not just provide infrastructure, but also advice on how to get your company up and running and how to think about your architecture so it'll scale eventually and, and a whole bunch of things that we do for the startups. But we also think that for us it's just good business, 'cause-

   6. RRRaghu Raghuram3:42

      Yeah

   7. MGMatt Garman3:42

      ... the startups today are the enterprises of tomorrow. And, and, you know, it's, it's an imprecise number, but today we estimate that, you know, maybe thirty, forty percent of, of AWS revenue comes from companies that was a one time, you know, startups in, in AWS' lifetime.

   8. RRRaghu Raghuram3:58

      Yeah. Yeah, yeah.

   9. MGMatt Garman3:58

      So and it's, uh, you know, and it-- So it's fun to see, for us to see the companies grow over time and, and to get bigger and become enterprises effectively. Um, and so that's why we invest so much in startups and why we pay so much attention to the, the brand new, you know, two people in a garage type startups. And, um, and, and not f- um, and, and not just because of the business outcomes, but also because that's who we learn from. These are the, these are the innovators that are at the edge of technology, understanding what's possible, uh, pushing our services to say what they'd want more of, what could help them go faster, what could help them achieve their, uh, outcomes. And a lot of times they're pushing more than the, the banks or the healthcare companies or the governments. The startups are the ones that are pushing that envelope, and it really helps us to be better and make sure that we're ahead of that wave where the banks are gonna want some tech, some capability five years down the road that startups want today.

   10. RRRaghu Raghuram4:48

       Yeah, yeah. And, uh, compared from then to now-

   11. MGMatt Garman4:54

       Mm-hmm

   12. RRRaghu Raghuram4:54

       ... what have the, how have the startups changed in what they want from you? Obviously-

   13. MGMatt Garman4:59

       Yeah

   14. RRRaghu Raghuram4:59

       ... they all want a lot of GPUs, and we can talk about that. But-

   15. MGMatt Garman5:02

       Yeah

   16. RRRaghu Raghuram5:02

       ... besides that.

   17. MGMatt Garman5:04

       Well, I will say, uh, there's a couple of things that have changed. One is, um, you know, startups started at a much smaller size than, than [chuckles] when, when we first started, right? They, they might have gotten ten million dollars of funding, and they had a app idea that they were kind of slowly iterating on. Now it's like, you know, from day one, they're valued at a billion dollars. They have two hundred million dollars of funding. They-

   18. RRRaghu Raghuram5:25

       Yeah. Yeah.

   19. MGMatt Garman5:25

       [laughs] It's like it's a, you know, it's a, a, a, a team and an idea, and all of a sudden they're worth a billion dollars, which, um... So it's, you know, I think the brand-

   20. RRRaghu Raghuram5:33

       They just go from our offices to your offices.

   21. MGMatt Garman5:35

       [laughs] That's right, that's right. Um, uh, and you know, like the, the size and scale and ambition of the ideas, I think requires more capital. They're, they're bigger to start with. Um, they're obviously more expensive too, right? To, you know, to, um, uh, to go after kind of a training a model or doing something that, that a lot of the folks are doing today. Um, so that's number one, which is just the size that they're, that, that they start from is, is really big. Um, but I would say number two is, uh- There's some things haven't changed, right? They, they actually are still thinking about, okay, once I scale, how do I think about an architecture? Um, how do I think about, um, security? How do I think about performance? How do I think about having all the capabilities that I need? How do I think about setting up my IM setup so that when I have more than three employees, this thing-

   22. RRRaghu Raghuram6:21

       Yeah

   23. MGMatt Garman6:21

       ... is gonna work and scale? Um, which is a lot of times why they like AWS as opposed to just, you know, going to a, a Neocloud or something like that. They actually need the, all of those other security and, and, um, and capabilities that, that come around from, um, from training. Um, and so I think that's, that's, you know, that's something that hasn't changed and, and is exactly the same. I think the, the scale at which the ramp-up, um, is definitely different today. I also think increasingly we're seeing, um, teams that want a cloud that is great to work with agents and not just with people.

   24. RRRaghu Raghuram6:53

       Mm-hmm.

   25. MGMatt Garman6:54

       Um, and I think that's where we've spent a lot of time is-

   26. RRRaghu Raghuram6:55

       Yeah, that plays to your strengths. Yeah

   27. MGMatt Garman6:56

       ... how do you think about, you know, e-exactly, how do you think about broad scale? How do you think about performance? How do you think about, um, you know, an interface that is, you know, a, a, a well-defined API interface that agents can actually-

   28. RRRaghu Raghuram7:08

       Yeah

   29. MGMatt Garman7:08

       ... easily traverse and work across? Um, and so that's also something that we both, I think we're naturally set up to do well, but that we've actually doubled down on to ensure things like, you know, how can you start a database in three seconds and, and, and, and really get to, um, some capabilities that agents are excited about.

   30. RRRaghu Raghuram7:25

       Um, I should have looked, but I haven't lately. But have you introduced, uh, any specific new services that are explicitly targeted at agents or people building agents?
4. 12:26 – 16:24

   Rethinking cloud for agents

   1. RRRaghu Raghuram12:27

      what has been some of the hardest things to accommodate as agents have taken over? I mean, you made a name-

   2. MGMatt Garman12:34

      Mm-hmm

   3. RRRaghu Raghuram12:34

      ... just the way you guys became the phenomenon that you became was serving developers-

   4. MGMatt Garman12:40

      Yeah

   5. RRRaghu Raghuram12:40

      ... right? Um, and then, uh, infra teams. Um, now developers are substituted by agents, and pretty soon infra teams are being substituted by agents.

   6. MGMatt Garman12:50

      Yeah.

   7. RRRaghu Raghuram12:51

      Are there-- What has been some of the hardest things for you as the largest service provider in the world to, to handle that transition?

   8. MGMatt Garman13:00

      Well, you know, look, I think, um, like I said, much of our infrastructure was, is-

   9. RRRaghu Raghuram13:05

      Yeah, the whole building blocks are broad

   10. MGMatt Garman13:06

       ... is, is actually pretty well set up-

   11. RRRaghu Raghuram13:07

       Yeah

   12. MGMatt Garman13:07

       ... to, to handle the scale, um, which is good. I, I think there's been a couple of things which are interesting, which are, I think, interesting paradigm shifts where, um, you could argue that some of our systems are, um, I don't, wouldn't say overengineered, but, um, as an example, many agents wanna create a database, uh, do a little bit of work, and then have the database go away. You don't really need that database to have five nines of durability.

   13. RRRaghu Raghuram13:32

       Exactly.

   14. MGMatt Garman13:33

       Right. And so there's some things that we rethink there, where when you're creating an Aurora database, like does your production database, like you do want five or-

   15. RRRaghu Raghuram13:40

       Yeah

   16. MGMatt Garman13:40

       ... n nines of data, you know, you want-

   17. RRRaghu Raghuram13:41

       Yeah, yeah

   18. MGMatt Garman13:42

       ... you, you, you need durability, you want availability, you want all of those things. And for the agent use case, that's, that's arguably or maybe not arguably like overengineered for, for what we need. And so, [clears throat] you know, it's hard. We don't, um-- And we kind of have this, this, uh, um, like a belief that we don't really wanna have like a non-durable option that's-

   19. RRRaghu Raghuram14:02

       Yeah

   20. MGMatt Garman14:02

       ... that's gonna cause problems either, 'cause you never really quite know if the database that's created wants to stay around for a long time or a short amount of time. And so we're, we're, um, trying to do the hard work to think about how do you accomplish both of those things, where you can create it quickly and throw it away, and you don't really waste a lot of resources. But, um, but, um, but if you do want it, and it can be durable and stay for a long time, it can actually grow into a big production database. So those are some trade-offs that we think about actively as we think about how does the, the, the more traditional kind of, "This is gonna be my production system," kind of capabilities, um, match with, uh, with, with some of the more transient nature of the infrastructure that agents, uh, want to use. Um, so that, that's one, I think. But there's, there's a, there's a bunch. I think the, um, scale and speed and latency of creation of, of durable resources is another one that's interesting. Um, you know, I think the other one that we actively think about too is just are there new building blocks that agents are gonna want that we just didn't really need before?

   21. RRRaghu Raghuram14:57

       Exactly.

   22. MGMatt Garman14:58

       And so compute sandboxes, um, uh, gateways, um, agent permissions versus people or, or, or service role permissions. Um, a lot of those are things that we have built and are building and thinking actively about 'cause they are just brand-new building blocks. It's not using the existing building blocks differently, but it's brand-new ones, where pretty, pretty clearly you want a different permissions for agents. You don't just wanna give it Raghu's permissions-

   23. RRRaghu Raghuram15:22

       Yeah, exactly

   24. MGMatt Garman15:22

       ... and let it go do whatever you can do. You actually want, um, you know, very time boxed, short-term permissions to just go do a task. You actually may not wanna give permissions to a whole tool in at all. You may wanna get very fine-grained permissions-

   25. RRRaghu Raghuram15:33

       Exactly

   26. MGMatt Garman15:34

       ... of what an agent's able to do from a sandbox, right? You actually want a sandbox-

   27. RRRaghu Raghuram15:38

       Yeah

   28. MGMatt Garman15:38

       ... that, um, uh, you wanna, um-

   29. RRRaghu Raghuram15:41

       Virtualization is having its act too-

   30. MGMatt Garman15:42

       That's right
5. 16:24 – 21:27

   The GPU allocation problem

   1. MGMatt Garman16:24

      And yeah.

   2. RRRaghu Raghuram16:25

      So I've rat-holed enough into agents. I'll come back to it later, but-

   3. MGMatt Garman16:28

      Yeah

   4. RRRaghu Raghuram16:28

      ... such a fascinating topic.

   5. MGMatt Garman16:29

      Super cool.

   6. RRRaghu Raghuram16:30

      But, uh, let me t- uh, switch gears a little bit and ask about something that, uh, every one of our startups face-

   7. MGMatt Garman16:37

      Yeah

   8. RRRaghu Raghuram16:37

      ... and we get asked most frequently, which is: How can we get GPUs, right?

   9. MGMatt Garman16:43

      [chuckles] Yeah.

   10. RRRaghu Raghuram16:44

       I mean, obviously you guys are running massive GPU farms-

   11. MGMatt Garman16:48

       Yeah

   12. RRRaghu Raghuram16:48

       ... and increasing it every day. But, uh, the top frontier labs gobble up every available GPUs, more power to them. How do you deal with the internal-- At the same time you wanna-

   13. MGMatt Garman16:59

       Yeah

   14. RRRaghu Raghuram16:59

       ... promote newer companies that are gonna become the enterprises of tomorrow, like you said. Um, how are you thinking about balancing-

   15. MGMatt Garman17:06

       Yeah, it's, it's a great question

   16. RRRaghu Raghuram17:07

       ... the capacity needs and these small companies that don't have a lot of credit and whatnot, so.

   17. MGMatt Garman17:13

       Yeah. Well, yeah, it, it's a great question. Um, and, and a couple of these things make it more challenging too. One is, um, the, the CapEx expense needed to go deploy kind of all of the compute that everyone needs right now, um, is massive, right? And so we are, uh, I think-

   18. RRRaghu Raghuram17:27

       I think your CapEx this year is what, two hundred or something?

   19. MGMatt Garman17:30

       Two hundred and twenty billion dollars, uh, for '26. It's, uh, I, I-- By, by-- At, at one point I saw that it's, you know, it's, it's a pretty-- I mean, that is a larger expense than we've ever had, maybe any company has ever had in a, in a single year. Um, and, um, and we don't anticipate slowing down anytime soon because the demand is just massive. And so at some point, you're limited by how fast can you build data centers, how fast can you deploy capital, how, how fast can you get memory and chips and all of those kind of things. Um, and so all of those are different points, constraints that we're, you know, whether it's- Power data centers, capital memory chips, they're all kind of constraints at various times. Construction people that build buildings like-

   20. RRRaghu Raghuram18:07

       Yeah

   21. MGMatt Garman18:07

       ... you know, construction people are, are, are at a premium today. Um, and so all of those we, you know, we, we work really hard to make sure happen. Um, and then with the capacity that we are able to deploy, which is still a huge amount and, and, and not enough, we know, um, we really think intentionally about what is that allocation strategy. And so, um, we're great partners with the, the large frontier labs, the Anthropics and OpenAIs and Meta and, and other large customers. And so those folks are, are really good customers of ours.

   22. RRRaghu Raghuram18:34

       Yep.

   23. MGMatt Garman18:34

       And, uh, and we wanna make sure that we invest in them. We have large enterprise customers, whether it's, uh, Salesforces or, or JPMCs or, you know-

   24. RRRaghu Raghuram18:42

       Yep

   25. MGMatt Garman18:42

       ... other large companies that, that have demand for fewer numbers of GPUs, but or, or accelerators. Sometimes they want Trainium, sometimes they want, um, uh, NVIDIA GPUs. Um, but we wanna make sure that we can support them as well. And we're very intentional about how we make sure that we have capacity for startups. And, and so what we do is we actually do allocate, and we basically say, okay, we're gonna keep-- We, we could... You're right, we could sell every single, um, uh, ex- you know, GPU or, or AI accelerator we had to probably just to the la- the, the big frontier labs and call-

   26. RRRaghu Raghuram19:11

       Yeah

   27. MGMatt Garman19:11

       ... it a day. Um, we choose not to do that 'cause we actually wanna keep growing the, the full ecosystem. They get a large number, but, but, um, we wanna keep supporting, um, a broader set of customers 'cause we actually think that both the whole ecosystem will be more healthy for us. There's some diversification, but it's also just we know these are gonna be big companies over time. Um, and so we try to support them. Um, I saw recently that we say, you know, yes, in some way, shape, or form to something like 60% of the, the requests we eve-eventually get. Sometimes it's a little bit later, sometimes it's in a different region, sometimes it's a slightly different configuration than the customers are looking for. Um, but we really try to lean in and, and, and try to allocate as much as we can. And every single startup you have wants more. [laughs]

   28. RRRaghu Raghuram19:50

       Yeah, exactly.

   29. MGMatt Garman19:51

       You know, and so-

   30. RRRaghu Raghuram19:52

       There's no question about it
6. 21:27 – 30:22

   Is the AI CapEx a bubble?

   1. RRRaghu Raghuram21:27

      Yeah. Yeah, of course.

   2. MGMatt Garman21:28

      And, um, uh, and you know, I think we were for the longest time actually, um, we, you know, we, we were investing ahead of where the demand was and, um-

   3. RRRaghu Raghuram21:39

      Yeah. No, I think that's-

   4. MGMatt Garman21:39

      ... you know, one of the, one of the most painful things is that with the real ramp of, of GPUs, like a lot of the elasticity has unfortunately, you know, kind of gone away.

   5. RRRaghu Raghuram21:46

      Just gone. Yeah.

   6. MGMatt Garman21:47

      Um, and so hopefully we'll get back to... And in our core compute and, and storage and things, that elasticity is still there and that, but, um, but you know, the, the, but, but so we've been spending for, for quite a bit of time and, and we feel really good about the spend that we're making now. Um, which is, you know, I think I get lots of questions sometimes about how you feel about that spend and like, are you nervous about-

   7. RRRaghu Raghuram22:06

      Yeah

   8. MGMatt Garman22:06

      ... the bubble and other things like that.

   9. RRRaghu Raghuram22:07

      Yep.

   10. MGMatt Garman22:07

       And I will say, you know, we, um, because of the position we have, like one, we do take this diversified approach, and so not all of our capacity is bundled up in one customer. And I think you go to a, some of these, whether they're Neoclouds or some of the, the other providers out there and, you know, you'll see sometimes concentrations of 30, 40, 50, 60%-

   11. RRRaghu Raghuram22:27

       Yep. Yep

   12. MGMatt Garman22:27

       ... with one or two customers. We're nowhere near that obviously. We're, you know, single digit percentages at the highest and, and, and usually it's less than that. For one, I think we have a lot re- less risk on, on one particular customer. But also because we have that rich set of services, like AWS is where people really are coming to launch their production workloads. And so the majority of our usage today actually is either core compute and storage and inference, which is part of that application. And so those are the workloads that I think just aren't gonna go away.

   13. RRRaghu Raghuram22:55

       Yep.

   14. MGMatt Garman22:55

       Just 'cause we see enterprises getting positive ROI. You go talk to the customers, and you say, "At the capability today and the cost today, are you seeing positive returns to your business?" And almost to a person, they'll say like, "Oh, yeah."

   15. RRRaghu Raghuram23:07

       Yeah.

   16. MGMatt Garman23:08

       And so you're like, well, that's not gonna go away. There's no bubble in which they, they stop spending-

   17. RRRaghu Raghuram23:11

       Yep

   18. MGMatt Garman23:11

       ... on that, you know.

   19. RRRaghu Raghuram23:12

       Yep.

   20. MGMatt Garman23:12

       And so, you know, and you know this, like the VC model, like is every billion-dollar startup gonna make it? No, they won't. But you know, that's, that's kind of the game.

   21. RRRaghu Raghuram23:20

       That is the game.

   22. MGMatt Garman23:20

       And it's been, that's been true for 50 years. [laughs]

   23. RRRaghu Raghuram23:22

       Yeah. Yep.

   24. MGMatt Garman23:22

       It is. They haven't always been billion-dollar startups, but-

   25. RRRaghu Raghuram23:24

       The numbers have changed, but the principles haven't changed

   26. MGMatt Garman23:26

       ... but, but the principles are the same, right?

   27. RRRaghu Raghuram23:27

       Yeah.

   28. MGMatt Garman23:27

       You, you bet on 10, and one makes it-

   29. RRRaghu Raghuram23:29

       Yeah

   30. MGMatt Garman23:29

       ... and pays for the other 10 or whatever the, whatever the percentage is. Hopefully it's higher than one out of 10.
7. 30:22 – 33:07

   Clearing up data center myths

   1. RRRaghu Raghuram30:22

      Yeah. So, um- Obviously, there's a lot of wide-ranging debate about data centers.

   2. MGMatt Garman30:30

      Mm-hmm.

   3. RRRaghu Raghuram30:31

      Right? And, uh, it's clear that folks like us where we stand, but do you think as an industry we have not done a good job-

   4. MGMatt Garman30:39

      Yeah

   5. RRRaghu Raghuram30:39

      ... of explaining, uh, why data centers are good for America-

   6. MGMatt Garman30:44

      Mm-hmm

   7. RRRaghu Raghuram30:45

      ... and generally the world, but, uh-

   8. MGMatt Garman30:47

      Yeah

   9. RRRaghu Raghuram30:47

      ... and, uh, how do you-- what's the internal talk-

   10. MGMatt Garman30:51

       Yeah

   11. RRRaghu Raghuram30:51

       ... amongst Andy's team on, uh, how do we deal with this?

   12. MGMatt Garman30:54

       Yeah. Well, look, I, I think, um, and, and I think you'll hear more from us over this, and I agree, I think we need to be more vocal and be more fr-upfront, um, 'cause we actually do a ton that's, that's really beneficial, both for the communities we operate in, um, for the... You know, we, we think a ton about how do we bring renewable energy to these data centers?

   13. RRRaghu Raghuram31:11

       Yeah.

   14. MGMatt Garman31:11

       How do we think about being water positive? Um, actually, our, our, our data centers use a, a really, really small amount of water. We mostly use free air cooling. How do we think about being great participants in the communities where we are and, um, and how do we bring high-paying jobs to the communities we operate in? Um, and not all data center operators do that.

   15. RRRaghu Raghuram31:28

       Yeah.

   16. MGMatt Garman31:28

       And I think there are some well-

   17. RRRaghu Raghuram31:29

       And you have a 20 year-

   18. MGMatt Garman31:29

       There are some well-chronicled examples of others out there that are not great, um, at that, and they just don't really pay attention to regulations. They think that the rules don't apply or they just, you know, launch really quickly without thinking about those. And I think that it causes a problem for the whole industry 'cause everybody kinda gets lumped into that.

   19. RRRaghu Raghuram31:45

       Yeah.

   20. MGMatt Garman31:45

       And so, um, so look, I think we'll be-- we, we, I think you're right. We, um, you know, vocally self-critical, we need to be more vocal about the benefits that we do bring, and think about additional ways that we can help communities understand the benefits that, that we bring to them, both for the, the services they use, right? If, if you usually go to a community and say, "Well, do you not wanna use Netflix?" And they'll be like, "No, no. Like I, I still want Netflix." [laughs] Like it's, uh, um... And um, you know, like it's important for us to think about and, and highlight the benefits that we bring, where I, I recently saw a report where one of the communities that we operate in, the-- everybody in that county pays $5,000 a year less in taxes because of the taxes that we bring to that.

   21. RRRaghu Raghuram32:26

       Yep.

   22. MGMatt Garman32:26

       And we don't, we don't tell them. They don't even know it, right?

   23. RRRaghu Raghuram32:28

       Yep. Exactly.

   24. MGMatt Garman32:28

       It's just invisible to them.

   25. RRRaghu Raghuram32:29

       Yeah.

   26. MGMatt Garman32:29

       And so I think we just need to be more, um, um, clear about those benefits that we bring. 'Cause I think if you told the communities, "By the way, your tax bill is $5,000 less than it would otherwise be if we weren't here," they might have a little bit of different thought about-

   27. RRRaghu Raghuram32:43

       Yeah

   28. MGMatt Garman32:43

       ... the building that's over there.

   29. RRRaghu Raghuram32:44

       Yeah, yeah.

   30. MGMatt Garman32:45

       Um, so, and, and but, but not everyone does that. Not everyone kinda, um, um-
8. 33:07 – 40:06

   The Graviton and Trainium bet

   1. RRRaghu Raghuram33:07

      Yeah. And, uh, before we leave the hardware topic, uh, um, I wanna touch upon Trainium.

   2. MGMatt Garman33:13

      Mm-hmm.

   3. RRRaghu Raghuram33:14

      And your whole history with, uh, building your own, uh, chips, right?

   4. MGMatt Garman33:19

      Yeah.

   5. RRRaghu Raghuram33:20

      Um, we were one of your first partners using Nitro a long time ago.

   6. MGMatt Garman33:24

      Mm-hmm.

   7. RRRaghu Raghuram33:25

      And since then, uh, Graviton and you made tremendous progress. So what was the thinking that led to saying, "Look, we're gonna do our own thing"?

   8. MGMatt Garman33:33

      Yeah.

   9. RRRaghu Raghuram33:33

      And then how has that progress been and where are y-you-

   10. MGMatt Garman33:37

       Yeah

   11. RRRaghu Raghuram33:37

       ... with respect to Trainium?

   12. MGMatt Garman33:38

       It's, it's, it's actually a fascinating story, and I think it's a great example of where, um, Amazon AWS will innovate and will iterate over time and continue to think bigger about what we can do, but, but, um, but kind of prove our way there as opposed to, you know... And so take this as an example. Um, this is probably now, I don't know, ten, eh, it was probably about 13, 14 years ago, we were seeing that there was a pretty significant, um, virtualization tax on the overall number of resources. And, um, and we were kinda thinking about how do we-- Our customers were telling us, "I want bare metal performance," and they, you know, they're comparing to-

   13. RRRaghu Raghuram34:15

       Yeah

   14. MGMatt Garman34:16

       ... having all the resources of a, of a server. And so the first thing that we did is that, um, we took a, a network offload card and virtualized all of our network virtualization and pulled it off onto a, an offload card so that network virtualization got closer to, to bare metal performance.

   15. RRRaghu Raghuram34:31

       Yeah.

   16. MGMatt Garman34:31

       And back then it was not quite bare metal, but it was close- it was closer. And then we got really excited about that, and we said, "Okay, what if we could move storage virtualization off as well?" Right? And, and, um, and none of the network offload cards could do that. And then we s- we found this one company who had some Arm cores on an offload card, and they were doing it for other reasons. I can't remember their original purpose, but we're like, "Could you use those to do storage virtualization and some of these other functions?" And they're like, "Maybe." And so we really iterated with them. This was the Annapurna team.

   17. RRRaghu Raghuram34:58

       Yeah, this is Annapurna team.

   18. MGMatt Garman34:59

       And, um, and just loved that team. Like really innovative, really mission driven-

   19. RRRaghu Raghuram35:03

       Fantastic team

   20. MGMatt Garman35:03

       ... really wanting to solve problems. Um, and so we acquired them, and we said, "Look, could you build us a slightly bigger card that actually could take all the network virtualization off and basically give us a bare metal server that has no virtualization on it, no VM virtualization, everything is through APIs on the card?" And 'cause we had this view that, one, performance would be much better, resource re- uh, resource utilization would be better. It'd be-- We'd, we'd get-

   21. RRRaghu Raghuram35:26

       The security isolation.

   22. MGMatt Garman35:27

       Se- and security isolation would be much, much better. And we tell people, you know, could then legitimately tell people, we have no access to any of your VMs that are running there. And, and, um, and this has been a huge benefit for us for the last decade, where frankly, like we've been leading and, and others-

   23. RRRaghu Raghuram35:42

       Yeah

   24. MGMatt Garman35:42

       ... have been kinda slow to do this because they... This is not a generalized thing that people can do. But, um, so we got to that, and we basically said, "Look, we're, we're making a lot of progress here. Um, what if we take..." You know, and there's a bunch of Arm cores that were-

   25. RRRaghu Raghuram35:56

       Yeah

   26. MGMatt Garman35:56

       ... on this offload card, and we said, "What if we turn that into a server?" Uh, and we did that first with Graviton. It was a very underpowered, um, very small server that we launched. Um, and, and customers were excited. They're like, "Ah, I'd love to have an Arm server. This is super interesting." And so we went down the path and, um, and Graviton, um, you know... And part of what we did is we looked and saw that there's the, the slope of, you know, Arm cores were getting faster.

   27. RRRaghu Raghuram36:19

       Yep.

   28. MGMatt Garman36:20

       And where the-- You saw the power utilization and, and the graph where the... And you, you knew the intercept was gonna happen for where this architecture was gonna be really good for, for parts, and they just needed somebody to drive the ecosystem and, and get some of the pieces in place. So we did that with Graviton, um, and Graviton's been a, a runaway hit at this point. Um, you know, it's-

   29. RRRaghu Raghuram36:37

       Have you been public about what percent of your fleet is Graviton?

   30. MGMatt Garman36:39

       We land more Graviton ships, uh, uh, every year than, um- Uh, than, than any other-
9. 40:06 – 45:43

   Where enterprises are stuck on agents

   1. MGMatt Garman40:06

      it's both.

   2. RRRaghu Raghuram40:06

      Yeah. Now, uh, let's get back to, uh, uh, talking about agents, but from a perspective of, uh, large enterprises-

   3. MGMatt Garman40:13

      Mm-hmm

   4. RRRaghu Raghuram40:13

      ... or large medium enterprises.

   5. MGMatt Garman40:15

      Yeah.

   6. RRRaghu Raghuram40:16

      Where are they in their adoption? Um-

   7. MGMatt Garman40:18

      Mm-hmm

   8. RRRaghu Raghuram40:18

      ... and, uh, have they-- What sort of benefits are you seeing them reap-

   9. MGMatt Garman40:24

      Yeah

   10. RRRaghu Raghuram40:24

       ... already, and, uh, what is the roadmap for them as far as you can tell from your vantage point?

   11. MGMatt Garman40:30

       Yeah. It's, it's a really good question. I think it's one that, um, that we've spent a lot of time thinking about. And when I talk to customers all out there, um, today they view... You know, they're getting a lot of value out of what they've done today, and I would say the agents that most enterprises have built are, are relatively simple and straightforward. Um, and they're starting to think about, um... And they're, um, mostly non-autonomous, right?

   12. RRRaghu Raghuram40:57

       Yeah.

   13. MGMatt Garman40:57

       There are still kind of people in the loop, if you will. And so I think we're... And, and by the way, there's, customers are still getting lots of value out of that today, and so they're really thinking about, "How do I have these be autonomous but in a safe way?" And I think there's two things that, that I think hold customers back today from just continuing to, to scale. And it's, it's already a pretty big business today, but I think it's, it has a massive opportunity to really change every single customer out there and every single workflow and, and really thinking about it. And so number one is just how to think about it. I think what we vers- originally saw was that enterprises had a workflow, and they're saying, "Great, I would have the enterpri-- I would have an agent go do the same workflow."

   14. RRRaghu Raghuram41:35

       Yeah.

   15. MGMatt Garman41:36

       And what we encourage them to really do is think, not just replicate, you know, Bob does step one, two, three, four, five, so agent is gonna do one, step one, two, three, four, five, and then Bob's gonna check it at the end. That's not really the model you want. You want to actually step back and say, "If I wanna accomplish something, how can an agent do it differently?"

   16. RRRaghu Raghuram41:52

       Yeah.

   17. MGMatt Garman41:53

       It can do it in a massively parallelized way. It can try 50 different things and get to that. And, and how do you help it get to that right outcome and rethink how a computer would solve a problem versus a human solving a problem? And so one of the things is us just helping customers understand how to think about that and really kind of have that blank slate, because that's where you really get value, is not just replicating what you're doing today, but, but thinking from a greenfield approach about how you go solve a problem completely differently. And I'm sure that's how many of your startups are thinking about this too.

   18. RRRaghu Raghuram42:22

       Yeah, yeah.

   19. MGMatt Garman42:22

       Is how do you help customers greenfield solve a problem, not replicate the thing that happens today.

   20. RRRaghu Raghuram42:27

       Yeah.

   21. MGMatt Garman42:27

       So that, that is number one.

   22. RRRaghu Raghuram42:28

       People running fleets of agents and swarms or whatever you wanna call them-

   23. MGMatt Garman42:32

       Yeah

   24. RRRaghu Raghuram42:32

       ... very common these days.

   25. MGMatt Garman42:33

       And, and, and you just wanna think about it, you know. And, and so enterprises are not as, as... Again, this is one where you learn from the startups and you try to how do you apply that to an enterprise world where they're, you know, an insurance company is not necessarily as forward-leaning, but they would love to figure out how they can have a better, um, you know, approval workflow or something like that. Um, so that's number one. But then the second one is this, this how do you turn those into fully autonomous workflows, and how do you actually trust the agents?

   26. RRRaghu Raghuram42:57

       Yeah.

   27. MGMatt Garman42:57

       And so we're spending a lot of time thinking about how do we build services to- Help enterprises feel like their systems are secure, and that they can trust an agent to make a decision, that they can have the right guardrails, they can have the right permissions on their data, that it's not gonna delete production systems, that it's not gonna make tragic mistakes. And right now, I think that nervousness is probably holding people back-

   28. RRRaghu Raghuram43:20

       Yeah, that's huge

   29. MGMatt Garman43:20

       ... that maybe appropriately, by the way, is holding enterprises back from just saying, "Okay, go nuts." Like, you don't actually want an agent to just go crazy and accidentally delete a production database. That's gonna be-

   30. RRRaghu Raghuram43:28

       Yeah
10. 45:43 – 56:03

    AI risk, Hugging Face, and Continuum

    1. RRRaghu Raghuram45:44

       going into a big frontier model. What enterprises should really do is to take an open source model and then post train on your own data and workflows and traces and whatnot.

    2. MGMatt Garman45:54

       Mm-hmm.

    3. RRRaghu Raghuram45:55

       Where do you stand on that? Are you seeing customers actually trying to do that-

    4. MGMatt Garman45:58

       Yeah

    5. RRRaghu Raghuram45:58

       ... or do you guys-

    6. MGMatt Garman45:59

       Well, yeah

    7. RRRaghu Raghuram46:00

       How do, how do you think about that?

    8. MGMatt Garman46:00

       It's, it's a great-- The first point, I, I wholeheartedly agree on that first point, like the customer's en-- like enterprise data is their most valuable asset.

    9. RRRaghu Raghuram46:08

       Yeah.

    10. MGMatt Garman46:08

        And so from the very beginning, it's why we built Bedrock like we did. Um, we have a guarantee that your data never leaves your VPC. And so you're-- if you're running inside of Bedrock, um, your data doesn't go back to the model provider.

    11. RRRaghu Raghuram46:21

        Yeah.

    12. MGMatt Garman46:21

        They never see your prompts. Um, that stays inside of your own trusted environment. Um, and so that, that is why, um, enterprises kind of run, they, they prefer to run on top of Bedrock, and it's why you see that business growing massively, like hundreds and hundreds. I mean, it's e-every, um, every month we see that just business, just every week we see that business exploding, and then, and it's why, um, you see OpenAI workloads migrating to Bedrock. It's why you see Anthropic really growing really rapidly. I mean, so whether you're using open models or, um, uh, or, or, um, closed frontier models, um, I think Bedrock is a great solution that our en-- our, our customers told us, by the way, like, you know we-- if you remember three years ago, I got a lot of heat-

    13. RRRaghu Raghuram47:02

        Speaking of bad names, that's a good name, though.

    14. MGMatt Garman47:04

        Yeah, Bedrock is a good name.

    15. RRRaghu Raghuram47:05

        Yeah.

    16. MGMatt Garman47:05

        That's good. But we, we got a lot of heat actually for being slow to the AI world-

    17. RRRaghu Raghuram47:08

        Yes

    18. MGMatt Garman47:08

        ... because we actually built the foundations of this where we said, "Look, we're not just gonna rush out a service. We really wanna think about how do we make sure that we protect our customers' data and build a service that we think is gonna be durable for the use cases that we knew about." And if you remember, we got a lot of heat, and we said, "Look, we're gonna go build the right thing." And now as people move from proof of concepts to production, vast majority of them are landing in AWS on Bedrock for, for muchly-- One of the reasons is because of this. It's also because of the, the set of services that we have. We also offer open models. We offer proprietary models. Um, we offer, um, a whole set of capabilities around those Agent Core. We build these building blocks, so it's easier to build agents with any of the models that you want, um, whether they're in Bedrock or out of Bedrock-

    19. RRRaghu Raghuram47:49

        Yeah, yeah

    20. MGMatt Garman47:49

        ... for that matter. You can use Gemini or other things for it. But, um, but I think that's a, it's a, it's a differentiating piece for us, and it's a super important thing to think about because having that data go back into the, the model provider, I think is a, is a dangerous thing. You talk about open-weights models, though. I do think that there's a, a scenario that I'm excited about. Um, uh, we're, we're really ramping up our support of open-weights models and, and trying to build a good, um, environment. And, and frankly, this is where today, I, I think, and I think it's true, a lot of customers believe that they have meaningful proprietary data that if they could-

    21. RRRaghu Raghuram48:25

        Yeah

    22. MGMatt Garman48:25

        ... mix in, you know, do some post-training, do some fine-tuning, um, um, to an open-weights model that they could distill down, they can actually get a better performing model at a lower price. Um, there's a bunch of pieces in here that have to work out well. They actually have a good eval to actually prove that that's true.

    23. RRRaghu Raghuram48:40

        Yeah.

    24. MGMatt Garman48:40

        Uh, most people are doing that in SageMaker today. I think there's more that we can do to make that easier.

    25. RRRaghu Raghuram48:44

        Mm-hmm.

    26. MGMatt Garman48:44

        But actually, like, if you go look at where are people doing that, they actually do it in SageMaker on AWS today.

    27. RRRaghu Raghuram48:48

        Oh, really? Okay.

    28. MGMatt Garman48:48

        And they actually host the inference via SageMaker.

    29. RRRaghu Raghuram48:51

        So SageMaker, SageMaker is getting a new lease of life as a-

    30. MGMatt Garman48:53

        Uh, I mean, it is. It's, it's-

Episode duration: 56:17