Tom Brown: How Building GPT-3 Led to Founding Anthropic

1. 0:00 – 2:30

   From Failure to Success

   1. SPSpeaker0:00

      When we started out, we didn't seem like we were gonna be successful at all. (laughs) OpenAI had a billion dollars and like all of these, all of this star power, and we had seven co-founders (laughs) in COVID, like, trying to build something, and we didn't know if we were necessarily going to make a product or what the products would look like. One thing that's interesting to look at is just that humanity is on track for like the largest infrastructure build-out of all time.

   2. TBTom Brown0:25

      Tell us about the early days of Anthropic. So, you had a general idea of this sort of like long-term mission that you wanted to do to, you know, not destroy humanity. But like, what did you actually work on for the first year? How did that converge on an actual product?

   3. GTGarry Tan0:45

      Welcome back to another episode of The Light Cone. Today, we've got a real treat, co-founder of Anthropic, Tom Brown.

   4. SPSpeaker0:52

      Excited to be here.

   5. GTGarry Tan0:53

      So, Tom, one of the things that a lot of the people watching, uh, would love to figure out is, you got started in tech at the age of 21, fresh from MIT. How does someone go from that in 2009 to literally co-founding something as important as Anthropic?

   6. SPSpeaker1:12

      Summer 2009, Linked Language, two of my friends had started that out. I think they had seen one of our other friends, Kyle Vogt, kind of do a YC company, and so it was in the water that that's a thing that we could try to do. They started out, I was the first employee. Back then, yeah, you guys let me join for all the dinners and stuff like that too. I could have instead gone to like a big tech company or something like that, and I think probably just as a software engineer, I might have learned more software engineering skills. But I think by being there with the other co-founders, without anyone telling us what to do. (laughs)

   7. TBTom Brown1:47

      (laughs)

   8. SPSpeaker1:47

      Basically, we like, we had to figure out how to live, how to like... The company would die by default. I- I think in school, there was a lot of like a feeling of more of, people would give me tasks and I would do the tasks. It was kind of like a dog waiting for like food to be-

   9. TBTom Brown1:58

      (laughs)

   10. SPSpeaker1:58

       ... like, fed to them in their bowl or something like that. And I think for that company, it was more like wolves, and we have to like hunt our for like food, otherwise like we're, our kids are gonna starve or something like that. I think that that mindset, I think, has been like the most valuable mindset, that shift that I've had for trying to do like bigger, more exciting things.

   11. GTGarry Tan2:17

       Yeah, big tech just teaches you to work at a big tech company, whereas, uh, it's much more fun to be a wolf.

   12. SPSpeaker2:24

       Yeah. (laughs)

   13. TBTom Brown2:25

       How did you go from like, so working at friend's startup to then you started your own one?

2. 2:30 – 4:12

   Early Startup Days at Linked Language

   1. SPSpeaker2:30

      So Linked was, um, we ran the company for a bit. I ended up going back to, to school afterwards, and then when I c- left school, I went to this company, MoPub.

   2. TBTom Brown2:41

      That mobile advertising thing, right?

   3. SPSpeaker2:42

      Yeah, yeah. I was like the first engineer there. I was like, "Okay, well, I want to be a wolf," but like I was really bad at programming also. (laughs)

   4. TBTom Brown2:50

      (laughs)

   5. SPSpeaker2:50

      I was like very, very struggling as like a, a, a like software engineer. I know I want to do more, but I don't know how to do it yet. And so I think that was kind of like a experience getting to scale something. Winter 2012, one of my friends, who was my smartest friend from college, pitched me on, "Let's go and start a YC company." We did, at the time, Solid Stage. This was before Docker existed. And so the idea was try to make it easier to do DevOps, but Docker doesn't exist, so it's going to be a more flexible Heroku, which basically meant a more complicated (laughs) like Heroku. And so we... I remember we like, we interviewed with you guys. I think folks didn't really understand what we were trying to build. I think we didn't really understand what we were trying to build that much.

   6. GTGarry Tan3:30

      When you're trying to do something new, that's actually sometimes common.

   7. SPSpeaker3:33

      Yeah, I think we were an outlier there 'cause we like did our interviews and then we got called back, driving back to San Francisco, and TLB had written on the board like an angry frowny face-

   8. TBTom Brown3:42

      (laughs)

   9. SPSpeaker3:43

      ... and "What are you actually going to build?" (laughs)

   10. GTGarry Tan3:46

       (laughs)

   11. TBTom Brown3:46

       (laughs)

   12. SPSpeaker3:46

       And so he like wanted us to explain that. I guess we explained it enough or he was just like, "These guys still don't know what they're doing, but maybe they'll figure it out." Halfway through, I kind of felt, I still didn't actually understand what we were gonna build and how we would attach a mission to it that like I wanted to, to work on for my whole life.

   13. TBTom Brown4:05

       Yeah.

   14. SPSpeaker4:06

       Um, and so I left. PG actually introed me to Michael Waxman.

   15. TBTom Brown4:10

       I knew him as he had made the entry.

   16. SPSpeaker4:10

       Who was the Grouper founder.

3. 4:12 – 6:10

   The Grouper Dating Experiment

   1. SPSpeaker4:12

      Yeah.

   2. TBTom Brown4:12

      Yeah.

   3. GTGarry Tan4:12

      So Grouper was a dating app, only it was novel in that you had, what, three guys and three girls?

   4. SPSpeaker4:19

      Yeah.

   5. GTGarry Tan4:19

      This is before AI in a lot of ways. So there was like a set of, a team of people who would manually link people up, right?

   6. SPSpeaker4:25

      Yeah, yeah.

   7. GTGarry Tan4:25

      And they'd meet up at a bar and-

   8. SPSpeaker4:28

      Yeah.

   9. GTGarry Tan4:28

      ... shenanigans would ensue.

   10. SPSpeaker4:29

       Yes. Reliably shenanigans would ensue.

   11. TBTom Brown4:32

       (laughs)

   12. GTGarry Tan4:32

       (laughs)

   13. SPSpeaker4:32

       People didn't always have a great time. I think you went, you went on a couple with them, right?

   14. TBTom Brown4:35

       I went on a bunch with these guys. Yeah.

   15. SPSpeaker4:36

       Okay, yeah. (laughs) The pitch for Grouper for me, for like why I was excited for it, was just, I was like an incredibly awkward kid. What I wanted to do was to basically have a thing that lets awkward people like me go out and talk to other people, for me to talk to girls, and feel like I was safe doing it with like my friends around and stuff like that. And so I think who are going to be our employees was important. I did like all of our engineering interviews. We would take someone. The only person who went on more was Greg Brockman.

   16. TBTom Brown5:06

       Oh. (laughs)

   17. SPSpeaker5:07

       But I think he had-

   18. TBTom Brown5:08

       (laughs)

   19. SPSpeaker5:08

       I think he had a, he had a phase where like...

   20. GTGarry Tan5:10

       That's fascinating.

   21. SPSpeaker5:10

       ... every single week he would go and like post on, uh, Slack or HipChat at the time or whatever.

   22. TBTom Brown5:15

       'Cause he moved to New York and he was hanging out at the Recurse Center during this period, I think.

   23. SPSpeaker5:18

       Oh, I- I think he was at Stripe. May- maybe, maybe for part of it he was at Recurse, yeah.

   24. TBTom Brown5:23

       Yeah.

   25. SPSpeaker5:23

       But he also had, uh, I think just like a phase-

   26. TBTom Brown5:26

       As his money was at Stripe.

   27. SPSpeaker5:26

       ... where he would just, at Stripe, he would just like post in their thing...

   28. TBTom Brown5:28

       Oh, right, right, right.

   29. SPSpeaker5:28

       ... and be like, "I'm going on Grouper." Who's going for like a whole year. (laughs) So I, I ended up being close with Greg, which, which ended up being the connection to the OpenAI.

   30. SPSpeaker5:36

       What was the journey like? Because you started as, um, you just graduated from MIT CS. You were 21. You became first an early employee for all these YC startups. Then you started your company just a couple of years later. And what was the path for you to eventually become the co-founder of Anthropic? It was like a long path, but it's pretty impressive. How- how did you get there?
4. 6:10 – 8:42

   Making the Leap to OpenAI

   1. SPSpeaker6:10

      Yeah, so I left Grouper 2014, June 2014, and I joined OpenAI...I think a year later, I tried to, like, build up courage to make the switch to be a... to try to learn AI research. At the time, I was like, "Okay. It seems like sometime in our lifetimes, we might end up making transformative AI. If we do, that would be the biggest thing. Maybe there's some way that I could help out." But also, I got, like, a B- in linear algebra in college, and so it seemed like at the time you needed to be just top superstar in order to try to help out with that at all. And so I think I had, like, a lot of uncertainty about whether I would be able to help. And also, I'd had some success with startups, and so a lot of me was just like, rather than trying to retool at this, like, I could try to do another startup or something like that.

   2. SPSpeaker6:57

      I feel like in that period, um, going to work on AI research was just not seen as, like, a serio- like, not like a practically serious thing to do.

   3. SPSpeaker7:04

      Yeah.

   4. SPSpeaker7:04

      And you're in a world where it's like, people try and build companies and do these, like, really practical things.

   5. SPSpeaker7:08

      Yeah.

   6. SPSpeaker7:08

      So what did your... Were your friends like, "Oh, that's really cool, you're gonna go work on AI stuff," or was it-

   7. SPSpeaker7:12

      Not really.

   8. SPSpeaker7:14

      ... something... (laughs) Yeah, yeah, well, what was it?

   9. SPSpeaker7:14

      I think my friends were like, "That sounds, that sounds weird and bad," kind of. Like, it, it doesn't really seem like, it doesn't seem like, like AI safety is a thing we should be wor... Like, overpopulation on Mars doesn't make any sense.

   10. SPSpeaker7:24

       (laughs)

   11. SPSpeaker7:24

       And my friends were also just like, "I don't know if you're gonna be good at that, Tom." (laughs)

   12. SPSpeaker7:27

       (laughs)

   13. SPSpeaker7:27

       I think that for that reason, I think I didn't try very hard for... I, like, kind of flip-flopped on it for, like, six months, trying to build up courage to do it.

   14. SPSpeaker7:36

       A- and what were you specifically, at this point, like, you're reading research papers? Like, what did, what does it look like?

   15. SPSpeaker7:41

       Yeah. So first, I was just kinda hanging out. I built, like, an art car for Titanic cent-

   16. SPSpeaker7:46

       (laughs)

   17. SPSpeaker7:46

       ... and stuff like that. (laughs)

   18. SPSpeaker7:47

       Oh, that was fun, yeah. We did that.

   19. SPSpeaker7:48

       Yeah, yeah, yeah. (laughs)

   20. SPSpeaker7:49

       That was amazing.

   21. SPSpeaker7:49

       So I, I spent, like, a whole summer, like, three months after Grouper, doing that, 'cause honestly I was, I was, like, kind of burned out, uh, for Grouper, where, I don't know, startups, like, the highs are high, like, the lows are low, and we weren't working at the end. Our business wasn't succeeding. Our revenue was going down, but I... My main job still was, like, recruiting engineers, and so I had to, like, pitch them on this dream that I'd had but I, like, no longer really believed in. (laughs)

   22. SPSpeaker8:08

       It sounds like a death march.

   23. SPSpeaker8:10

       Yeah.

   24. SPSpeaker8:10

       It's tough.

   25. SPSpeaker8:11

       And so I was super burnt out, and I was like, "Okay, Tom," like, "chill out, do some yoga, like, do some CrossFit."

   26. SPSpeaker8:16

       (laughs)

   27. SPSpeaker8:16

       Like, "build an art car." And so-

   28. SPSpeaker8:17

       What was the hindsi- like, you know, hindsight's 20/20. What's the retrospective on, like, Grouper obviously attracted all these really, really smart people. The graphs were up and to the right, and then it flat-lined and maybe started declining. What happened?

   29. SPSpeaker8:31

       I think that when we started, the competition was like OkCupid.

   30. SPSpeaker8:34

       Mm-hmm.
5. 8:42 – 10:12

   First Product Launch Challenges

   1. SPSpeaker8:42

      put yourself out there and go, like, talk to someone new, and they might just be like, "I don't want to talk to you. You seem weird." And so we solved that by just m- blind matching. Tinder came out while we were doing Grouper, and Tinder solved that same problem with both people have to show interest before you get matched. So there's also no worries about getting rejected, and I think that they just had better... that was a better solution to that same problem. So good work, Tinder. Good work, all the swipers. I think that that, that solved the, like, mission that we were trying to solve better than we solved it.

   2. SPSpeaker9:11

      And then, yeah, like, when did you get serious about AI, and just how did you approach it?

   3. SPSpeaker9:16

      Three months of, like, kind of playing and having fun, and then I ran out of money (laughs) also.

   4. SPSpeaker9:20

      (laughs)

   5. SPSpeaker9:21

      (laughs)

   6. SPSpeaker9:21

      When I had, like, my personal runway, I, I ran out. And so I was like, "Okay. I think that I'm gonna need six months of self-study to have a shot at getting a job." At that point, it was DeepMind or Google Brain were the two places to do work there. Or MIRI. MIRI was the third one that I was, like, looking at.

   7. SPSpeaker9:39

      Okay. I see.

   8. SPSpeaker9:39

      So I was like, "If I wanna help out with that, those are the three places to look at. I don't have any of the skills yet. I need six months of self-study to feel like I would not be a drag on them and, like, actually be helping instead."

   9. SPSpeaker9:49

      Can you, um, ex- maybe explain a bit what was that self-study like? Because I'm sure there's a lot of software engineers right now in their twenties that are looking-

   10. SPSpeaker9:57

       Yeah.

   11. SPSpeaker9:57

       ... to retool, to become AI researchers. What was, what was that six months like? Even though (laughs) , as you said, you had, uh, gotten a B- in linear algebra-

   12. SPSpeaker10:05

       Yeah.

   13. SPSpeaker10:05

       ... which is, like, core.

   14. SPSpeaker10:06

       Might have been a C+. I'm not (laughs) I should check. (laughs)

   15. SPSpeaker10:10

       (laughs)

   16. SPSpeaker10:10

       I'm gonna give ..........................

   17. SPSpeaker10:10

       But that's pretty impressive, where, where you got

6. 10:12 – 12:44

   Self-Teaching AI Research

   1. SPSpeaker10:12

      to.

   2. SPSpeaker10:12

      Yeah, yeah. It turned out okay. First, I did a contract actually with Twitch, um, and, like, earned, like, enough to have that six months of runway.

   3. SPSpeaker10:19

      Okay.

   4. SPSpeaker10:19

      So I did, like, three-month contract with Twitch, and then I made a, a plan to self-study. I don't think it's the right plan now for people too, 'cause this is 2015. What did it look like? It was, like, take a Coursera course on machine learning, try to solve some Kaggle projects, read Linear Algebra Done Right-

   5. SPSpeaker10:37

      Mm-hmm.

   6. SPSpeaker10:37

      ... and, uh, I had a statistics textbook. I think I had YC alumni credits, and so I bought, like, a GPU (laughs) and I would, like, SSH into the GPU to, like, work through my courses for it.

   7. SPSpeaker10:50

      And this is right after... Yeah, it was al- already after AlexNet, right?

   8. SPSpeaker10:54

      This is after AlexNet, yeah. So I was mostly doing image, image classification stuff that I was trying to learn. It was, like, the thing that all the courses would teach you to do.

   9. SPSpeaker11:02

      How did you get the OpenAI job?

   10. SPSpeaker11:04

       Yeah.

   11. SPSpeaker11:04

       Because you were one of the few engineers. It was mostly researchers, and they had a pretty stacked team of researchers.

   12. SPSpeaker11:10

       I messaged Greg, um, as soon as OpenAI was announced, and I was like, "I'd love to help out in some way. I got a B- (laughs) in my linear algebra-"

   13. SPSpeaker11:18

       (laughs)

   14. SPSpeaker11:19

       "... but I know some engineering, I've done a bit of distributed systems work. If you guys need help, I'm, like, happy to mop floors if, if you guys need. I want to help out, however." And I think Greg was like, "Yeah, I think there's, like, a paucity of people who..." And, and he said paucity too, and I was like-

   15. SPSpeaker11:32

       (laughs)

   16. SPSpeaker11:33

       ... "Fancy word there." (laughs)

   17. SPSpeaker11:33

       (laughs)

   18. SPSpeaker11:34

       "There's a paucity of people who know both machine learning and distributed systems, so, like, yes, you should do that." I think he introduced me to Pieter Abbeel also, to help me put together, like, a little course for myself too. And then I checked in on... with him, I think, every month or something, and then after a couple months, he was like, "Oh, we actually have a project which is, uh, we need to put together. We wanna play a ga- like, play games."

   19. SPSpeaker11:56

       Oh.

   20. SPSpeaker11:56

       "Can you help, uh, make StarCraft environment?" And so I joined to, like, help them with the StarCraft, uh, environment. So that, that ended up, I think, getting my foot in the door. I, I didn't do any machine learning work with them for the first nine months that I was there, basically.

   21. SPSpeaker12:11

       And what did OpenAI feel like at this point? Like, had it raised much funding? Did it have, like, an office? Does what... Or did-

   22. SPSpeaker12:17

       Yes.

   23. SPSpeaker12:17

       ... did it feel like a startup?

   24. SPSpeaker12:18

       So it was in the chocolate, uh, on top of the Dandelion-

   25. SPSpeaker12:21

       Oh, okay.

   26. SPSpeaker12:21

       ... Chocolate Factory, um...

   27. SPSpeaker12:22

       This is after Greg's apartment, that's... That they got to-

   28. SPSpeaker12:25

       After Greg's apart-

   29. SPSpeaker12:25

       Yeah.

   30. SPSpeaker12:26

       So, like, right after Greg's apartment, in the chocolate factory. When it kicked off, right, it was, like, a billion dollars of committed funding from Elon. It felt like it was, like, very solid.
7. 12:44 – 15:44

   Building GPT-3 Infrastructure

   1. SPSpeaker12:44

      Yeah, for GPT-3.

   2. SPSpeaker12:45

      And for ... and g- how, how, what was that? Because you got from ... GPT-2 was in TPUs, right?

   3. SPSpeaker12:50

      Yep.

   4. SPSpeaker12:51

      And the big breakthrough in GPT-3 was, like, use more compute and using GPUs.

   5. SPSpeaker12:55

      Yep. So I ended up working at OpenAI for a year. Left, went to Google Brain for a year, came back, and then GPT-3 was 2018 through 2019 was, like, building up to GPT-3, which exactly as you said, was like scaling things up. I think that, like, Dario had seen the big trend of scaling laws, basically.

   6. SPSpeaker13:13

      You, you, you published a paper for that.

   7. SPSpeaker13:15

      Yeah, yeah.

   8. SPSpeaker13:15

      And that's like a pretty important paper that now has withstood the test of time, and we're living now the dream of it.

   9. SPSpeaker13:23

      Definitely like seeing that line of reliably you get more intelligence if you spend more compute with the right recipe, was the main thing that was ... at least for me, was like this is a thing that's, like, happening, happening now. 'Cause you could look, even at the time, we weren't spending very much money- (laughs)

   10. SPSpeaker13:36

       Mm-hmm.

   11. SPSpeaker13:36

       ... on the, on the training jobs at the time, and you could see that there was scaling there. And then also Danny Hernandez did a paper at the time that showed, uh, how much cheaper algorithmic efficiency was making stuff over time too. And like those two things stack together, that was like, "Oh, wow, we're going to get a lot more intelligence over the next few years."

   12. GTGarry Tan13:56

       So it was noteworthy and surprising when you saw it?

   13. SPSpeaker14:01

       Yeah. And I, I think the thing that seemed the weirdest to me is like I'm not a physicist, but like all these physicists were, were doing this stuff. The like original scaling laws paper, just the like very straight line over like 12 orders of magnitude. I'm just like-

   14. GTGarry Tan14:12

       Got it.

   15. SPSpeaker14:12

       ... 12 orders of magnitude is like (laughs) , just like a stupidly large amount of ... I've like never seen anything go over 12 orders of magnitude. That convinced me to definitely pivot all of my work into scaling, which I, I hadn't been doing before.

   16. GTGarry Tan14:25

       Can I ask a, like, kind of layperson question?

   17. SPSpeaker14:27

       Yeah, yeah.

   18. GTGarry Tan14:27

       I mean, is it fair to say that the scaling law m- might show up in all of these other domains, then they're like ... are there like two, five, 100, 10,000 domains where the scaling law could hold that we're just not investing into?

   19. SPSpeaker14:42

       Yeah, so I think in physics, scaling laws hold all over the place, which I didn't know at the time. But, um, within physics, like there's a whole field called phenomenology that basically looks at various aspects of the world and then does those types of fits. And they, they find these like power log distributions all over the-

   20. GTGarry Tan15:00

       Yeah.

   21. SPSpeaker15:00

       ... all over the place. This was like I think the first one that I had ever seen in a, um, like computer science adjacent thing, which I th- I think was like interesting and surprising and ...

   22. GTGarry Tan15:12

       And at the time, it was ... people were mad about it. They were, actually were like, "You're throwing money at GPUs." Or-

   23. SPSpeaker15:17

       Yep.

   24. GTGarry Tan15:17

       ... "You're just like wasting money. This is very wasteful."

   25. SPSpeaker15:20

       Yep.

   26. GTGarry Tan15:20

       That was sort of the vibe.

   27. SPSpeaker15:20

       People are still mad about that. (laughs)

   28. GTGarry Tan15:22

       (laughs) Yes. Different people now, but-

   29. SPSpeaker15:24

       Yeah.

   30. GTGarry Tan15:25

       ... still people mad about it.
8. 15:44 – 18:23

   The Anthropic Spinoff

   1. SPSpeaker15:44

      Can you, uh, tell us then how you ended up collecting the last Infinity Stone?

   2. SPSpeaker15:49

      With Anthropic. (laughs)

   3. SPSpeaker15:50

      Yeah, with Anthropic, because there's very few people in the world that have basically worked at OpenAI, DeepMind, and Anthropic, and you s- were part of the team that spun off from GPT-3.

   4. SPSpeaker16:00

      Yep.

   5. SPSpeaker16:00

      And then started Anthropic. So how was, how was that jump?

   6. SPSpeaker16:04

      There were two teams there. There was the safety org and the scaling org, were the two orgs that reported in to Dario and Daniela. I think we had just like worked together extremely well. One thing I think that was great, both at OpenAI and at, and at Anthropic was just like we had a culture where like everything is on Slack, 100% of things on Slack. And within that, all public channels. Great communication. I think that that group also was the group that took the scaling laws the most seriously, where it was like, okay, like this actually is going to be transformative. There's going to be a handoff where like humanity will hand off control to transformative AI at some point, and hopefully like they'll be aligned with us and like that'll be a good transition that goes well. But it might not be. The stakes are incredibly high. And so I think that group was very focused on like how do we make sure that that's taken seriously enough and that like we've built an institution that can handle the weight of that. That ended up being the core group that left to join Anthropic. And I think, I think it wasn't clear at all to me that like that was the right thing for the world at the time. In hindsight now, it seems like that was a good choice. I think what was kind of cool then too is when we started out, we didn't seem like we were gonna be successful at all. (laughs) OpenAI had a billion dollars and like all of these ... all of the star power, and we had seven co-founders (laughs) in COVID like trying to build something. And we didn't know if we were necessarily gonna make a product or what the products would look like. And so I think that what was interesting from that too is that all of the initial people who joined were there for the mission too. They all could have worked somewhere else for more prestige, more, more, more money. People would have known what they were doing, et cetera.

   7. SPSpeaker17:45

      Or stayed at OpenAI.

   8. SPSpeaker17:47

      E- exactly, yeah, that, that ... exactly. That's been an interesting thing then that I think has been like the key to like letting our culture or like let our org scale. We're like 2,000 people now. But we still have a thing where it doesn't seem like politics have creeped in. And I think a lot of that is like the first 100 people all were just there for the mission, so like if something starts to go wrong, they'll like raise their hand and be like, "It seems like this person might not be acting for the, for the mission."

   9. GTGarry Tan18:10

      YC's next batch is now taking applications. Got a startup in you? Apply at ycombinator.com/apply. It's never too early, and filling out the app will level up your idea. Okay, back to the

9. 18:23 – 20:21

   Early Days of Building Claude

   1. GTGarry Tan18:23

      video.

   2. SPSpeaker18:23

      Maybe tell us about the early days of Anthropic. So the, the seven of you broke off from OpenAI. You had a general idea of the sort of like ...

   3. JFJared Friedman18:32

      ... long-term mission that you wanted to do to, you know, not destroy humanity. But, like, how did, (laughs) what did you actually work on for the first year? How did that converge on an actual product?

   4. SPSpeaker18:42

      So first year, the main thing that I tried to do was just build the training infrastructure that we needed to train a model, and then get the compute that we needed to train the model. Those were, like, my two main projects. All the other things that you need to do when you're, like, uh, starting up a company too, so, like, set up a Brex account. (laughs) And like, I don't know, like, all of, all of that, all of that stuff. We started out with seven co-founders. Within, like, a few months, I think, like, 25 folks from OpenAI-

   5. JFJared Friedman19:09

      Mm-hmm.

   6. SPSpeaker19:09

      ... um, overall had joined. So we have, like, a pretty substantial team that, like, already knew how to work together too. And so that helped us get up and running faster.

   7. JFJared Friedman19:18

      And at what point did you launch the first product, and when did things begin to actually start working?

   8. SPSpeaker19:22

      So the first product that we launched was after ChatGPT. We had like, uh, maybe nine months before ChatGPT, we had a Slack bot version of, like, Claude 1.

   9. JFJared Friedman19:33

      Oh, yeah, we had that in the YC, uh-

   10. SPSpeaker19:35

       I remember that.

   11. JFJared Friedman19:35

       ... Slack actually. (laughs)

   12. SPSpeaker19:36

       Yeah, yeah. Yeah, I remember, uh, like, Tom Bloomfield adding all of you guys to it also. (laughs)

   13. SPSpeaker19:42

       (laughs)

   14. JFJared Friedman19:42

       It was really cool.

   15. SPSpeaker19:43

       Um, and then I think that at the time though, we didn't know whether or not we wanted to launch it as a product. We didn't know if doing so would be good for the world at the time. I think we hadn't really thought through our theory of impact that much, for, like, how we actually will make stuff work well. Plus, I think act- in hindsight, like, if we had tried to launch it, we, like, wouldn't have had the serving infrastructure to have done it. And I think because we weren't sure whether or not we wanted to, we, like, hesitated for too long on building that infrastructure, which I think is learning for, for me. (laughs) Um-

   16. JFJared Friedman20:13

       I mean, at this time, ChatGPT had not launched yet.

   17. SPSpeaker20:15

       ChatGPT hadn't launched. And so I guess we didn't know that it would be a big deal too.

   18. SPSpeaker20:18

       This is around the pandemic, 2022?

10. 20:21 – 22:08

    The ChatGPT Wake-Up Call

    1. SPSpeaker20:21

       This is summer of 2022, yeah. And then ChatGPT launched fall 2022. And then we relaunched our API after that, and then Claude AI after that also. I think it didn't seem like it was working basically until Claude 3.5 and coding. I think, like, really, really, like, through that whole time then until about a year ago, it seemed like it wasn't clear that we were gonna end up being, like, a successful company.

    2. SPSpeaker20:48

       We actually saw that in the startups, because we kinda get a bit of a vibe check in terms of what is the preferred model for startups. So all of 2023, OpenAI, OpenAI was the response.

    3. SPSpeaker20:59

       Yeah.

    4. SPSpeaker20:59

       Then things started to turn in 2024, is when, uh, we saw Claude 3.5 and especially Sonnet-

    5. SPSpeaker21:07

       Yeah.

    6. SPSpeaker21:07

       ... was starting to get up market share per se in the YC batches, going from single digit to at some point, like, 20 and to 30%. And especially for coding-

    7. SPSpeaker21:16

       Yeah.

    8. SPSpeaker21:17

       ... became the default choice, which is very interesting. Can you tell us about how that emergent behavior and the spikiness on that particular skill in-

    9. JFJared Friedman21:25

       Must be 80% now or 90.

    10. SPSpeaker21:27

        Yeah, for coding even more.

    11. SPSpeaker21:28

        Yeah.

    12. JFJared Friedman21:28

        Yeah.

    13. SPSpeaker21:28

        Especially now Claude Code. What was that? Was that on purpose or just kind of happened?

    14. SPSpeaker21:33

        I think that we invested more in trying to make the model really good at code because we wanted the model to be good at code, (laughs) was one thing. Um-

    15. SPSpeaker21:41

        And you did it. (laughs)

    16. SPSpeaker21:43

        (laughs) Yeah. And then I think seeing, seeing the reaction of everyone too, it was like, "Okay, yeah, like, let- let's go much harder on that also."

    17. JFJared Friedman21:50

        And this is before 3.5 Sonnet. You'd already invested enough in coding to realize that that was really promising and you decided, decided to double down.

    18. SPSpeaker21:57

        I think this really was, like, individuals within the org being like, "We wanna do coding, uh, before 3.5 Sonnet." And then when we saw 3.5 Sonnet's really good product market fit, that was good signal to, like, go,

11. 22:08 – 24:13

    Claude 3.5 Sonnet Breakthrough

    1. SPSpeaker22:08

       go for that.

    2. JFJared Friedman22:08

       And did you guys know, like, the day that you guys launched 3.5 Sonnet, did you know that you had something really special and this was going to be the turning point for the company? Or were you as surprised as OpenAI when they launched ChatGPT and it just, like, unexpectedly took off?

    3. SPSpeaker22:21

       I, yeah, I, I wish that, I wish that we had, like, more foresight on that.

    4. JFJared Friedman22:25

       (laughs)

    5. SPSpeaker22:25

       But no, I think, I think it was surprising for us too, like how, how big of a deal it was. And then I think 3.7 Sonnet also, like, surprised us by how much it unlocked, like, agentic coding. I think for, for each of these things, yeah, we moved quite fast in rolling them out. And so we really, um, often don't know what the results are going to be there.

    6. SPSpeaker22:43

       I think it's what made a lot of these coding agent startups work. I mean, there's a crazy story of Replit winning, going to 100 million in, uh, just 10 months, right? There's Cursor of course, a story and all built on all these with, with Sonnet.

    7. SPSpeaker22:59

       I think that all, all of those things have been surprising to me. And then also just, like, in my working with Claude too, like, I think I continue to be surprised by, like, the type of stuff that it can do. And I, I do think with each one there's, like, more stuff that kind of unlocks. But one of my friends was telling me that she had some code that she, uh, some closed source tool that she wanted to modify, but she didn't have the source code for it. She had the compiled binary and she's like-

    8. SPSpeaker23:20

       Oh.

    9. SPSpeaker23:20

       ... "Claude, can you, can you decompile this?"

    10. JFJared Friedman23:22

        No way.

    11. SPSpeaker23:22

        Like, yeah, "Can you, can you disassemble the assembly?"

    12. SPSpeaker23:25

        Oh, man.

    13. SPSpeaker23:25

        And Claude, Claude chewed on it for 10 minutes and, like, made a C version of it. And so then she had the thing-

    14. JFJared Friedman23:30

        Oh, my God.

    15. SPSpeaker23:31

        ... that she could modify. Yeah.

    16. SPSpeaker23:32

        (laughs)

    17. SPSpeaker23:32

        Which is insane. She's like, "Yeah, and like, if I spent three days on it, I probably could have gotten the hex tables and, like, written a little code to do it." But, like, it did the whole thing, made up variable names for them, et cetera. So I do think that, like, we keep getting surprised by stuff that model has memorized all the hex tables, it can think through, try to work through it. I think we're gonna continue to be surprised by that sort of stuff too.

    18. JFJared Friedman23:51

        If you poll, like, the YC founders, they prefer using Anthropic models for coding by, like, a huge margin that's much larger than what you would predict if you just looked at the benchmark results.

    19. SPSpeaker24:01

        Mm-hmm. Yeah.

    20. JFJared Friedman24:02

        So there, there seems to be some X factor (laughs) that makes people really like these models for coding. Do you know what it is? And is it intentional in some way or it just came out of the black box

12. 24:13 – 26:20

    Why Benchmarks Don't Tell the Whole Story

    1. JFJared Friedman24:13

       somehow?

    2. SPSpeaker24:13

       I think that the benchmarks, benchmarks are, like, easy to game, where I think that all the other big labs I think have teams where they, like, their whole job of the team is to, like, make the benchmarks scores good. And we don't have such a team. And so I think that, I think that that is probably the biggest factor there.

    3. JFJared Friedman24:29

       You don't teach to the test.

    4. SPSpeaker24:31

       We don't teach to the test.... because I, I do feel like if you start doing that, then like, it has weird bad incentives. Maybe we could, like, put that team under marketing or something like that and then ignore all the benchmarks. But I think that that's one reasons why there's some train test mismatch there. (laughs)

    5. SPSpeaker24:45

       So the evaluations are more qualitative, uh, internally? You have your internal benchmarks-

    6. SPSpeaker24:49

       Um, we have internal benchmarks, yeah. But we don't, we don't publish them.

    7. SPSpeaker24:52

       And is it the internal benchmarks that the teams are really focused on improving?

    8. SPSpeaker24:56

       That's right, yeah. So we have internal benchmarks that the team focuses on im- improving, and then we also have a bunch of tasks. Like, I think that, uh, accelerating our own engineers is like a top, top priority for us too. And so we, we do a ton of, like, dogfooding there to make sure that it's helping with our folks too.

    9. GTGarry Tan25:12

       Going back to Gold- Golden Gate Claude, there's a lot of sort of inter- the interpretability seems like it's a big part of it, and then most people would say that, you know, Claude's personality just feels better.

    10. SPSpeaker25:23

        Yeah.

    11. GTGarry Tan25:24

        And then how do you sort of at once be very quantitative, but then also, you know, build evals around personality?

    12. SPSpeaker25:30

        The evals for personality are kind of complicated too, for like-

    13. GTGarry Tan25:34

        Mm-hmm.

    14. SPSpeaker25:34

        ... how, how do you tell if like Claude has like a good heart or something like that? (laughs)

    15. SPSpeaker25:38

        (laughs)

    16. GTGarry Tan25:38

        (laughs)

    17. SPSpeaker25:38

        It's like hard to, hard to know. Um, but I do think that that's like, uh, Amanda Askell's team's mandate is... I think she describes it as like being like a, a good world traveler where like it can like... Claude goes and talks with all sorts of people from different backgrounds, and like each of the people should come from it, come to that being like, "I, I, like, feel good about, like, this conversation that I've had." Interpretability I think is like a long term bet, right, where it's like right now the models aren't that scary but at some point they're gonna be more scary. And so I think the hope there is to have some ability to know what's actually going on under the hood when it becomes more intense.

    18. SPSpeaker26:09

        Then more recently, Claude Code's been a real success.

    19. SPSpeaker26:12

        Mm-hmm.

    20. SPSpeaker26:12

        Can you talk us through like how did that project get started internally? And again was it like a, uh... did you, like, know this time it was going to work or was it a surprise?

13. 26:20 – 28:51

    Claude Code's Secret Sauce

    1. SPSpeaker26:20

       Claude Code was, um, an internal tool also. So like try to help out our, our engineers within Anthropic that, uh, yeah, Boris, um, had like hacked together.

    2. SPSpeaker26:31

       As an internal Anthropic engineer wanting to build it for themselves?

    3. SPSpeaker26:33

       For internal eng- for other internal engineers, yeah.

    4. SPSpeaker26:35

       Okay, cool.

    5. SPSpeaker26:35

       Yeah, for him and other internal engineers. And then, um, I think yeah. I think we definitely didn't know that it would be successful out there. And I think, I think to s- to some degree like we really had fully just bet on the API before that with the intention being like there's like so many, so many startups out there with so many good ideas, who are we to, like, figure out what the right product is to build on top of this stuff? Uh, everyone out there is gonna build better stuff than us, and so put all of our effort into just making the best possible API. And I think that this surprised me as like okay, like, we actually were able to make something that like as a product was like better than the other products out on the market for this agentic use. I have like a... some theory that like part of that came from like a mind shift of seeing Claude as like the user, uh, for this thing too. For like Link that we were like trying to build things for teachers who are like our users. For, for Grouper it was like single people in New York mostly I guess. (laughs) Um, for this, the... I think really the, the like users are the developers but also I think the users is Claude. It's like give Claude the right tools that Claude can actually do that effectively, help Claude get the right contexts to work effectively. This team was like the most focused on Claude as like a user, which I think is like a little bit weird.

    6. SPSpeaker27:49

       That's true, and it makes sense that you guys would understand Claude the best.

    7. SPSpeaker27:52

       Yeah. I, I do think that that's a place where like startup founders though like can, can do that too, and I think that that's, that's probably a rich vein for people to like make tools that are better for models as users.

    8. GTGarry Tan28:04

       That's the perfect anthropomorphization (laughs) of like the LLM itself. Like the agent is one of the stakeholders, is one of the users that you would go after-

    9. SPSpeaker28:12

       Yeah.

    10. GTGarry Tan28:12

        ... and try to like empower.

    11. SPSpeaker28:13

        Yeah. Yeah, totally.

    12. SPSpeaker28:14

        Which actually makes a lot of sense why you guys actually got MCP to work-

    13. SPSpeaker28:20

        Hm.

    14. SPSpeaker28:20

        ... to do tool calling, because a bunch of other labs had tried to do something and the standard that stook... that, that really took off was yours.

    15. SPSpeaker28:29

        Yeah I g- I think that that seems like a similar one too where it's like-

    16. SPSpeaker28:31

        From, from that philosophy.

    17. SPSpeaker28:32

        ... it's like a model, model-focused... (laughs)

    18. SPSpeaker28:33

        Going back to Claude Code, so like success is really exciting, it's also scary for like Cursor and other companies that have built on top of the API. Like what's your advice to founders building products? Like how should they think about building on the API but also worrying about like Anthropic or one of the labs building something better than they can build?

14. 28:51 – 31:11

    Building for the AI Agent

    1. SPSpeaker28:51

       I think I was kind of surprised that Claude Code... like we, we did build (laughs) a thing that was like, uh, like the best in the market there too. It's not super clear to me what the big advantage was for us for Claude Code besides more empathy for Claude or something. (laughs)

    2. SPSpeaker29:05

       That's actually... I think that's actually a really interesting insight. Like it seems like the thing that, yeah, you were building for a specific user that you knew really well that other people wouldn't have thought to build for-

    3. SPSpeaker29:13

       Yeah.

    4. SPSpeaker29:13

       ... versus like you had some like intrinsic technology advantage.

    5. SPSpeaker29:17

       Yeah. Like I think a startup could, could have done that same thing too, right?

    6. SPSpeaker29:20

       Yeah.

    7. SPSpeaker29:20

       I think we're the most like developer focused lab. I think we're the most like API focused lab too. So I think we, we want to make sure that we have the best platform for people to build stuff on because this thing is growing so incredibly quickly. Like we're not gonna be the fastest at figuring out all the ways that we need to empower Claude to do the work that connects Claude to the entire human business that's like human, human world is all designed for humans but like we need to get the models to be able to be productive members of the economy.

    8. SPSpeaker29:50

       Are, are there like ideas or areas you would love to see developers building in? Or like areas you don't... you, you think are like underappreciated right now?

    9. SPSpeaker29:58

       Yeah. Claude Code is like how do you get Claude to be a useful pair programmer kind of, um, or like junior engineer. You've got like, uh, SWE level two or three or something like that that you can work with, or like very spiky because also it can do the like weird disassembly stuff that like a super high level SWE would struggle with. Less good at knowing what type of work to do, needs kind of a lot of handholding, needs a lot of context from it. That's like one very particular subset of work that can be done. Uh, if you look at like all the stuff that happens in businesses...... besides that, (laughs) it's like a very tiny fraction of, like, all the work that's done in businesses that, like, a smart person who knows how to code and, like, use lots of tools, but doesn't have that much context yet, uh, would want to do. So I think, I think finding ways to coach Claude or, uh, Claude ... co- coach whatever model to, like, do useful tasks for businesses seems like there's just, like, a huge, huge space there.

    10. JFJared Friedman31:01

        So Tom, y- a big part of your job is, like, owning all the compute infrastructure that makes Anthropic work. Can you talk about, like, what, what is the compute infrastructure behind this giant thing now?

15. 31:11 – 32:46

    The Largest Infrastructure Buildout Ever

    1. SPSpeaker31:11

       One thing that's interesting to look at is just that humanity is on track for, like, the largest infrastructure build-out of all time now.

    2. JFJared Friedman31:18

       Is it going to be larger than the Apollo project? Larger than the Manhattan Project?

    3. SPSpeaker31:22

       It'll be bigger than both of them next year if it keeps on the current trajectory, which is, like, roughly 3X per year increase in spending on AGI compute, which is just bonkers.

    4. JFJared Friedman31:34

       (laughs)

    5. SPSpeaker31:34

       Yeah, like 3X per year is wild. I think it's going to keep up on the 3X per year trajectory. It's already locked in for that for, for next year, and then it's a little bit open for, for 2027, 2028.

    6. GTGarry Tan31:46

       I mean, anecdotally, internal to YC, uh, we can't get enough, you know, credits across all of the top frontier models-

    7. SPSpeaker31:53

       Yeah.

    8. GTGarry Tan31:54

       ... including Claude.

    9. SPSpeaker31:54

       Yep.

    10. GTGarry Tan31:54

        So you got to help us out on that.

    11. SPSpeaker31:56

        (laughs)

    12. SPSpeaker31:57

        (laughs) Yep, yep. Yeah.

    13. GTGarry Tan31:58

        Which is, I mean, everyone's bottlenecked.

    14. SPSpeaker32:00

        Yeah.

    15. GTGarry Tan32:00

        Literally every, you know, it's like, "Give me more intelligence. I can't have enough."

    16. SPSpeaker32:03

        Yeah, yeah. And I know you guys have been looking at more hardware startups also for, like, more accelerators. I think that we will see more accelerators coming online to 2027. That's a good, a good space also, like data center tech I think is a big one.

    17. JFJared Friedman32:15

        Where are the bottlenecks for you guys now? Is it, like, getting enough electricity, getting enough GPUs, getting construction permits?

    18. GTGarry Tan32:23

        Power. People are using jet engines to get power.

    19. SPSpeaker32:26

        Yeah, yeah.

    20. GTGarry Tan32:26

        That's nuts.

    21. SPSpeaker32:27

        Overall, for the build-out, I think power is going to be the biggest bottleneck, especially power in the US. Like, we want to build in the US. That's one of our biggest policy goals is to, like, get the US to, like, build more data centers-

    22. JFJared Friedman32:40

        Yeah.

    23. SPSpeaker32:40

        ... permit more data centers, make it easier to build.

    24. GTGarry Tan32:42

        Is the answer renewables or is it, uh, nuclear?

16. 32:46 – 34:38

    Multi-Chip Strategy

    1. SPSpeaker32:46

       I, I de- definitely I feel like yes. Yes, all, all, all of those things. (laughs) I wish, I wish that nuclear was easier to build. (laughs)

    2. JFJared Friedman32:52

       And Anthropic is the only major lab that uses not just one kind of GPU, but the GPUs from three different manufacturers. Can you talk about that and how, how, how that strategy has played out?

    3. SPSpeaker33:03

       Yeah, yeah. So we use, um, GPUs, TPUs, and Trainium. Downside of doing that is that we split our performance engineering teams across all of those platforms, which is a ton of extra work. The positive thing is it gives us the flexibility to both, one, like soak up that extra capacity, 'cause there, there just is more of those altogether than just one. And then two is we can use the, like, right chips for the right jobs, where some chips will be better for inference, some chips will be better for training. And we can match the, the right chips to the right jobs. So yeah, I think that, that's kind of the, the trade-off there.

    4. SPSpeaker33:36

       I guess one cool thing is just connecting the dot through your career and how all of this compounded, because you, you were the one engineer building that change of the architecture from TPUs to GPUs back at OpenAI that got GPT-3 to actually scale. And now you're in charge of that at a much, much bigger scale year, years later. I don't know if that kind of connected dots for you.

    5. SPSpeaker33:59

       The big move from TPUs to GPUs at OpenAI I think was partly driven just that PyTorch was a better software stack on top of them than TensorFlow on top of GPUs. And I think that that then unlocked fast iteration, where, like, if you have, like, a good reliable software stack, then you can experiment quickly, just like build a whole system that works. I think that that's a thing that we really strive for now at Anthropic too, is a challenge of having many more platforms is that it's harder to write all the good software. I think building the muscle of knowing how to build that software well so that all of the people who build on top of that low level can have a great experience with it is the most

17. 34:38 – 35:56

    Advice for the Next Generation

    1. SPSpeaker34:38

       important thing there.

    2. SPSpeaker34:38

       Do you have advice for, um, kind of like a younger Tom version of yourself who now you've seen and went through this crazy journey? If someone was you back in their 20s living today and they wanted to arrive and join the AI revolution, what would you say to them?

    3. SPSpeaker34:54

       And very specifically, something we see from a lot of, hear from a lot of college students at the moment is they, uh, they don't know what, like, if they should stay in college. Like, are there going to be jobs for them?

    4. SPSpeaker35:03

       Yeah.

    5. SPSpeaker35:04

       What, like, how is the world going to change, and what should they do?

    6. SPSpeaker35:07

       Taking more risks, I think, is, is wise. And then also trying to work on stuff where your friends would be really excited and impressed (laughs) if you did it, or a more idealized version of yourself would be really, like, proud of yourself if you succeeded at it, I think is, like, probably the thing that I would, I would try to (laughs) tell a younger version of myself. (laughs)

    7. GTGarry Tan35:27

       More intrinsic, less extrinsic. Like, don't chase these other credentials and getting the degree or what e- you know, working at Fang. Like, those are just-

    8. SPSpeaker35:35

       Yeah.

    9. GTGarry Tan35:35

       ... irrelevant-

    10. SPSpeaker35:36

        Yeah.

    11. GTGarry Tan35:36

        ... as of today.

    12. SPSpeaker35:37

        Yeah, exactly.

    13. GTGarry Tan35:38

        That's all we have time for today. We'll see you guys next time. (instrumental music)

Episode duration: 35:56