Stanford CS153 Frontier Systems | Scale, AGI, and the Future of Everything

1. SPSpeaker0:09

   Please join me in welcoming Sam Altman. [audience applauding] This class was designed as an inspiration from a, you know, from a set of different experiences, uh, while I was a student here. One of them was Terry Winograd's, uh, intro seminar, CS47N, Computers and the Open Society. Uh, but a second one that was a pretty formative experience, uh, for me and a lot of my friends and peers on campus at the time in 2014 was, uh, CS183, How to Start a Startup by Sam. Um, and so it's really cool to have you back. Uh, what's it like? How, how's it feeling for you to be back?

2. SASam Altman0:48

   I was thinking as I was walking in, if I had just a little more time, I would do, uh, an update to that class because I think everything about starting a startup has changed so much, and I have not seen anyone do a good version of how you're supposed to make a startup now. Uh, so I, I had that, like, just walking in here, I had that, like, ah, it'd be fun to do it again.

3. SPSpeaker1:06

   So, uh, timeline-wise, you know, you, you taught that in '14. I think OpenAI was founded in 2015. Is that right?

4. SASam Altman1:13

   '16, basically.

5. SPSpeaker1:14

   '16. Okay. So, so then you went, you know, it was like you were-- it, it, it felt to me from the, from an observer perspective that you had, like, come up with your working theory for how to do it right, and then you went and tried to implement it. Is that, is that a fair assessment, or is that not the case?

6. SASam Altman1:29

   O-OpenAI was, like, the strangest startup of the last maybe couple of decades in Silicon Valley. Uh, 'cause it started as a research lab. It was, it was really not a company at all.

7. SPSpeaker1:39

   Right.

8. SASam Altman1:40

   Um, and that-- the kind of normal course of, of startups is that you start a product company, and then it, like, grows for a while, and then growth slows down, and then you start a research lab, and you, like, bolt that on, and you try to figure out the next thing to do. And we were the opposite of that. We were a research lab first that later had to bolt on a startup.

9. SPSpeaker1:59

   Right.

10. SASam Altman2:00

    And, uh, I don't really recommend that. It's kind of an unusual thing. But that, that's not quite what I meant. What I meant is, like, we still followed the pre-AI rules of a startup because-

11. SPSpeaker2:11

    Mm-hmm

12. SASam Altman2:11

    ... we were trying to make AI. We didn't have it yet. But now, like, watching what the best startups do is so different than how startups worked even a couple of years ago, um, that I think someone, I'm probably not gonna do it, someone should do that class again.

13. SPSpeaker2:25

    And what would be the biggest updates you'd, you'd make based on new data?

14. SASam Altman2:29

    Um, you-- with, like, an affordable amount of spend on tokens, you can do what a hundred-person incredibly great engineering team would do as a startup, and that was just totally impossible. That was, like, not in the set of options for a startup, and now it is. So, so I think what you can take on, uh, the level of ambition you can have, the speed at which you can move, the amount of stuff you can do at once, uh, is just totally different.

15. SPSpeaker3:00

    And, um, does that change the shape of the problems you feel like you'd assign at the end of the class for people to attack, you know, at the end of that quarter if you were teaching it again?

16. SASam Altman3:11

    I don't think assigning problems to attack ever works because if you, like-- If I can think of a problem, if I can think of, like, a really great startup idea, uh, if it's, like, obvious enough to me, uh, then it's probably obvious to a lot of people. When we started OpenAI, we were, we were, like, the, uh, you know, one of maybe generously speaking, four AGI efforts in the world.

17. SPSpeaker3:32

    Right.

18. SASam Altman3:32

    And you wanna find something like that, and I'm sure that there exists something today that just wasn't possible at all pre, like, automated coding era, uh, that is totally not obvious that will be, you know, a multi-trillion-dollar market soon, uh, and that only four companies are working on right now. But I don't know what that is. It's much more likely you all know what that is than I know what that is. I just, you know, my brain has, like, taken over by OpenAI. Um, but, you know, the kind of idea someone can assign you to work on is probably not what you want.

19. SPSpeaker4:02

    Yep. Um, okay. So that, that's fair. Um, but I, I think it'd be helpful since this is a systems class to maybe, uh, reason about a particular problem that you had to reason through so that they can then apply the shape of the techniques used to break down from a systems perspective that problem to solutions to their own problem.

20. SASam Altman4:21

    Yeah.

21. SPSpeaker4:21

    Um, and a, and a concept that, uh, you had started to tease in the class, you know, back in 2014 and then, uh, clearly you've talked about publicly over the years is, um, scale, right? Scale is its own beast. It's, it's, it's, you know, quantity is its own quality. You know, what-what, uh, scale as a concept has been something it seems like you've, um, empirically investigated in all kinds of ways over the last 10 years. So, um, could you help un- help us first unpack, like, what you mean by scale now 10 years later? How would you deconstruct that as a systems design, uh, attribute to apply, whether it's a, as, as a tool? Um, can, can we start there?

22. SASam Altman4:59

    Yes. Uh, so I don't know why the following observation is true. I offer no theory that I find satisfying to explain it, and that makes me a little bit nervous to suggest you follow it, but I'm going to anyway because empirically it does seem to be true, which is all of the most interesting things I have observed in my career in watching other, uh, things happen, all of the most interesting ones, uh, have had something to do with emergent properties that scale or scale continuing to provide returns far beyond what the consensus thinks will work. And this obviously happens with, like, scaling laws for AI models, um, but this happens with, uh, you know, getting more smart people together to think about one problem. This ha- in a, in a research setting, um, this happens with, uh, companies and the sort of economy of scale you can get all the, in all these different ways. I really learned this at Y Combinator when, uh, it became clear to me that everybody was saying, "Oh, Y Combinator's gotten too big. It should shrink. We should fund less companies per batch." You know, the best times of Y Combinator were when it was, like, 10 companies per batch. And a lot of, like, very smart people were saying this. And, and it was, like, tempting 'cause it would've been, like, much less work, and the theory was that, you know, the best companies are always kinda obvious, and then you fund the rest, and it's not as helpful. Um, but a huge part of the magic of what made YC work were, uh, was the sort of the network effects inside of the batch, and that was an emergent property at scale that just hadn't been discovered before. No one had tried to fund startups at scale in the same way, and, and thus no one had ever happened upon this observation of when you do that, um, there's, there's something important that happens that just didn't exist at all at the one-tenth or one-one-hundredth of the scale. There's a bunch of other examples like this. Uh, I-- And I'll skip them in the interest of time, but I, I would say, again, I offer no explanation for why, but empirically speaking, when you find a time that you can push on-- you can push something to a scale people have not tried before, and it's already working in some interesting way at the smaller scale, more often than not, that seems to be a good idea. And it also seems to be something that most people don't do enough.

23. SPSpeaker7:29

    Mm.

24. SASam Altman7:30

    And I don't of- offer an explanation for this either, but, like, in, you know, when we were like, "We're really gonna scale AI models," um, all of the, like, geniuses in the field, most of them were, "Oh, this isn't really working. You know, that's ne- that's barely a scientific result. It's not interesting that it gets better at scale. You've already shown that. Why keep scaling it?" So I mentioned the YC example. Um, I've seen a lot of startup founders where they're like, "Well, you know, there might be something interesting that would happen if I scaled this up, but I, I'm a little worried about it for non-specific reasons." And again, looking back at, like, a huge data set of people that have scaled their companies in all these different ways, there's almost always interesting stuff there. So I think directionally that's, like, a interesting thing to push on and, and severely underexplored. Um, on the systems design part of that, uh, I think one reason people don't do it as much is stuff breaks, uh, at an accelerating rate and in an unpredictable way as you scale it. And if you are gonna really scale something, um, it's always, like, a little bit broken. There are always, like, very smart people who say why you shouldn't do this. You know, "Don't get too ambitious. Don't get too big. Let's try this smaller." And so breaking that down is a systems problem. I'll use the thing of when we were, like, scaling up AI models. There was, "Technically, can we do this at all? This seems crazy." Like, no one had ever thought about trying to do a run across 10,000 or 100,000 GPUs, and that was gonna require stacks of engineering talent. Um, there was the capital requirements and what it was going to take to do this, and, like, how is there ever gonna be a business? How can you think about taking this risk? Uh, there was the sort of, like, cultural stuff of researchers saying, "Well, if we're gonna get all this compute, why do we put it all into this one project where we're not gonna learn something? Why not divide it up among all these, all these projects?" And this also happens in kind of every area I've looked at, or almost every area for scale. And breaking it down into the sort of each difficult area or each reason not to do it and trying to address them one at a time-

25. SPSpeaker9:35

    Yeah

26. SASam Altman9:35

    ... that's been really important.

27. SPSpeaker9:36

    Um, I- I'm gonna push on that a little bit because there's very few people who've been able to sort of s- repeatedly scale new products and systems the way, uh, the OpenAI team has over the years. But it seems like one of the issues is there, there are all these prior conditioning, uh, sort of mental models and expectations humans have, and you said things break. And one of the things it seems often breaks that's har- the hardest to refactor is, is human, the human side of the, the system's design, right? Wherever there's human implementers or there's, uh, human participants in that. And so what have you learned about humans at scale, like organizing humans at scale to participate in a system that may not be, uh, like just a, a redo of some past system that they, they get naively on at, you know, a priority on first blush?

28. SASam Altman10:27

    Um, I think, like, clear, a clear goal, a clear plan to get there, uh, and, like, a clear answer to the way that you're gonna get there and kind of how you're gonna make decisions along the way, that's, that's very important. So, um, y- you know, if we go back to the example of when we decided to scale up models, there were a lot of people who were like, "Ah, this isn't really gonna work. It's gonna have these problems. It's also not, you know, we need a more diversified portfolio." But once we say, "No, we're gonna make a bet on scaling deep learning," like, that's our thing. If we're wrong, we'll fail, but we're gonna do that. Here's why we're gonna do that. Here's what we believe about what the state of the world could be like if we get there. Uh, that's very powerful. And then for whatever reason, um, we did not evolve to be good at thinking about exponentials. People have a hard time imagining that scaling laws are gonna continue exponentially, that revenue will grow exponentially, that an organization can take on exponential complexity. And in my experience, it takes a lot of time to really reason through first principles with people about why, why that can happen.

29. SPSpeaker11:39

    Can we take two examples, uh, to walk through that? The first being ChatGPT and the second being Codex. You know, both of these have transformed... Can, can everyone hear? I'm gonna try to project it. Yeah? Okay. Um, so let, let me put in a frame, and you can challenge both the assumption, and then we can hopefully reason through example of what happened. In the case of ChatGPT, you know, for a long time in scaling of models, a big mental block that- Seem to be prevalent in the space is wha-what are these things gonna be useful for? This is, you know, it's a research, uh, sort of solution, uh, solution chasing a problem, research first approach. It's not a product. Um, and then, you know, ChatGPT came out and it proved to the world that con- you know, ch- that chat experience was a killer app for general models, um, at scale in, for consumers. And then a couple of years later, you know, s- it's clear that coding has been the killer enterprise app. So wh- what, how would you compare and contrast the systems you guys used to discover those use cases, ship them, scale them, monetize them? Any, any salient learnings from those two systems?

30. SASam Altman12:43

    Yes. Um, so we had made GPT-3, and we needed to make money 'cause we wanted to go scale up to, you know, a billion and multi-billion dollar computers, and we had GPT-3, and it was kind of interesting. It was a cool demo, but we couldn't figure out a product to build around it. And we had been thinking, thinking. We just couldn't do it. We had tried a few things. They, they hadn't worked. Um, and so we knew the models were gonna get better, but we also wanted to, like, start a revenue engine sooner. And we said, "Well, since we can't figure out what product to build, we're just gonna put this into an API, and we're gonna hope that somebody else can figure out what product to build." And so we launched in, like, I don't know, somewhere in the summer of 2020, the GPT-3 API. And initially, it kinda got no traction at all, and then about a month later, randomly as far as we can tell, it went viral on Twitter. On the same day, uh, a few different developers kinda found, got it to do something cool, posted it, other people started trying. And, and then, like, a lot of people started trying the API. Um, but it was shockingly bad. If you go back and use GPT-3 or 3.5, um, you will be astonished at how bad the models were then, uh, relative to the amount of excitement they generated at the time. Uh, so people tried all of these things, and really, the only business that people got to work in a significant way with GPT-3 was copywriting. Um, and that was, like, not that great and not that exciting, and we were kind of like, you know, "Ah, it's just gonna have to wait for a better model." But although n- that was the only business that was working, developers had figured out how to, like, put in a prompt and get, and be able to chat with it. And we saw this a lot. Like, more people were using... They couldn't get the API to work for their business, but they were using their API key to just chat. And we said, "Well, we can build a good chatbot. People clearly want that." And we had a new model. We actually had GPT-4 done, but we had a new model we were ready to release in between called 3.5, and we had figured out a new kind of post-training where we could get the models to do, like, a good job with instruction following so it can make it easier to chat with. And we said, "Well, you know, the API is not working great." Maybe it was like a 10 or a $20 million run rate kinda business. But there is this thing that people love, uh, and under the YC principle of see what your users love and do that, we said, "Well, we'll build a chatbot around it." And we put that out, and we still didn't think it was gonna do that well. Uh, there was-- It was really meant as, like, a research demo, uh, to convince other people that they should build chat-like products and pay us for the API. But that went, like, crazy viral. And another thing I had learned from YC is when something really starts growing and it's not very good, you have, like, a guaranteed hit on your hands. And so we had, like, five days where the traffic would shoot up, fall off, and everybody would be like, "Well, that was just a hype cycle." But then the next day it would get to a higher peak, fall off again later in the day. People would say, "That's a hy-hype cycle." By the fourth or fifth day, I was like, "I know how this works. I know what's gonna happen." Like, we have the potential here-

Episode duration: 41:09