Wojciech Zaremba: OpenAI Codex, GPT-3, Robotics, and the Future of AI | Lex Fridman Podcast #215

Lex Fridman 0:00 The following is a conversation with Wojciech Zaremba, co-founder of OpenAI, which is one of the top organizations in the world doing artificial intelligence research and development. Wojciech is the head of language and co-generation teams, building and doing research on GitHub Copilot, OpenAI Codex, and GPT-3 and who knows, 4, 5, 6, and, and, n+1, and he also previously led OpenAI's robotic efforts. These are incredibly exciting projects to me that deeply challenge and expand our understanding of the structure and nature of intelligence. The 21st century, I think, may very well be remembered for a handful of revolutionary AI systems and their implementations. GPT, Codex, and applications of language models and transformers in general to the language and visual domains may very well be at the core of these AI systems. To support this podcast, please check out our sponsors. They're listed in the description. This is the Lex Fridman Podcast, and here is my conversation with Wojciech Zaremba.

Lex Fridman 1:18 You mentioned that Sam Altman asked about the Fermi paradox, and, uh, the people at OpenAI had really sophisticated, interesting answers, so that's when you knew this is the right team to be working with. So let me ask you about the Fermi paradox about aliens. Why have we not found overwhelming evidence for aliens visiting Earth?

Wojciech Zaremba 1:41 I don't have a conviction in the answer, but rather kind of probabilistic perspective on what might be a, let's say, possible answers. It's also interesting that the question itself even can touch on the, you know, your typical question of what's the meaning of life because like-

Lex Fridman 1:56 Yes.

Wojciech Zaremba 1:56 ... if you assume that like we don't see aliens because they destroy themselves, that kind of upweights the, um, focus on making sure that we won't destroy ourselves.

Lex Fridman 2:06 Yeah.

Wojciech Zaremba 2:06 Um, but at the moment, the place where I am actually with my belief, and these things also change over the time, is I think that, uh, we might be alone in the universe, which actually makes life more or let's say consciousness life more kind of valuable, and that means that we should more appreciate it.

Lex Fridman 2:26 Have we always been alone? So y- what's your intuition about our galaxy, our universe? Is it just sprinkled with graveyards of intelligent civilizations, or are we truly... Is, is life, intelligent life truly unique?

Wojciech Zaremba 2:40 At the moment, my belief that it, it is unique, but I would say I could also, uh, you know... There was like some footage, uh, released, uh, with UFO objects-

Lex Fridman 2:49 Mm-hmm.

Wojciech Zaremba 2:49 ... which makes me actually doubt my own belief.

Lex Fridman 2:51 Yes.

Wojciech Zaremba 2:52 Uh, yeah, I can tell you one crazy answer that I have heard.

Lex Fridman 2:56 Yes.

Wojciech Zaremba 2:56 Um, so, um, apparently when you look actually at the limits of computation, you can compute more if the temperature of the universe were to drop down. So, one of the things that aliens might want to do if they are truly optimizing to maximize amount of compute which, you know, maybe can lead to more let's say simulations or so, it's instead of wasting current entropy of the universe, because you know, we by living we are actually somewhat wasting entropy.

Lex Fridman 3:28 Mm-hmm.

Wojciech Zaremba 3:29 Then you can wait for the universe to cool down, such that you have more computation. Now that's kind of a funny answer, I'm not sure if I believe in it, but that would be one of the reasons why you don't see aliens. It's also possible, see some people say that maybe there is n- not that much point in actually going to other galaxies, uh, if you can go inwards. So, there is no limits of what could be an experience if we could, uh, you know, connect machines to our brains. Uh, while there are still some limits if we want to explore universe.

Lex Fridman 4:00 Yeah, there could be a lot of ways to go inwards too. Once you figure out some aspect of physics we haven't figured out yet, maybe you can travel to different dimensions. I mean, travel in three-dimensional space may not be the most fun kind of travel. There may be like just a huge amount of different ways to travel and doesn't require a spaceship going slowly in 3D space or spacetime.

Wojciech Zaremba 4:28 It, it also feels, you know, one of the problems is that speed of light is low and universe is vast.

Lex Fridman 4:34 Yeah.

Wojciech Zaremba 4:35 And, um, it seems that actually most likely if we want to travel very far, then, uh, then we would instead of actually sending spaceships with humans that weight a lot, we would send something similar to what Yuri Milner is working on. These are like a huge, uh, sail which is at first powered... Or there, there is a shot of laser from an air, and it-

Lex Fridman 4:58 Yeah.

Wojciech Zaremba 4:58 ... can propel it to quarter of speed of light.

Lex Fridman 5:01 Mm-hmm.

Wojciech Zaremba 5:01 And, uh, sail itself contains a few grams of equipment, and that might be the way to actually transport matter through universe. But then when you think what would it mean for humans, it means that w- we would need to actually put their 3D printer, and you know, 3D print a human on other planet. I don't know, play them YouTube or let's say, or like a pre- 3D print like huge human right away or maybe a womb or so. Um, yeah.

Lex Fridman 5:29 With our current techniques of archeology if, if, if a civilization was born and died, uh, long, long enough ago on Earth, we wouldn't be able to tell. And so, that makes me really sad, and so I think about Earth in that same way. How can we leave some remnants if we do destroy ourselves? How can we leave remnants for aliens in the future to discover? Like, here's some nice stuff we've done. Like Wikipedia and YouTube, do we have it like in a satellite orbiting Earth?... with a hard drive. Like, how, how do we say, how do we back up human civilization, uh, for the good parts? Or all of it is good parts, so that, uh, it can be preserved longer than our bodies can. That's, uh, uh, that's kind of a, it's a difficult question. It also requires the difficult acceptance of the fact that we may die, and if we die, we may die suddenly as a civilization.

Wojciech Zaremba 6:29 So, let's see. I think it kind of depends on the cataclysm. We have observed in other parts of universe that births of gamma rays, uh, these are, uh, high energy, uh, rays of light that actually can apparently kill entire galaxy. So, there might be actually nothing even to, nothing to protect us from it. Um, also when I'm looking actually at the past civilizations, so, it's like Aztecs or so, they disappear from the surface of the Earth, and one can ask, "Why is it the case?" And, um, the way I'm thinking about this, you know, that definitely they had some problem that they couldn't solve, and maybe there was a flood and all of a sudden they couldn't drink, uh, there was no potable water and they all died. And, um, I think that, uh, so far, the best solution to such, uh, problems is, I guess, technology. So, I mean, if they would know that you can just boil water and then drink it after-

Lex Fridman 7:31 Mm-hmm.

Wojciech Zaremba 7:32 ... then that would save their civilization. And even now when we look actually at the current pandemic, it seems that, uh, once again, actually science comes to rescue. And somehow science increases size of the action space, and, um, I think that's a good thing.

Lex Fridman 7:47 Yeah, but nature has a vastly larger action space.

Wojciech Zaremba 7:52 Yeah, but still it might be a good thing for us to keep on increasing action space. (laughs)

Lex Fridman 7:57 (laughs) Well, the, well, okay. Uh, looking at past civilizations, yes. But looking at the destruction of human civilization, perhaps expanding the action space will add, um, actions that are easily acted upon, easily executed, and as a result destroy us.

Wojciech Zaremba 8:19 So, let's see.

Wojciech Zaremba 8:20 I was wondering, um, why actually even, uh, we have negative impact on the, uh, globe. Because, you know, if you ask every single individual, they would like to have clean air, they would like healthy planet, but somehow it actually is not the case that a- as a collective, we are not going in this direction. I think that there exists very powerful system to describe what we value, that's capitalism. You know, science actually monetarize values, uh, to various activities. At the moment, the problem in the current system is that there's some things which we value, there is no cost assigned to it.

Lex Fridman 8:56 Mm-hmm.

Wojciech Zaremba 8:56 So, even though we value clean air or maybe we also, uh, value lack of distraction on, let's say, internet or so, at the moment, these quantities, you know, companies, corporations can pollute them, uh, for free. So, in some sense, I wish or like, and that's, I guess, purpose of politics to, to align the incentive systems. And we are kind of maybe even moving in z- this direction. The first issue is even to be able to measure the things that we value. Then we can actually assign the monetary value to, to them.

Lex Fridman 9:34 Yeah, and that's, so it's getting the data and also probably through technology enabling people to vote and to move money around in a way that is aligned with their values, and that's very much a technology question. So, like, having one president and congress and voting that happens every four years or something like that, that's a very outdated idea. There could be some technological improvements to that kind of idea.

Wojciech Zaremba 10:03 So, I'm thinking from time to time about these topics, but it also feels to me that it's, it's a little bit like, uh, it's hard for me to actually make correct predictions what is the appropriate thing to do. I extremely trust, uh, Sam Altman, our CEO o- on these topics. He, um, okay, I'm more on the side of being, I guess, naïve hippie that, uh, yeah. (laughs)

Lex Fridman 10:29 (laughs) That, that's your life philosophy. Um, well, like, I think self-doubt and, uh, I, I think hippie implies optimism. Those, those two things are pretty, pretty good way to operate.

Wojciech Zaremba 10:43 I mean, still it is, uh, hard for me to actually understand how the politics works, or like, uh, how this, like, uh, exactly how the things would play out. And Sam is, uh, really excellent with

Wojciech Zaremba 10:57 it.

Lex Fridman 10:57 What do you think is rarest in the universe? You said we might be alone. What's hardest to build is another engineering way to ask that. Life, intelligence or consciousness? So, like, you said that we might be alone. Which is the thing that's hardest to get to? Is it just the origin of life? Is it the origin of intelligence? Is it the origin of consciousness?

Wojciech Zaremba 11:22 So, um, let me at first explain you my kind of mental model what I think is needed for life to appear.

Lex Fridman 11:29 Mm-hmm.

Wojciech Zaremba 11:30 Um, so, I imagine that at some point there was this primordial, uh, soup of, uh, amino acids and maybe some proteins in the ocean, and, uh, you know, some proteins were turning into some other proteins through reaction. And, uh, you can almost think about this, uh, cycle of what, uh, turns into what as there is a graph essentially describing which substance turns into some other substance. And essentially life means that all the sudden in the graph has been created a cycle such that the same thing keeps on happening over and over again. That's what is needed for life to happen. And in some sense you can think almost that you have this gigantic graph and it needs, like, a sufficient number of edges for the cycle to appear.

Lex Fridman 12:15 Mm-hmm.

Wojciech Zaremba 12:16 Um-... then, um, from perspective of intelligence and consciousness, uh, my current intuition is that they might be quite intertwined. First of all, it might not be that it's like a binary thing, that you have intelligence or consciousness. It, it seems to be a, a, a more a c- c- continuous component. Let, let's see. If we look for instance on the event networks, uh, recognizing images, uh, people are able to show that the activations of n- these networks correlate, uh, very strongly, uh, with activations in visual cortex-

Lex Fridman 12:51 Mm-hmm.

Wojciech Zaremba 12:51 ... uh, of some monkeys. The same seems to be true about language models. Um, also if you, for instance, um, look, um, if you train agent in, um, 3D world, um, at first, you know, it, it, it, it barely recognizes what is going on. Over the time, it kind of recognizes foreground from background. Over the time, it kind of knows where there is a food, uh, and it just follows it. Um, over the time, it actually starts having a 3D perception. So, it is possible for instance to look inside of the head of an agent and ask what would it see if it looks to the right, and the crazy thing is, you know, initially when the agents are barely trained, the- these predictions are pretty bad. Over the time, they, they become better and better. You can still see that if you ask what happens when the head is turned by 360 degrees, for some time, they think that a different thing appears, and then at some stage, they understand actually that the same thing's supposed to appear-

Lex Fridman 13:52 Wow.

Wojciech Zaremba 13:52 ... so they get like a understanding of 3D structure. It's also, you know, very likely that they have inside some level of un- uh, of like a symbolic reasoning, like a, there are particular, these symbols for other agents. So, when you look at DOTA, uh, agents, they collaborate together and, uh, and, uh, you know, they, they, th- they have some anticipation of, uh, if, if they would win battle. They have some, some expectations with respect to o- other agents. I might be, you know, too much anthropomorphizing, um, the, the, the how the things look, l- look for me, but, um, the fact that they have a symbol for other agents, uh, makes me believe that, uh, at some stage as the, uh, you know, as they are optimizing for skills, they would have also symbol to describe themselves.

Lex Fridman 14:42 Mm-hmm.

Wojciech Zaremba 14:43 Uh, this is like a very useful symbol to have, and this particularly I would call it like a self-consciousness or self-awareness.

Lex Fridman 14:51 Mm-hmm.

Wojciech Zaremba 14:51 Uh, and, uh, still it might be different from the consciousness. So, I guess the, the way how I'm understanding the word consciousness, let's say the experience of drinking a coffee or let's say experience of being a bat.

Lex Fridman 15:04 Mm-hmm.

Wojciech Zaremba 15:04 Uh, that's the meaning of the word consciousness. It doesn't mean to be awake. Uh, yeah. It feels n- it might be also somewhat related to memory and recurrent connections. So, um, it's kind of like a... if you look at anesthetic drugs, they might be, uh, prev- like a... they, they essentially, they, they disturb, uh, uh, brain waves-

Lex Fridman 15:28 Mm-hmm.

Wojciech Zaremba 15:28 ... uh, such that, um, mm, maybe memory is not, not formed.

Lex Fridman 15:33 Mm-hmm. And so there's a lessening of consciousness when you do that.

Wojciech Zaremba 15:37 Correct.

Lex Fridman 15:37 And so that's one way to intuit what is consciousness. The- there's also kind of another element here. It could be that it's, you know, this kind of self-awareness module that you described.

Wojciech Zaremba 15:51 Mm-hmm.

Lex Fridman 15:52 Plus the actual subjective experience is a storytelling module that tells us a story about, uh, what we're experiencing.

Wojciech Zaremba 16:05 The crazy thing... So let's say, I mean, in meditation they teach people not to speak story inside of the head.

Lex Fridman 16:12 Mm-hmm.

Wojciech Zaremba 16:12 And there is also some fraction of population who doesn't have actually narrator. I know people who don't have a nar- nara- narrator and, you know, they have to use external people in order to, uh, kind of solve tasks that require internal nar- nar- narr- narrator. Uh, so it seems that it's possible to have the experience without the talk.

Lex Fridman 16:37 What are we talking about wh- when we talk about the internal narrator?

Wojciech Zaremba 16:40 So-

Lex Fridman 16:41 Is, is that the voice when you're like reading a book?

Wojciech Zaremba 16:42 Yeah, I thought that, that that's what you are referring to.

Lex Fridman 16:45 Well, I was referring more on the like not an actual voice. I meant like there is some kind of... like subjective experience feels like it's, it's fundamentally about storytelling to ourselves. It, it feels like (laughs) , like the feeling is a story that is much, uh, much simpler abstraction than the raw sensory information. So there feels like it's a very high-level abstraction that, uh, is useful for me to feel like entity in this world. M- mo- most useful aspect of it is that because I'm conscious, I think there's an intricate connection to me not want- wanting to die. So like, it's a useful hack to really prioritize not dying. Like those seem to be somehow connected, so I'm telling the story of like it's richly feels like something to be me and the fact that me exists in this world, I wanna preserve me, and so that makes it a useful agent hack.

Wojciech Zaremba 18:00 So I will just refer maybe to the first part as you said about the kind of story of describing who you are. Um, I was,

Wojciech Zaremba 18:10 uh, thinking about that even... so, you know, obviously I'm, I, I like thinking about consciousness. Uh, I like thinking about AI as well, and I'm trying to see analogies of these things in AI, what would it correspond to. So, uh-Um, you know, OpenAI trained, uh, uh, a model called GPT, uh, which, uh, can generate, uh, 3:00 PM using text on arbitrary topic.

Lex Fridman 18:40 Mm-hmm.

Wojciech Zaremba 18:41 And, um, um, and one way to control GPT is, uh, by putting into prefix at the beginning of the text some information, what would be the story about.

Lex Fridman 18:53 Mm-hmm.

Wojciech Zaremba 18:53 Uh, you can have even chat with, uh, um, you know, with GPT by saying that the chat is with Lex or Elon Musk or so.

Lex Fridman 19:01 Mm-hmm.

Wojciech Zaremba 19:01 And GPT would just pretend to be you or Elon Musk or so. And, um, uh, it almost feels that this, uh, story that we give ourselves to describe our life, it's almost like things that you put into context of GPT-

Lex Fridman 19:19 Yeah.

Wojciech Zaremba 19:19 ... to prime-

Lex Fridman 19:19 And it generates the ... And so but the, the context we provide the GPT is, uh, is multimodal.

Wojciech Zaremba 19:27 It's mul- So GPT itself is multimodal. GPT itself, uh, hasn't learned actually from experience of single human but from the experience of humanity. It's a chameleon. You can turn it into anything and in some sense by providing context, uh, it, it, it, you know, behaves as the thing that you wanted it to be.

Lex Fridman 19:47 Mm-hmm.

Wojciech Zaremba 19:47 It's interesting that, uh, you know, people have stories of who they are and, uh, as you said, these stories, they help them to operate in the world. Uh, but it's also, you know, interesting, because various people find it out through meditation or so that, uh, there might be some patterns that you have learned when you were a kid that actually are not serving you anymore, and you also might be thinking that that's who you are and that's actually just a story.

Lex Fridman 20:15 Mm-hmm. Yeah, so it's a useful hack but sometimes it gets us into trouble. It's a local optima.

Wojciech Zaremba 20:21 It's a local optima.

Lex Fridman 20:23 You wrote that Stephen Hawking ... You tweeted, "Stephen Hawking asked what breathes fire into equations, which meant what makes given mathematical equations realize the physics of a universe? Similarly, I wonder what breathes fire into computation. What makes given computation conscious?" Okay, so how do we engineer consciousness? How do you breathe fire and magic into the machine?

Wojciech Zaremba 20:52 So, um, it seems clear to me that not every con- uh, computation is conscious. I mean, you can, let's say, just keep on multiplying one matrix over and over again and it might be gigantic matrix, you can put a lot of computation, I don't think it would be conscious. So in some sense the question is, uh, what are the computations which could be conscious? Uh, I mean, so, so, uh, one assumption is that it has to do purely with computation that you can abstract the way matter. Uh, other possibility is there ... it's very important what's the realization of computation that it has to do with some, uh, uh, force fields or so and they bring consciousness.

Lex Fridman 21:30 Mm-hmm.

Wojciech Zaremba 21:30 At the moment my intuition is that it can be fully abstracted away. So in case of compu- computation you can ask yourself, what are the mathematical objects or so that could bring such a properties? So for instance, if we think about the models, uh, AI models, the ... what they truly try to do, uh, or like models like GPT is, uh, uh, you know, they try to predict next word or so.

Lex Fridman 22:00 Mm-hmm.

Wojciech Zaremba 22:00 And this turns out to be equivalent to, uh, compressing, uh, text. Um, and, uh, because in some sense compression means that, uh, you learn the model of reality and you have just to, uh, remember what are your mistakes, the better you are in predicting the ... And, and, and in some sense when we look at our experience also when you look, for instance, at the car driving you know in which direction it will go, you are good, like, uh, in prediction and, um, you know, it might be the case that the consciousness is intertwined with, uh, compression. It might be also the case that self-consciousness, uh, has to do with compressor trying to compress itself.

Lex Fridman 22:42 Mm-hmm.

Wojciech Zaremba 22:42 So, um, okay, I was just wondering what are the objects in, you know, mathematics or computer science which are mysterious-

Lex Fridman 22:51 Mm-hmm.

Wojciech Zaremba 22:51 ... that could, uh, that, that, that could have to do with consciousness and th- uh, then I thought, um, you know, you, you see in, uh, mathematics there is something called Gödel theorem.

Lex Fridman 23:01 Mm-hmm.

Wojciech Zaremba 23:02 Uh, which means, okay, you have ... if you have sufficiently complicated mathematical system it is possible to point the mathematical system back on itself.

Lex Fridman 23:11 Mm-hmm.

Wojciech Zaremba 23:11 In computer science there is, uh, something called halting problem.

Lex Fridman 23:14 Mm-hmm.

Wojciech Zaremba 23:14 It's, it's somewhat similar construction. So I thought that, you know, if we believe that, uh, that, uh, that under assumption that consciousness has to do with, uh, with compression, uh, then you could imagine that the... that the ... As they keep on compressing things then at some point it actually makes sense for the compressor to compress itself.

Lex Fridman 23:37 Meta-compression.

Wojciech Zaremba 23:38 Yeah.

Lex Fridman 23:39 Consciousness is meta-compression. (laughs)

Wojciech Zaremba 23:41 That's, uh, uh, that's an i- an, an, an idea.

Lex Fridman 23:45 (laughs)

Wojciech Zaremba 23:45 And in some sense, you know, the crazy-

Lex Fridman 23:46 I love it.

Wojciech Zaremba 23:47 Th- um, thank you.

Lex Fridman 23:48 So, uh, but do you think if we think of a Turing machine, a universal Turing machine, can that achieve consciousness? So is there something beyond our traditional definition of computation that's required?

Wojciech Zaremba 24:03 S- so it's a specific computation and I said this computation has to do with compression.

Lex Fridman 24:07 Mm-hmm.

Wojciech Zaremba 24:08 And, uh, the compression itself ... Uh, maybe other way of putting it is like you're internally creating the model of reality in order ... Like a, like a you try inside to simplify reality in order to predict what gonna happen.

Lex Fridman 24:21 Mm-hmm.

Wojciech Zaremba 24:22 And, uh, that also feels somewhat similar to how I think actually about my own conscious experience. So clearly I don't have access to reality. The only access to reality is through, you know, cable going to my brain and my brain is creating a simulation of reality-

Lex Fridman 24:37 Mm-hmm.

Wojciech Zaremba 24:38 ... and I have access to the simulation of reality.

Lex Fridman 24:40 Are you by any chance, uh, aware of, uh, the Hutter Prize? Marcus Hutter? He, uh, he made this prize for compression of Wikipedia pages and, uh, there's a few qualities to it. One I think has to be perfect compression, which makes ... I think that little quirk makes it much less, um, applicable to the general task of intelligence 'cause, uh, it feels like intelligence is always going to be messy. Uh, i- like, perfect compression is ... feels like it's not the right goal but it's nevertheless a very interesting goal. So for him, intelligence equals compression, and so the smaller you make the file given a large Wikipedia page, the more intelligent the system has to be.

Wojciech Zaremba 25:31 Yeah, that makes sense. So you can make perfect compression if you store errors.

Lex Fridman 25:34 Mm-hmm.

Wojciech Zaremba 25:35 And I think that actually what he meant is you have algorithm plus errors.

Lex Fridman 25:38 Yeah.

Wojciech Zaremba 25:38 Uh, by the way, Hutter h- Hutter is a ... He was PA- uh, uh, PhD advisor of Yann LeCun who is a DeepMind, uh, uh, DeepMind co-founder.

Lex Fridman 25:48 Yeah. Yeah, so there's an interesting, uh, and now he's at DeepMind. There's an interesting, uh, network of people. He's one of the people that I think seriously took on the task of, what would an AGI system look like? I think for the longest time, the, the question of AGI was not taken seriously or rather rigorously and he, he did just that. Like, mathematically speaking, what would the model look like? If you remove the constraints of it having to be, uh, um ... having to have a reasonable amount of memory, reasonable bo- amount of, uh, running time complexity, uh, c- computation time, what would it look like? And essentially it's, it's a half math, half philosophical discussion of, uh, how would a, like, a reinforcement learning type of, uh, framework look like for an AGI.

Wojciech Zaremba 26:47 Yeah, so he developed a framework even to describe what's optimal with respect to reinforcement learning, like, there is a theoretical framework which is, as you said, under assumption there is infinite amount of memory and compute.

Lex Fridman 26:59 Mm-hmm.

Wojciech Zaremba 26:59 Um, there was actually one person before. His name is Solomonoff. Hutter extended, uh, Solomonoff work to reinforcement learning but there exists a, a theoretical algorithm which is optimal algorithm to build intelligence and I can actually explain you the algorithm.

Lex Fridman 27:17 Yes.

Wojciech Zaremba 27:18 (laughs)

Lex Fridman 27:18 Let's go. Let's go. Let's go.

Wojciech Zaremba 27:20 So the task itself you can-

Lex Fridman 27:22 C- Can I just pause how absurd it is for a brain in a skull trying to explain the algorithm for intelligence? Just go ahead.

Wojciech Zaremba 27:31 It is pretty crazy. It is pretty crazy that, you know, the brain itself is actually so small and it can't f-

Lex Fridman 27:36 (laughs)

Wojciech Zaremba 27:36 ... ponder, uh ...

Lex Fridman 27:39 How to design algorithms that optimally solve the problem of intelligence? Okay. All right. So what's the algorithm?

Wojciech Zaremba 27:45 So let's see. So first of all, the task itself is, uh, described as, uh, you have infinite sequence of zeros and ones, okay? You read, uh, n bits and you are about to predict n plus one bit.

Lex Fridman 27:58 Mm-hmm.

Wojciech Zaremba 27:59 So that's the task and you could imagine that every task could be casted as such a task. So if for instance you have images and labels, you can just turn every image into sequence of zeros and ones, then label ... You concatenate labels and you ... and that, that's actually the, the ... and you could, you could start by having training data first, and then afterwards you have test data. So theoretically any problem could be casted as a problem of predicting zeros and ones on this, uh, infinite type.

Lex Fridman 28:29 Mm-hmm.

Wojciech Zaremba 28:30 So, um, so let's say you read already n bits and you want to predict n plus one bit. Then I will ask you to write every possible program that generates these n bits. Okay? So, um, and you can have ... you, you, you choose programming language. It can be in Python or C++. And the difference between programming languages, uh, might be there is a difference by constant. Asymptotically, your predictions will be equivalent.

Lex Fridman 28:58 Mm-hmm.

Wojciech Zaremba 29:00 So you, you read n bits, you enumerate all the programs that produce these n, n bits in their output.

Lex Fridman 29:07 Mm-hmm.

Wojciech Zaremba 29:07 And then in order to predict n plus one bit, you actually weight the programs according to their length, uh, there is-

Lex Fridman 29:17 Hm.

Wojciech Zaremba 29:17 ... like some specific formula how you weight them and then the n plus, uh, one bit prediction is the prediction, uh, from each of this program according to that weight.

Lex Fridman 29:28 Like statistically you-

Wojciech Zaremba 29:29 Statistically, yeah.

Lex Fridman 29:30 ... pick ... So the smaller the program, the more likely you, you are to pick the, its output. So, uh, that's, that algorithm is grounded in the hope or the intuition that the simple answer is the right one.

Wojciech Zaremba 29:46 It is a formalization of it.

Lex Fridman 29:47 Yeah.

Wojciech Zaremba 29:48 Um, it also m- means, like, if you would ask the question, "After how many years would, you know, the sun explode?"

Lex Fridman 29:57 Mm-hmm.

Wojciech Zaremba 29:58 Uh, you can say, "Hmm, it's more likely the answer is two to, to some power," because it's a shorter program-

Lex Fridman 30:04 Yeah.

Wojciech Zaremba 30:04 ... um, than other. (laughs)

Lex Fridman 30:06 (laughs) Well I, I don't have a good intuition about, uh, how different the space of short programs are from the space of large programs. Like, what is the universe where short programs, uh, like, run things? (laughs) Uh ...

Wojciech Zaremba 30:22 So as I said, the things have to agree with n bits so even if you have sh- y- you, you need to start like a ... If, if you have very short program and they're, like, uh, still some ... has ... I- if it's not perfect with prediction of n bits, you have to store errors, what are the errors, and that gives you the full program that agrees on n bits.

Lex Fridman 30:41 Oh, so you don't agree perfectly with the, the n bits and you store the errors.

Wojciech Zaremba 30:44 That's like a longer bl- a longer program, slightly longer program-... because it contains these extra bits of errors.

Lex Fridman 30:50 That's fascinating. What's, what's your intuition about d- the programs that are able to do cool stuff, like intelligence and consciousness? Are they, uh, perfectly... Like, is it, is it, uh... Is there if/then statements in them? So, like, is there a lot of exceptions that they're storing?

Wojciech Zaremba 31:11 So, um, you could imagine if there would be tremendous amount of if statements-

Lex Fridman 31:16 Yeah.

Wojciech Zaremba 31:16 ... then they wouldn't be that short. In case of neural networks, you could imagine that, um, what happens is, uh, they, uh... when you start with an uninitialized neural network, uh, it stores internally many possibilities how the, uh, how the problem can be solved, and SGD is kind of magnifying some, uh, some, uh, some, uh, paths which are slightly similar to the correct answer, so it's kind of magnifying correct programs and in some sense, SGD is the search algorithm in the program space, and the program space is represented by, uh, you know, kind of the wiring inside of the neural network, and there's like an insane number of ways how the features can be computed.

Lex Fridman 32:02 Let me ask you the high-level basic question that's not so basic. What is deep learning?

Wojciech Zaremba 32:09 Hmm.

Lex Fridman 32:10 Is there a way you'd like to think of it that is different than like a generic textbook definition?

Wojciech Zaremba 32:16 The thing that I hinted just a second ago is maybe the, uh, closest to how I'm thinking these days about, um, deep learning. So, uh, you know, the statement is, uh, neural networks can represent some programs. Uh, it seems that various modules that we are actually adding up to- or like a... You know, we, we want networks to be deep because we, we want multiple steps of the computation.

Lex Fridman 32:42 Mm-hmm.

Wojciech Zaremba 32:45 And, uh, uh, and deep learning provides the way to represent space of programs which is searchable, and it's searchable with, uh, stochastic gradient descent. So, we have an algorithm to search over a humongous number of programs and gradient descent kind of bubbles up the things that are, uh, tend to give correct answers.

Lex Fridman 33:04 So, a neural network with a, with fixed weights that's optimized. Do you think of that as a single program?

Wojciech Zaremba 33:14 Um, so there is a, uh, work by Christopher Olah where he, uh... So, he works on interpretability of neural networks, and he was able to, uh, to identify inside of the neural network, for instance, a detector of a wheel for a car or the detector of a mask for a car-

Lex Fridman 33:33 Mm-hmm.

Wojciech Zaremba 33:34 ... and then he was able to separate them out and assemble them, uh, together using a simple program, uh, for the detector, for a car detector.

Lex Fridman 33:43 That's like a... I- if you think of traditionally defined programs, that's like a function within a program that this particular neural network was able to find, and you can tear that out just like you can copy and paste from Stack Overflow. That... So, uh, a- any program is a composition of smaller programs?

Wojciech Zaremba 34:03 Yeah. I mean, the nice thing about the neural networks is that it allows the things to be more fuzzy than in case of programs. Uh, in case of programs, you have this like a branching this way or that way, and the neural networks, they, they have an easier way to, to be somewhere in between or to share things.

Lex Fridman 34:22 What to you is the most beautiful or surprising idea in deep learning-

Wojciech Zaremba 34:26 Mm.

Lex Fridman 34:26 ... and the utilization of these neural networks? Which, by the way, for people who are not familiar, neural networks is a bunch of, um... What would you say? It's inspired by the human brain. There's neurons. There's connection between those neurons. There's inputs and there's outputs, and there's millions or billions of those neurons, and the learning happens, uh, by adjusting the weights on the edges that connect these neurons.

Wojciech Zaremba 34:54 Thank you for giving definition. (laughs) I'm supposed to do it, but I guess you have enough empathy to-

Lex Fridman 34:59 No, you-

Wojciech Zaremba 34:59 ... listeners to actually know that the, that that might be useful.

Lex Fridman 35:02 No, that's like... So I'm asking Plato of like, "What is the meaning of life?" He's not gonna answer str- you're, you're being philosophical and deep and quite profound talking about the space of programs, which is, which is very interesting, but also (laughs) for people who's just not familiar what the hell we're talking about when we talk about deep learning. Anyway, sorry. What is the most beautiful or surprising idea to you in, in, um, in all the time you've worked at deep learning? And you worked on a lot of fascinating projects, applications of neural networks. It doesn't have to be big and profound. It could be a cool trick.

Wojciech Zaremba 35:38 Yeah. I mean, I'm thinking about the trick, but like, uh, it's still a- amusing to me that it works at all.

Lex Fridman 35:44 Yeah.

Wojciech Zaremba 35:44 That, let's say that the extremely simple algorithm stochastic gradient descent, which is something that I would be able, you know, to derive on the piece of paper to high school student, uh, when put at the ins- at the scale of, you know, thousands of machines actually, uh, can create the behaviors we, which we call kind of human-like behaviors.

Lex Fridman 36:08 So, in general any application of stochastic gradient descent to neural networks is, is amazing to you? So, the... Or is there a particular application in natural language, reinforcement learning? Uh, and also, would you attribute that success to... is it just scale? What profound insight can we take from the fact that (laughs) the thing works for gigantic, uh, sets of variables?

Wojciech Zaremba 36:40 I mean, the interesting thing is these algorithms, they were, uh, invented, uh, decades ago and, uh, people actually, uh, gave up on the idea.

Lex Fridman 36:51 Yeah.

Wojciech Zaremba 36:51 And, um...... you know, back then, they thought that we need profoundly different algorithms and they spent a lot of cycles on very different algorithms. And I believe that the, you know, we have seen that various al- uh, various innovations, let's say, like transformer or, uh, or dropout or so they can, uh, you know, vastly help. But it's also remarkable to me that this algorithm from '60s or so, uh, are, I, I mean, you can even say that the gradient descent was invented by Leibniz in, I guess, 18th century or so, that actually is the core of learning. In the past, people are, uh, it's almost like a, out of the maybe an ego, people are saying that it cannot be the case that such a simple algorithm-

Lex Fridman 37:39 Mm-hmm.

Wojciech Zaremba 37:39 ... is the, you know, uh, could solve complicated problems. So they were, uh, in search for the other algorithms. And as I'm saying, like, I believe that actually we are in the game where there is, there are actually, frankly, three levers. There is compute, there are algorithms, and there is data. And, uh, if we want to build intelligent systems, we have to pull, uh, all three levers and they are actually multiplicative. Um, it's also interesting, so you ask, is it only compute? Uh, people internally, they did the studies to determine how much gains they were coming from different levers. And so far, we have seen that more gains came from compute than algorithms. But also we are in the world that in case of compute, there is a kind of, you know, exponential increase in funding and, at some point, it's impossible to, uh, invest more. It's impossible to, you know, invest $10 trillion 'cause we are speaking about the, let's say, all taxes in US, uh-

Lex Fridman 38:38 But you're talking about money, there could be innovation in the compute.

Wojciech Zaremba 38:43 That's, that's true as well. Uh, s- so, I mean, there are, like, a few pieces. So one piece is human brain is an incredible supercomputer.

Lex Fridman 38:50 Mm-hmm.

Wojciech Zaremba 38:51 And, uh, the, like, uh, it, i- i- it has 100 trillion parameters.

Lex Fridman 38:59 Mm-hmm.

Wojciech Zaremba 39:00 Or, like, uh, if you try to count various quantities in the brain, there are, like, neuron synapses, there are small number of neurons, there is a lot of synapses.

Lex Fridman 39:08 Yeah.

Wojciech Zaremba 39:08 Uh, it's unclear even how to map, uh, synapses to, uh, to parameters of neural networks.

Lex Fridman 39:15 Mm-hmm.

Wojciech Zaremba 39:16 But it's clear that there are many more.

Lex Fridman 39:18 Yeah.

Wojciech Zaremba 39:19 Uh, so it might be the case that our networks are still somewhat small.

Lex Fridman 39:23 Mm-hmm.

Wojciech Zaremba 39:24 Uh, it also might be the case that they are more efficient than brain or less efficient by some, uh, by some huge factor. Uh, I also believe that there will be, like, a, you know, at the moment, we are at the stage that the, these neural networks, they require 1,000X or, or, like, a huge factor of more data than humans do. And it will be a matter of, uh, um, th- there will be algorithms that vastly decrease sample complexity, I believe so.

Lex Fridman 39:51 Mm-hmm.

Wojciech Zaremba 39:51 But the place where we are heading today is data domains, which contains million X more, uh, data. And even though computers might be 1,000 times slower than humans in learning, that's not a problem.

Lex Fridman 40:05 Mm-hmm.

Wojciech Zaremba 40:05 Like, uh, for instance, uh, I believe that, uh, it should be possible to create superhuman therapist-

Lex Fridman 40:13 Mm-hmm.

Wojciech Zaremba 40:13 ... uh, by, uh, and, and, uh, there are, like, uh, even simple steps of, of, of doing good, of, of doing it. And, you know, the, the core reason is there is just machine will be able to read way more transcripts of therapist and then it should be able to speak simultaneously with many more people-

Lex Fridman 40:31 Yeah.

Wojciech Zaremba 40:31 ... and it should be possible to optimize it, uh, all in parallel. And, uh-

Lex Fridman 40:36 Well, there's, now you're touching on something I deeply care about and think is way harder than we imagine. Um, what's the goal of a therapist?

Wojciech Zaremba 40:45 What's the goal of therapist?

Lex Fridman 40:47 Mm. So that, okay, so one goal, now this is terrifying to me, but there's a lot of people that, uh, contemplate suicide, suffer from depression, uh, and they could significantly be helped with therapy. And the idea that an AI algorithm might be in charge of that, it's like a life-and-death task. (laughs) It's, uh, the stakes are high. So one goal for a, a therapist, whether human or AI, is to prevent suicide ideation, to prevent suicide. How do you achieve that?

Wojciech Zaremba 41:25 So let's see. So to be clear, I, I don't think that the current models are good enough for such-

Lex Fridman 41:33 Yeah.

Wojciech Zaremba 41:33 ... a task because it requires insane amount of understanding, empathy, and the models are far from this place. But it's-

Lex Fridman 41:40 But do you think that understanding empathy, that signal is in the data?

Wojciech Zaremba 41:45 Um, I think there is some signal in the data, yes. I mean, there are plenty of transcripts of conversations, and it is possible to, uh, it is possible from it to understand personalities. It is possible from it to understand, uh, if conversation is, uh, friendly, uh, amicable, uh, uh, antagonistic. It is, I believe that the, you know, given the fact that the models that we train now, they can, uh, they, they can have ... They are chameleons that they can have any personality-

Lex Fridman 42:18 Mm-hmm.

Wojciech Zaremba 42:18 ... they might turn out to be better in understanding, uh, personality of other people than anyone else. And they-

Lex Fridman 42:24 Be empathetic. (laughs)

Wojciech Zaremba 42:25 To be empathetic.

Lex Fridman 42:26 Yeah. Interesting. Uh, but I wonder if there's some level of, um, multiple modalities required to be able to, um, be empathetic of the human experience, whether language is not enough to understand death, to understand fear, to understand, uh, childhood trauma, to understand, uh, wit and humor required when you're dancing with a person who might be depressed or suffering.... both humor and hope and love and all those kinds of things. So, there's another underlying question which is s- self-supervised ver- versus supervised. (smacks lips) So, can you get that from the data by just reading a huge number of transcripts?

Wojciech Zaremba 43:16 I actually ... so I think that reading huge number of transcripts is step one. It's like the same way as you cannot learn to dance if- just from YouTube, by watching it. You have to actually try it out yourself.

Lex Fridman 43:28 Yeah.

Wojciech Zaremba 43:28 Uh, so I think that here does a similar situation. I also wouldn't deploy the system in the high-stakes situations right away-

Lex Fridman 43:36 Yeah.

Wojciech Zaremba 43:36 ... but kind of see gradually where, uh, uh, uh, it goes. And, uh, obviously, initially, uh, it would have to go hand w- uh, uh, hand in hand with humans. But, uh, at the moment, we are in this situation that actually there is many more people who actually would like to have a therapy or s- or speak with, with someone than there are therapists out there. Like, um, you know, I was s- so fundamentally, I was thinking what are the things that, uh, can vastly increase people wellbeing.

Lex Fridman 44:08 Mm-hmm.

Wojciech Zaremba 44:08 Therapy is one of them. I think meditation is other one. I guess, maybe human connection is a third one. And I guess, pharmacologically, it's also possible, maybe direct brain stimulation or something like that. But these are pretty much options out there. Then let's say, the way I'm thinking about the AGI endeavor is, by default, that's an endeavor to, uh, increase amount of wealth. And I believe that we can vastly increase amount of wealth, uh, for everyone. And simultaneously ... so I mean, these are like, uh, two endeavors that make sense to me.

Lex Fridman 44:40 Mm-hmm.

Wojciech Zaremba 44:40 One is like essentially increase amount of wealth, and second one is, uh, increase overall human wellbeing.

Lex Fridman 44:47 And those are coupled together.

Wojciech Zaremba 44:48 And they, they can ... like I would say these are different topics. One can help another.

Lex Fridman 44:54 And, uh, you know, therapist is, is a funny word 'cause I see friendship and love as therapy. I mean, so therapy is broadly defined as just friendship, as a friend. So like therapist is, has a very kinda clinical sense to it. But what is human connection? You're like, uh, (laughs) not to get all Camus and Dostoevsky on you, but, you know, life is suffering and we draw ... we seek connection with other humans as we, uh, desperately try to make sense of this world in the deep overwhelming loneliness that we feel inside.

Wojciech Zaremba 45:34 So I think connection has to do with understanding. And I think that almost like a lack of understanding causes suffering. If you speak with someone and you ... you feel ignored, that actually causes pain. If you are feeling deeply, uh, understood, that actually c- ... that they, they might not even tell you what to do in life-

Lex Fridman 45:53 Yeah.

Wojciech Zaremba 45:54 ... but like a pure understanding.

Lex Fridman 45:56 Or just being heard. Understanding is a kinda ... it's a lot, you know. Just being heard, feel like you're being heard. Like, somehow that's, uh, alleviation temporarily of the loneliness, that if somebody f- ... knows you're here, with their body language, with the way they are, with the way they look at you, with the way they talk, you feel less alone for a brief moment.

Wojciech Zaremba 46:23 Yeah. Ver- very much agree. So I thought in the past about, uh, somewhat similar question to yours which is, what is love? Uh, rather what is connection.

Lex Fridman 46:33 Yes.

Wojciech Zaremba 46:33 And, um, and obviously I think about these things from AI perspective, what would it mean. Um, so I said that the, you know, intelligence has to do with some compression which is more or less like you can say almost understanding of what is going around. It seems to me that, uh, other aspect is there seem to be reward functions and you can have a, a, you know, reward for, uh, food, for maybe human connection, for, uh, let's say warmth, uh, sex, and so on. And, um, and it turns out that, uh, various people might be optimizing slightly different, uh, reward functions. They essentially might care about different things. And, um, uh, in case of, uh, love, at least the love between two people, you can say that the, um, you know, boundary between people dissolves to such extent that, uh, they end up optimizing each other reward functions.

Lex Fridman 47:30 Mm-hmm.

Wojciech Zaremba 47:31 Uh ... (laughs)

Lex Fridman 47:33 (laughs) Yeah. Oh, th- that's interesting. Um, celebrate the success of each other.

Wojciech Zaremba 47:39 Yeah. Or in some sense, I would say love means, uh, helping others to optimize their, uh, reward functions, not your reward functions, not the things that you think are important, but the things that the person cares about, you'll try to help them to, uh, optimize it.

Lex Fridman 47:55 So love is, uh, if you think of two reward functions, you just ... it's like addition.

Wojciech Zaremba 48:00 Yeah.

Lex Fridman 48:00 You combine them together.

Wojciech Zaremba 48:01 Yeah. Pretty much.

Lex Fridman 48:02 Maybe like with a weight and depends like the dynamic of the relationship.

Wojciech Zaremba 48:06 Yeah. I mean, you could imagine that if you are fully, uh, optimizing someone's reward function without yours, then-

Lex Fridman 48:11 Yeah.

Wojciech Zaremba 48:11 ... then maybe you are creating codependency or something like that.

Lex Fridman 48:14 Yeah. (laughs)

Wojciech Zaremba 48:14 The ... I, I'm not sure what's the appropriate weight. But the interesting thing is I even, I even think that the, uh, individual person, uh, uh, we ourselves we are actually less of a unified insight. So for instance, if you look at, at the donut, on the one level, you might think, "Oh, this like, uh, looks tasty. I would like to eat it." On other level, you might tell yourself, "I shouldn't be doing it because, uh, I want to gain muscles." So a- and, you know, you might do it regardless, kind of against yourself. So it seems that even within ourselves, they're almost like a kind of intertwined personas.

Lex Fridman 48:51 Mm-hmm.

Wojciech Zaremba 48:51 And, um, I believe that the self-love means that, uh, the love between all these personas, which also means being able to love...... love, uh, yourself when you are angry or stressed or so.

Lex Fridman 49:06 Combining all those reward functions of the different selves you have.

Wojciech Zaremba 49:09 Yeah, and accepting that they are there. Like, you know, often people, they have a negative self-talk or they say, "I don't like when I'm angry."

Lex Fridman 49:17 Yeah.

Wojciech Zaremba 49:17 And like, try to imagine, try to imagine if there would be like a small baby, Alex, like five years old, and-

Lex Fridman 49:26 Who's angry.

Wojciech Zaremba 49:27 ... angry-

Lex Fridman 49:28 (laughs)

Wojciech Zaremba 49:28 ... and then, you're like, uh, "You shouldn't be angry. Like, stop being angry."

Lex Fridman 49:31 Yeah.

Wojciech Zaremba 49:32 But, like, instead, actually you want, uh, Alex to come over, give him a hug, and he's like, gonna say, "It's fine. Like, uh, it's, you can be angry as long as you want."

Lex Fridman 49:40 Yeah.

Wojciech Zaremba 49:40 And then he would stop.

Lex Fridman 49:42 Well, or m- or, or maybe not.

Wojciech Zaremba 49:44 Or maybe not, but you cannot expect it even.

Lex Fridman 49:46 Yeah.

Lex Fridman 49:47 But still that doesn't explain the why of love. Like, why is love part of the human condition? Why is it useful to combine the reward functions? It seems like that doesn't... I mean, I don't think reinforcement learning frameworks can give us answers to why. Even e- even the hotter framework has a objective function that's static.

Wojciech Zaremba 50:09 So, we came to existence as a consequence of evolutionary process.

Lex Fridman 50:14 Mm-hmm.

Wojciech Zaremba 50:14 And, in some sense, the purpose of evolution is survival.

Lex Fridman 50:17 Yeah.

Wojciech Zaremba 50:17 And then the, this complicated optimization objective baked into us, let's say, compression, which might help us, uh, operate in the real world and be baked into us by these reward functions.

Lex Fridman 50:30 Yeah.

Wojciech Zaremba 50:31 Uh, then, to be clear, at the moment, we are operating in the regime which is somewhat out of distribution where the even evolution optimized us.

Lex Fridman 50:38 It's almost like love is a consequence of, uh, cooperation that we've discovered is useful.

Wojciech Zaremba 50:44 Correct. In some way, it's even the case, if you-

Lex Fridman 50:47 I just love the idea that love is, like, the out of distribution... (laughs)

Wojciech Zaremba 50:51 Oh, it's not out of distribution. It's like Assi said, it evolved for co- cooperation.

Lex Fridman 50:55 Yes.

Wojciech Zaremba 50:56 And I believe that the co-op- like, uh, in some sense, cooperation ends up helping each of us individually. So, it makes sense evolutionary. And there is a, in some sense... And, you know, love means there is this dissolution of boundaries that you have a shared reward function.

Lex Fridman 51:09 Mm-hmm.

Wojciech Zaremba 51:10 And we evolved to actually identify ourselves with larger groups. So, we, we can identify ourselves, you know, with our family. We can identify ourselves with a country to such extent that people are willing to give away their life for a country.

Lex Fridman 51:24 Mm-hmm.

Wojciech Zaremba 51:25 Uh, so there is... We are wired actually even, uh, uh, for love. And, at the moment, I guess the, hmm, maybe it would be somewhat more beneficial if you w- if we would identify ourselves with all the humanity as a whole. So, so you can clearly see when people travel around the world when they run into person from the same country, they say, "Oh, we see PR," and all this... Like, all the sudden, they f- find all these similarities. They, they, they find some... They, they befriend th- those folks earlier than others. Uh, so there is like a sense, some sense of the belonging. And I would say I think it would be overall good thing to the world, uh, for people to move towards, uh, I think it's even called open individualism. Uh, move toward the, uh, mindset of a larger and larger groups.

Lex Fridman 52:14 So, the challenge there, a- that's a beautiful vision and I share it, to expand that circle of empathy, that circle of love towards the entirety of humanity. But then you start to ask, well, where do you draw the line? Because why not expand it to other conscious beings? And then at the finally, for our discussion, something I think about is why not expand it to AI systems? Like, we, we start respecting each other when the oth- the person, the entity on the other side has the capacity to suffer, 'cause then we develop a capacity to sort of empathize. And so I could see AI systems that are interacting with humans more and more having conscious-like displays. So like, they display consciousness through language and through other means. And so then the question is like, well, is that consciousness? Because they're acting conscious. And so, you know, the reason we don't like torturing animals is because they look like they're suffering when they're tortured. And if AI looks like it's suffering when it's tortured, how is that not requiring of the same kind of empathy from us, and respect, and rights that animals do and other humans do?

Wojciech Zaremba 53:39 I think it requires empathy as well. I mean, I would like, guess, us or humanity or so make a progress in understanding what consciousness is, because I don't want just to be speaking about the, the philosophy but rather actually make a scientific... Uh, to have a, like a, you know, there was a time that people thought that, uh, there is a force of life. And, uh, the things that have this force, they are alive.

Lex Fridman 54:05 Mm-hmm.

Wojciech Zaremba 54:07 And, um, I think that there is actually a path to understand exactly what consciousness is. And, uh, um, in some sense, it might require essentially putting probes inside of a human brain-

Lex Fridman 54:21 Mm-hmm.

Wojciech Zaremba 54:21 ... uh, what Neuralink, uh, does.

Lex Fridman 54:23 So the goal there... I mean, there's several things with consciousness that make it a real discipline, which is one, is rigorous measurement of consciousness-

Wojciech Zaremba 54:31 Mm-hmm.

Lex Fridman 54:32 ... and then the other is the engineering of consciousness, which may or may not be related. I mean, you: could also run into trouble, like for example, in the United States for, for the department, DOT, Department of Transportation, you know, a lot of different places put a value on human life. I think DOT's, uh, value is $9 million per person. Sort of in that same way, you can get into trouble if you put a number on how conscious a being is, 'cause then you can start making policy. If a cow is, uh, 0.1...... or, like, um, 10% as conscious as a human, then you can start making calculations and it might get you into trouble. But then again, that might be a very good way to do it.

Wojciech Zaremba 55:19 I would like, uh, to move to that place that actually we have scientific understanding of what consciousness is.

Lex Fridman 55:24 Yeah.

Wojciech Zaremba 55:25 And then we'll be able to actually assign value. And I believe that there is even the path for the experimentation in, uh, in it. So, uh, you know, we, we said that, you know, you could put the probes inside of the brain. There is actually few other things that you could do with devices like Neuralink. So, you could imagine that the way even to measure if AI system is conscious is by literally just plugging into the brain.

Lex Fridman 55:52 Mm-hmm.

Wojciech Zaremba 55:53 Um, I mean, that assumes that it's kind of easy.

Lex Fridman 55:55 (laughs)

Wojciech Zaremba 55:55 But they're plugging into the brain and asking person if they feel that their consciousness expanded. Um, this direction, of course, has some issues. You can say, you know, if someone takes a psychedelic drug, they might feel that their consciousness expanded, even though that drug itself is not conscious.

Lex Fridman 56:11 Right. So, like, you can't fully trust the self-report of a person saying their, their consciousness is, is expanded or not. (sighs)

Lex Fridman 56:20 Uh, let me ask you a little bit about psychedelics 'cause, uh, there have been a lot of excellent research on, uh, different psychedelics, psilocybin, MDMA, uh, even DMT, drugs in general, marijuana too. Uh, what do you think psychedelics do to the human mind? It seems they take the human mind to some interesting places. Is that just a little, uh, hack, a visual hack? Or is there some profound expansion of the mind?

Wojciech Zaremba 56:49 So, let's see. I, I don't believe in magic. I believe in, uh, I believe in, uh, in science, in, in causality. Um, still, let's say, and then as I said, like, I think that the brain, that the, our subjective experience of reality is, uh, we, we live in the simulation run by our brain.

Lex Fridman 57:12 Mm-hmm.

Wojciech Zaremba 57:12 And the simulation that our brain runs, they can be very pleasant or very hellish. Drugs, they are changing some hyperparameters of the simulation. It is possible, thanks to change of these hyperparameters, to actually look back on your experience and even see that the given things that we took for granted, they are changeable. So, they allow to have a, a amazing perspective. There is also, for instance, the fact that after DMT, people can see the full movie inside of their head.

Lex Fridman 57:46 Mm-hmm.

Wojciech Zaremba 57:47 Gives me further belief that the brain can generate that full movie, that the brain is actually learning the model of reality to such extent that it tries to predict what gonna happen next.

Lex Fridman 58:00 Yeah, very high resolution, so it can replay realities, I'm sure.

Wojciech Zaremba 58:03 E- extremely high resolution. Yeah, it's also kind of interesting to me that somehow there seems to be some similarity between these, uh, drugs and meditation itself. And I actually started even these days to think about meditation as a psychedelic.

Lex Fridman 58:21 Mm.

Wojciech Zaremba 58:22 Um-

Lex Fridman 58:22 Do you practice meditation?

Wojciech Zaremba 58:24 I, I, I practice meditation. I mean, I went few times on the re- retreats. And it feels after, like, after second or third day of meditation, uh, th- th- there is a, there is almost like a sense of, you know, tripping.

Lex Fridman 58:41 What, what does a meditation retreat entail?

Wojciech Zaremba 58:44 So, when you w- you wake up, uh, early in the morning and you meditate for extended period of time, uh, and th-

Lex Fridman 58:52 Alone? Just, just so I do- did

Wojciech Zaremba 58:54 Uh, as- yeah.

Lex Fridman 58:54 ... keep in mind.

Wojciech Zaremba 58:54 So, it's optimized. Even though there are other people, it's, uh, optimized for isolation. So, you don't speak with anyone. You don't actually look into other people eyes.

Lex Fridman 59:03 Mm-hmm.

Wojciech Zaremba 59:04 And, uh, you know, you sit on that chair. Um, say, the Vipassana meditation tells you, uh, to focus on the breath. So, you try to put, uh, all the all- attention into breathing and, uh, breathing in and breathing out. And, um, crazy thing is that as you focus attention like that, uh, after some time, there's some s- starts coming back, like some memories that you've completely forgotten.

Lex Fridman 59:34 Mm-hmm.

Wojciech Zaremba 59:34 It almost feels like, um, that you have a mailbox and then you, you know, you are just like, uh, archiving email one by one.

Lex Fridman 59:42 Mm-hmm.

Wojciech Zaremba 59:43 And at some point, at some point, there is this, like, a amazing feeling of getting to mailbox zero.

Lex Fridman 59:50 (laughs)

Wojciech Zaremba 59:50 Zero emails. And, uh, it's very pleasant. It's, it's kind of... it's, it's, it's crazy to me that, um, that once you resolve these, uh, inner sto- stories or, like, uh, inner traumas-

Lex Fridman 1:00:07 Mm-hmm.

Wojciech Zaremba 1:00:08 ... then once there is nothing, uh, left, the default, uh, state of human mind is extremely peaceful and happy. Extreme... like, uh, some sense, it, it feels that the... it feels, at least to me, the way how when I was a child that I can look at any object and it's very beautiful. I have a lot of curiosity about the simple things.

Lex Fridman 1:00:34 Mm-hmm.

Wojciech Zaremba 1:00:34 And that's where usually meditation takes me.

Lex Fridman 1:00:38 Are you... what are you experiencing? Are you just taking in simple sensory information and they're just enjoying the rawness of that sensory information? So, there's no, there's no memories or all that kind of stuff? You're just enjoying being?

Wojciech Zaremba 1:00:55 Yeah, pretty much. I mean, still there is a, the, the, it's, it's thoughts are slowing down. Sometimes they pop up, but it's also s- s- somehow the extended meditation takes you to the space that they are way more friendly, you... way more positive. Um-There is also this, uh, this thing that, uh, we've ex- like, it almost feels that the ... It almost feels that the, we are constantly getting little bit of a reward function, and we are just spreading this reward function on various activities.

Lex Fridman 1:01:31 Mm-hmm.

Wojciech Zaremba 1:01:31 But if you stay still for extended period of time, it kind of accumulates, accumulates, accumulates.

Lex Fridman 1:01:37 Mm-hmm.

Wojciech Zaremba 1:01:37 And, uh, there is a, there is a sense, there is a sense that some point it passes some threshold and it feels as drop is falling into kind of ocean of love and peace. And that's like a-

Lex Fridman 1:01:51 Hmm.

Wojciech Zaremba 1:01:51 ... this is like a very pleasant, and that's what I'm saying, like a, that corresponds to the subjective experience. Some people, uh, I guess, in spiritual community, they describe it that that's their reality.

Lex Fridman 1:02:05 Mm-hmm.

Wojciech Zaremba 1:02:05 And I would say I believe that there are like a all sorts of subjective experience that one can have. And, uh, I believe that, for instance, meditation might take you to the subjective experiences which are very pleasant, collaborative.

Lex Fridman 1:02:17 Mm-hmm.

Wojciech Zaremba 1:02:17 And I would like a world to move toward a more collaborative, uh, uh, place. Yeah. I would say that's very pleasant and I-

Lex Fridman 1:02:27 Yeah.

Wojciech Zaremba 1:02:27 ... enjoy doing stuff like that.

Lex Fridman 1:02:28 I, I, um, I wonder how that maps to your, uh, mathematical model of love with, uh, the reward function combining a bunch of things. It, it seems like our life then is we're just ... we have this reward function and we're accumulating a bunch of stuff in it with weights. It's like, um, m- like multi-objective. And what meditation is, is you just remove them, remove them until the weight on one, uh, or just a few is, is very high, and that's where the pleasure comes from.

Wojciech Zaremba 1:03:05 Yeah, so s- something similar how I'm thinking about this. So I told you that there is this like a, uh, that there is a story of who you are.

Lex Fridman 1:03:13 Mm-hmm.

Wojciech Zaremba 1:03:14 And I think almost about it as a, you know, text prepended to GPT. (laughs)

Lex Fridman 1:03:20 (laughs) Yeah.

Wojciech Zaremba 1:03:21 And, uh, other ... Uh, some people refer to it as ego, okay?

Lex Fridman 1:03:24 Yeah.

Wojciech Zaremba 1:03:24 It's like a story-

Lex Fridman 1:03:25 (laughs)

Wojciech Zaremba 1:03:25 ... who, who, who you are, okay?

Lex Fridman 1:03:28 So ego is the prompt for GPT-3?

Wojciech Zaremba 1:03:30 Yes.

Lex Fridman 1:03:30 Or GPT, yeah.

Wojciech Zaremba 1:03:31 Yes, yes. And that's description of you. And then with meditation, you can get to the point that actually you experience things without a prompt.

Lex Fridman 1:03:38 Mm-hmm.

Wojciech Zaremba 1:03:39 And, uh, you experience things like a as they are. Th- you are not biased over the description how they supposed to be. Uh, that's very pleasant. And then with respect to the reward function, uh, it's possible to get to the point that, uh, there is dissolution of self, and therefore you can say that they are, they are having a, they are ... Or like your brain attempts to simulate the reward function of everyone else, or like everything. That's that there is this like a love which feels like a oneness with everything. And that's also, you know, very beautiful, very pleasant. At some point, you, you might have a lot of altruistic thoughts during th- that moment, and then the self, uh, always comes back.

Lex Fridman 1:04:19 How would you recommend, if somebody's interested in meditation, like a big thing to take on as a project? Would you recommend a meditation retreat? How many days? What kind of thing would you recommend?

Wojciech Zaremba 1:04:30 I think that actually retreat is the way to go. Um, it almost, uh, feels that the, um, as I said, like meditation is a psychedelic, but, uh, when you take it in the small dose, you might barely feel it.

Lex Fridman 1:04:43 Mm-hmm.

Wojciech Zaremba 1:04:43 Once you get the high dose, actually you're gonna feel it.

Lex Fridman 1:04:46 Mm-hmm.

Wojciech Zaremba 1:04:47 Um ...

Lex Fridman 1:04:48 So even cold turkey, if you haven't really seriously meditated for a prolonged period of time, just go to a retreat.

Wojciech Zaremba 1:04:54 Yeah, like a-

Lex Fridman 1:04:54 For how many days? How many days would you-

Wojciech Zaremba 1:04:55 I would start a weekend one.

Lex Fridman 1:04:57 Weekend, so like two, three days.

Wojciech Zaremba 1:04:59 Mm-hmm. And it's like a ... It's interesting that first or second day, it's hard, and at some point, it becomes easy.

Lex Fridman 1:05:07 There's a lot of seconds in a day. How hard is the meditation retreat, just sitting there in a chair?

Wojciech Zaremba 1:05:14 So the thing is, actually it literally just depends on your, uh, on the f- your own framing. Like if you are in the mindset that you are waiting for it to be over-

Lex Fridman 1:05:26 Mm-hmm.

Wojciech Zaremba 1:05:26 ... or you are waiting for, uh, nirvana to happen-

Lex Fridman 1:05:28 Mm-hmm.

Wojciech Zaremba 1:05:29 ... it will be very unpleasant.

Lex Fridman 1:05:30 Yeah.

Wojciech Zaremba 1:05:30 And in some sense, even the difficulty, it's not even in the lack of being able to speak with others. Like a, you are sitting there, your legs will hurt from sitting.

Lex Fridman 1:05:42 In terms of like the practical things, do you experience kind of discomfort, like physical discomfort of just sitting, like your, your butt being numb, your legs being sore, all that kinda stuff?

Wojciech Zaremba 1:05:54 Yes, you experience it, and then the, the, they teach you to observe it rather ... And it's like a ... Like, the crazy thing is you at first might have a feeling toward trying to escape it.

Lex Fridman 1:06:05 Yeah.

Wojciech Zaremba 1:06:05 And that becomes very apparent that that's extremely unpleasant.

Lex Fridman 1:06:08 Mm-hmm.

Wojciech Zaremba 1:06:09 And then you just, just observe it, and, uh, at some point, it, it just becomes, uh, it just is.

Lex Fridman 1:06:17 (laughs)

Wojciech Zaremba 1:06:17 It's like a ... I remember with Ilya told me some time ago that, uh, you know, he takes a cold shower-

Lex Fridman 1:06:23 Mm-hmm.

Wojciech Zaremba 1:06:23 ... and his mindset of taking a cor- cold shower was to embrace suffering.

Lex Fridman 1:06:28 Yeah. Excellent. I'm, I do the same.

Wojciech Zaremba 1:06:30 Th- this your style?

Lex Fridman 1:06:31 Yeah, it's my style. (laughs) I like this. (laughs)

Wojciech Zaremba 1:06:34 So my style is actually I, I also sometimes take cold showers.

Lex Fridman 1:06:38 Yeah.

Wojciech Zaremba 1:06:39 It is purely observing how the water goes through my body, like a purely being present, not trying to escape from there.

Lex Fridman 1:06:46 Yeah.

Wojciech Zaremba 1:06:46 And I would say then it actually becomes pleasant. It's not like, "Ugh."

Lex Fridman 1:06:53 Well, that, that's interesting. Um, I, I, I'm also ... That mean that's, that's the way to deal with anything really difficult, especially in the physical space, is to observe it. To, to say it's pleasant, hmm. It's a d- I would use a different word. You're, um ...... you're accepting of the full beauty of reality, I- I would say, 'cause it's pleasant. But yeah, I mea- in some sense, it is pleasant. That's the only way to deal with a cold shower, (laughs) is to, uh, become an observer and to find joy in it. Uh, same with, like, really difficult physical, uh, exercise, or like running for a really long time, endurance events. Just any time you're exhi- any kind of pain, I think the only way to survive it is not to resist it, just observe it.

Lex Fridman 1:07:46 You mentioned Ilya, Ilya Sutskever.

Wojciech Zaremba 1:07:49 He's very, he's our chief scientist, but also he's a very close friend of mine.

Lex Fridman 1:07:53 He co-founded OpenAI with you. I've, uh, spoken with him a few times. He's brilliant, I really enjoy talking to him. Uh, (laughs) his mind, just like yours, works in fascinating ways. Uh, both of you are not able to define deep learning simply. (laughs)

Wojciech Zaremba 1:08:10 (laughs)

Lex Fridman 1:08:12 Uh, what's it like having him as somebody you have technical discussions with on, in the space of machine learning, deep learning, AI, but also life? What's it like when these two, um, agents s- get into a self-play situation in, in a room? What's it like collaborating with him?

Wojciech Zaremba 1:08:33 So, I believe that we have, uh, extreme, uh, respect to each other. So, um, in, I love Ilya's insight, both like, uh, m- I guess about consciousness, uh, life, AI.

Lex Fridman 1:08:49 But, uh, in terms of the ... It's interesting to me 'cause y- y- you're a, a brilliant, uh, thinker in the space of machine learning, like intuition, like digging deep in what works, what doesn't, why it works, why it doesn't, and so is Ilya. I w- I'm wondering if there's interesting deep discussions you've had with him in the past, or disagreements that were very productive.

Wojciech Zaremba 1:09:15 So, I can say I also understood over the time where are m- my strengths. So obviously, we have plenty of AI discussions.

Lex Fridman 1:09:23 Mm-hmm.

Wojciech Zaremba 1:09:23 And, uh, um, and, you know, I myself have plenty of ideas, but like I consider Ilya, uh, one of the most prolific, uh, AI scientists in the entire world.

Lex Fridman 1:09:35 Mm-hmm.

Wojciech Zaremba 1:09:35 And, uh, I think that, um, I realized that maybe my super skill, uh, is, uh, being able to bring people to collaborate together, that I have some level of, uh, empathy that is unique in AI world. And that might come, you know, from either meditation, psychedelics, or let's say I read just hundreds of books on this topic. So, and I also went through a journey of, you know, I develop all sorts of, uh, algorithms. So, I think that maybe I can ... That's my s- superhuman skill. Uh, Ilya is, uh, one of the best AI scientists, but then I'm pretty good in assembling teams.

Lex Fridman 1:10:15 Mm-hmm.

Wojciech Zaremba 1:10:15 And I'm also not holding to people. Like, I'm growing people and then people become managers at OpenAI.

Lex Fridman 1:10:20 Gotcha.

Wojciech Zaremba 1:10:21 I grew many of them, like research managers.

Lex Fridman 1:10:23 So you, you find, you find places where you're excellent and, and he finds, like, his, his, his deep scientific insights is where he is, and y- you find ways you can d- the puzzle pieces fit together.

Wojciech Zaremba 1:10:36 Correct. Like, uh, you know, ultimately, for instance, let's say, Ilya, he doesn't manage people. Uh, that's not h- what he likes or so. Um, I, I like, I like hanging out with people. By default, I'm an extrovert and I care about people.

Lex Fridman 1:10:51 Oh, interesting. Okay. All right, okay. Cool. So, that, that fits perfectly together, right? But I, I mean, uh, I also just like your intuition about various problems in machine learning. He, he's definitely one I really enjoy. I remember talking to him about something I was struggling with, which is coming up with a good model for pedestrians, for human beings that cross the street, in the context of autonomous vehicles (laughs) . And he immediately started to, like, uh, formulate a framework within which you can evolve a model for pedestrians, like through self-play, all that kind of mechanisms. The depth of thought on a particular problem, especially problems he doesn't know anything about, is- is fascinating to watch. It's, makes you realize like, um, yeah, the- the- the limits of the ... that the human intellect might be limitless. It was just impressive to see a descendant of ape come up with clever ideas.

Wojciech Zaremba 1:11:52 Yeah, I mean, so even in the space of deep learning when you look at various people, there are people, you know, who invented, uh, some breakthroughs once, but there are very few people who did it multiple times.

Lex Fridman 1:12:06 Mm-hmm.

Wojciech Zaremba 1:12:06 And you can think, if someone invented it once, that might be just the sheer luck, and if someone invented it multiple times, you know, if a probability of inventing it once is one over million-

Lex Fridman 1:12:17 Yeah.

Wojciech Zaremba 1:12:17 ... then probability of inventing it twice or three times would be one over million squared, or, or to the power of three, uh, which- which would be just impossible. So, it literally means that it's- it's given that, uh, it's not the luck.

Lex Fridman 1:12:30 Yeah.

Wojciech Zaremba 1:12:31 Uh, and Ilya is one of these few people who, uh, uh, who have, uh, a lot of these inventions in his arsenal. It also feels that, um, you know, for instance, if you think about folks like Gauss or Euler, uh, you know, at first they read a lot of books, and then they did thinking, and then they figure out math.

Lex Fridman 1:12:53 Mm-hmm.

Wojciech Zaremba 1:12:53 And that's how it feels with Ilya.

Lex Fridman 1:12:56 Yeah.

Wojciech Zaremba 1:12:56 That, you know, at first he read stuff and then, like, he spent his thinking cycles and ...

Lex Fridman 1:13:02 That's a really good way to put it. When I talk to him, I, I see thinking. He's actually thinking. Like, he makes me realize that there's, like, deep thinking that the human mind can do. Like, most of us are not thinking deeply.... uh, like, you really have to put in a lot of effort to think deeply. Like, I have to really put myself in, in a place where I think deeply about a problem. It takes a lot of effort. It's like a, it's like an airplane taking off or something. You have to achieve deep focus. He, he's just, um, he's, uh, what is it? H- h- h- his brain is like a vertical takeoff, (laughs) in terms of airplane analogy. So it's interesting. But it... I mean, Cal Newport talks about this in his ideas of deep work. It's, you know, most of us don't work much at all in terms of, like, w- like, deeply think about particular problems, whether it's in math, engineering, all that kind of stuff. Y- you want to go to that place often, and that's real hard work, and some of us are better than, than others at that.

Wojciech Zaremba 1:14:09 So I think that the big piece, uh, has to do with actually even engineering your environment, that, such that it's conducive to that.

Lex Fridman 1:14:16 Yeah.

Wojciech Zaremba 1:14:16 So, um, see, both Ilya and I, uh, on the frequent basis, we kind of disconnect ourselves from the world in order to be able to do extensive amount of thinking.

Lex Fridman 1:14:27 Yes.

Wojciech Zaremba 1:14:28 So Ilya usually, he just, uh, uh, leaves iPad, uh, at hand. He loves his iPad. (laughs)

Lex Fridman 1:14:36 (laughs)

Wojciech Zaremba 1:14:36 And, uh, for me, I'm even sometimes, you know, just going for a few days to different location, to Airbnb, I'm turning off my phone, and there is no access to me.

Lex Fridman 1:14:47 Yeah.

Wojciech Zaremba 1:14:48 And, uh, that's extremely important for me to be able to actually just formulate new thoughts, to do deep work, rather than to be reactive.

Lex Fridman 1:14:57 Mm-hmm.

Wojciech Zaremba 1:14:57 And the, the, the older I am, the more of these, like, random tasks are a- at hand.

Lex Fridman 1:15:03 Before I go onto that, uh, thread, let me return to our friend GPT, and let me ask you another ridiculously big question. Can you give an overview of what GPT-3 is or, like you say in your Twitter bio, GPT n+1, how it works and why it works?

Wojciech Zaremba 1:15:24 So, um, GPT-3 is a humongous neural network. Um, let's assume that we know what is neural network, okay, by the definition. And, uh, it is trained on the entire internet just to predict next word. So let's say it sees part of the, uh, article and it... the only task that it has at hand, it is to say what would be the next word, uh, what would be the next word? And, uh, it becomes, uh, uh, really exceptional at the task of figuring out what's the next word. So you might ask, "Why would, uh, this be an important, uh, task? Why would it be important to predict what's the next word?" And it turns out that a lot of problems, uh, can be formulated, uh, as a text completion problem. So GPT is purely, uh, learning to complete the text. As you could imagine, for instance, if you are asking a question, uh, who is the president of United States, then GPT can give you an answer to it. Uh, it turns out that many more things can be formulated this way. You can format text in the way that you have sentence in English. You make it even look like, uh, some content of a website, uh, elsewhere which would be teaching people how to translate things between languages. So it would be EN: uh, text in English, FR:, and then you'll, uh, uh, and then you'll ask peop- uh, and then you'll ask model to, to continue. And it turns out that the... such a model is predicting translation from English to French. The crazy thing is that this model, uh, can be used for way more sophisticated tasks. So you can format text such that it looks like a conversation between two people, and that might be a conversation between you and Elon Musk and because the model read all the texts about Elon Musk, it will be able to predict Elon Musk words as it would be Elon Musk. It will speak about colonization of, uh, Mars, about s- sustainable future, and so on. And, um, it's also possible to, to even give arbitrary personality to the model. You can say, "Here is a conversation with a friendly AI bot."

Lex Fridman 1:17:36 Mm-hmm.

Wojciech Zaremba 1:17:38 And the model, uh, will complete the text as a friendly AI bot.

Lex Fridman 1:17:42 So, I mean, (sighs) how do I express how amazing this is? So just to clarify, uh, uh, a conversation, generating a conversation between me and Elon Musk, it wouldn't just generate good examples of what Elon would say, it would get the syntax all correct. So, like, interview style, it would say, like, Elon: and Lex:, like, it, it's not just like, uh, inklings of, um, semantic correctness. It's, like, the whole thing, grammatical, syntactic, semantic. It's just really, really impressive, uh, generalization.

Wojciech Zaremba 1:18:30 Yeah, I mean, I also want to, you know, provide some caveats though. It can generate few paragraphs of coherent text, but as you go to, uh, longer pieces, it, uh, it actually goes off the rails. If you would, uh, try to write a book, it won't work out, uh, this way. And-

Lex Fridman 1:18:45 What way does it go off the rails by the way? Is there interesting ways in which it goes off the rails? Like, (laughs) wha- wha- what falls apart first?

Wojciech Zaremba 1:18:54 So the model is trained on the... all the existing data, uh, that is out there, which means that it is not trained on its own mistakes. So for instance, if it would make a mistake, then, uh, I ke- so to give y- give you an example.

Lex Fridman 1:19:08 Sure.

Wojciech Zaremba 1:19:08 So let's say I have a conversation with a, a model pretending that is Elon Musk.

Lex Fridman 1:19:14 Mm-hmm.

Wojciech Zaremba 1:19:14 And then I start putting some, uh... I'm start actually making up things which are not factual.

Lex Fridman 1:19:20 Mm-hmm.

Wojciech Zaremba 1:19:21 Um, I would say-

Lex Fridman 1:19:22 Sounds like Twitter.

Wojciech Zaremba 1:19:23 (laughs)

Lex Fridman 1:19:23 (laughs) But I gotcha, sorry. Yeah.

Wojciech Zaremba 1:19:27 Um, like, uh, the... I don't know, I would say that Elon is my wife.... and the model will just keep on carrying, uh, it on and, uh-

Lex Fridman 1:19:35 As if it's true.

Wojciech Zaremba 1:19:37 Uh, yes. And in some sense, if you would have a normal conversation with Elon, he would be, "What the fuck?"

Lex Fridman 1:19:43 Yeah. There would be some feedback between... So the, the model is trained on things that humans have written but through the generation process, there's no human in the loop feedback.

Wojciech Zaremba 1:19:54 Correct.

Lex Fridman 1:19:55 That's fascinating.

Wojciech Zaremba 1:19:56 Ma- makes sense?

Lex Fridman 1:19:57 So it's magnified, it's like the errors get magnified and magnified.

Wojciech Zaremba 1:20:00 Correct.

Lex Fridman 1:20:01 And, and, and it's as, it's also interesting... I mean, first of all, humans have the same problem, it's just that we, uh, we make f- fewer errors and magnify the errors slower.

Wojciech Zaremba 1:20:13 Yeah, I think that the, actually what happens with humans is if you have a wrong belief about the world as a kid then very quickly you will learn that it's not correct because you are grounded in reality and you are learning from your new experience.

Lex Fridman 1:20:26 Yes. But do you think the model can correct itself too? Won't it, through the power of the representation and so the absence of (laughs) Elon Musk being your wife, information on the internet, won't it correct itself?

Wojciech Zaremba 1:20:43 There won't be examples like that.

Lex Fridman 1:20:45 So the errors will be subtle at first.

Wojciech Zaremba 1:20:48 Subtle at first and i- in some sense you can also say that the data that is not out there is a data which would represent how the human learns.

Lex Fridman 1:20:57 Mm-hmm.

Wojciech Zaremba 1:20:58 That's an A- and, and maybe model will be learned, trained on such a data then it would be better off.

Lex Fridman 1:21:03 How intelligent is GPT-3 do you think? Like, when you think about the nature of intelligence, it seems exceptionally impressive. But then if you think about the big AGI problem, is this footsteps along the way to AGI?

Wojciech Zaremba 1:21:21 So, um, let's see. It seems that intelligence itself is, there are multiple axes of it and, uh, I would expect that the, the systems that we are building, they might end up being superhuman on some axis and subhuman on some other axis. It would be surprising to me if on all axes simultaneously they would become superhuman. Um, of course people ask this question, is GPT a spaceship that, that would take us to m- to moon or are we putting a, building a ladder to heaven? That we are just building bigger and bigger ladder? And we don't know in some sense, uh, which one of these two-

Lex Fridman 1:22:00 But which one is better?

Wojciech Zaremba 1:22:01 (laughs)

Lex Fridman 1:22:03 I'm trying to... I like Stairway to Heaven, it's a good song, so I'm not exactly sure which one is better. But you're saying, like, the, the spaceship to the moon is actually effective?

Wojciech Zaremba 1:22:11 Correct. So people who criticize GPT-

Lex Fridman 1:22:14 Yeah.

Wojciech Zaremba 1:22:14 ... they say, "You guys just building a, a taller, uh, ladder-"

Lex Fridman 1:22:20 Mm-hmm.

Wojciech Zaremba 1:22:20 "... and it will never reach the moon." And, uh, at the moment I, I would say the way how I'm thinking is this is like a scientific question and, uh, I'm also, in heart I'm a builder creator and like, uh, I'm thinking, "Let's try out. Let's see how far it goes." And, uh, so far we see constantly that there is a progress.

Lex Fridman 1:22:42 Yeah. So w- do you think GPT-4, GPT-5, GPT n+1 will, um... there will be a phase shift, like a transition to, to a place where we'll be truly surprised? Then again, like, GPT-3 is already very, like, truly surprising. The people that criticize GPT-3 as a stair, uh, as a, what is it, ladder to heaven, I think too quickly get accustomed to how impressive it is, that the prediction of the next word can achieve such depth of semantics accuracy, of syntax, grammar and semantics. Um, do you th- do you think GPT-4 and 5 and 6 will continue to surprise us?

Wojciech Zaremba 1:23:28 Uh, I mean, definitely there will be more impressive models. The, there is a question of course if there will be a phase shift and, uh, the... also even the way I'm thinking about the, about these models is that they're... when we build these models, you know, we see some level of the capabilities but we don't even fully understand everything that the model can do and actually one of the best things to do is to allow other people to probe the model to even see what is possible.

Lex Fridman 1:23:57 Mm-hmm. Hence the, the using GPT as an API and opening it up to the world.

Wojciech Zaremba 1:24:05 Yeah. I mean... So when I'm thinking from perspective of kind of like a... obviously various people are, have concerns about AGI, including myself, um, and then when I'm thinking from perspective what's the strategy even to deploy these things to the world, the, the one strategy that I have seen many times working is the iterative deployment, that you deploy, um, slightly better versions and you allow other people to criticize you so you actually are tried out, you see where are there fundamental issues and it's almost, you don't want to be in that, uh, situation that you are holding into powerful system and there's, like, a huge overhang and you deploy it and it might have a random, chaotic impact on the world. So you actually want to be in the situation that they are gradually deploying systems.

Lex Fridman 1:24:56 I asked this question of Ilya, let me ask you of, uh, you this question. I've been reading a lot about Stalin and power. (laughs) If you're in possession of a system that's like AGI, that's exceptionally powerful, do you think your character and integrity might become corrupted? Like famously power corrupts and absolute power corrupts absolutely.

Wojciech Zaremba 1:25:25 So I believe that the, h- he want at some point to work toward distributing the power.... I think that, uh, you want to be in the situation that actually AGI is not controlled by a small number of people, uh, but, uh, essentially, uh, by a larger collective.

Lex Fridman 1:25:45 So, the thing is, that requires a George Washington style move. In the ascent to power there's always a- a moment when somebody gets a lot of power and they have to have the integrity and, uh, the moral compass to give away that power. That humans have been good and bad throughout history at this particular step, and I wonder... I wonder, we, like, blind ourselves in a, for example, between nations, a race, uh, towards, um, yeah, AI race between nations. We might blind ourselves and justify to ourselves the development of AI without distributing the power because we want to defend ourselves against China, against Russia, that kind of, that kind of logic. And, um, I wonder how we, um, how we design governance mechanisms that, uh, prevent us from becoming power hungry and in the process destroying ourselves.

Wojciech Zaremba 1:26:48 So, let's see. I have been thinking about this topic quite a bit, but I also wanna admit that, uh, once again I actually want to rely way more on Sam Altman on it.

Lex Fridman 1:26:57 Yes.

Wojciech Zaremba 1:26:58 He wrote an, uh, he wrote an, uh, excellent blog on how even to distribute wealth.

Lex Fridman 1:27:03 Mm-hmm.

Wojciech Zaremba 1:27:04 Um, and his propo- he proposed in his blog, uh, to tax, uh, equity of the companies rather than profit and to distribute it.

Lex Fridman 1:27:13 Interesting.

Wojciech Zaremba 1:27:13 And this is, this is an example of a Washington move.

Lex Fridman 1:27:18 Mm-hmm.

Wojciech Zaremba 1:27:20 I guess I personally have insane trust in Sam. He already spent plenty of money running, uh, universal basic income, uh, project.

Lex Fridman 1:27:31 Yeah.

Wojciech Zaremba 1:27:31 That like, uh, gives me, I guess, maybe some level of trust to him, but I also, I guess love him as a friend.

Lex Fridman 1:27:41 Yeah. I wonder, because we're sort of summoning a new set of technologies, I wonder if we'll be, um, cognizant... Like you're describing the processes at OpenAI, but it could also be at other places like in the US government, right? Uh, both China and the US are now full steam ahead on autonomous weapons systems development and that's really worrying to me because in the framework of something being a national security danger or a military danger you can do a lot of pretty dark things that, uh, blind our moral compass, and I think AI will be one of those things. Um, in some sense, the- the mission and the work you're doing at OpenAI is like the counterbalance to that, so you want to have more OpenAI and less autonomous weapons systems.

Wojciech Zaremba 1:28:38 Uh, uh, uh, I- I- I like these statements. Like, uh, to be clear, like, uh, this interesting, and I'm thinking about it myself, but, uh, this a place that I- I, like, uh, I put my trust actually in Sam's hands because it's extremely hard for me to reason about it.

Lex Fridman 1:28:53 Yeah. I mean, one important statement to make is, um, it's good to think about this.

Wojciech Zaremba 1:28:59 Uh, yeah, no question about it.

Lex Fridman 1:29:00 Right.

Wojciech Zaremba 1:29:00 No question about it.

Lex Fridman 1:29:01 So, even, like, low-level quote unquote "engineer," like there's such a... I remember I- I programmed a car, a- a- a RC car (laughs) that went really fast, like 30, 40 miles an hour, and I remember I was like sleep-deprived so I programmed it pretty crappily and it like, uh, the- the code froze. So, it's- it's doing some basic computer vision and it's going around on track, but it's going full speed (laughs) and, uh, there was a bug in the code that, uh, the car just went s- s- it didn't turn, it went straight full speed and smashed into the wall. And I remember thinking, the seriousness with which you need to approach the design of artificial intelligence systems and the programming of artificial intelligence systems is high 'cause the consequences are high. Like that- that little car smashing into the wall, for some reason I immediately thought of, like, an algorithm that controls nuclear weapons having the same kind of bug. And so like, the lowest level engineer and the CEO of a company all need to have the seriousness, uh, in approaching this problem and thinking about the worst case consequences.

Wojciech Zaremba 1:30:17 So, I think that is true. I mean, the... What I also recognize in myself and others even asking this question is that it evokes a lot of fear.

Lex Fridman 1:30:27 Mm-hmm.

Wojciech Zaremba 1:30:28 And, uh, fear itself ends up being actually quite dehabilitating. The place where I arrived at the moment might sound cheesy or so, but it's almost to build things out of love rather than fear.

Lex Fridman 1:30:44 Yeah.

Wojciech Zaremba 1:30:45 Uh, like a focus on how, uh, I can, you know, maximize the value, how the systems that I'm building might be, uh, useful.

Lex Fridman 1:30:56 Mm-hmm.

Wojciech Zaremba 1:30:56 Uh, I'm not saying that the fear doesn't exist out there, and like, uh, it totally makes sense to minimize it, but I don't want to be working because, uh, I'm scared.

Lex Fridman 1:31:06 Yeah.

Wojciech Zaremba 1:31:06 I want to be working out of passion, out of curiosity-

Lex Fridman 1:31:10 Yeah.

Wojciech Zaremba 1:31:10 ... out of the, you know, uh, looking forward for the positive future.

Lex Fridman 1:31:15 With, uh, the definition of love arising from, uh, rigorous practice of empathy, so not just, like, your own conception of what is, uh, good for the world, but, uh, always listening to others.

Wojciech Zaremba 1:31:26 Cor- correct. Like, uh, the love where I'm considering reward functions of others.

Lex Fridman 1:31:30 Of others (laughs) . To infi- l- limit to infinity is like a sum, like 1 to N where N is, uh, seven billion or whatever it is.

Wojciech Zaremba 1:31:38 Not- not projecting my reward functions on others.

Lex Fridman 1:31:40 Yeah, exactly (laughs) . Okay.

Lex Fridman 1:31:42 Can we just take a step back to something else super cool which is, uh, OpenAI Codex?

Wojciech Zaremba 1:31:47 Mm-hmm.

Lex Fridman 1:31:48 Can you give an overview of