Marcus Hutter: Universal Artificial Intelligence, AIXI, and AGI | Lex Fridman Podcast #75

1. 0:00 – 3:32

   Introduction

   1. LFLex Fridman0:00

      The following is a conversation with Markus Hutter, Senior Research Scientist at Google DeepMind. Throughout his career of research, including with Jürgen Schmidhuber and Shane Legg, he has proposed a lot of interesting ideas in and around the field of artificial general intelligence, including the development of AIXI, spelled A-I-X-I, model, which is a mathematical approach to AGI that incorporates ideas of Kolmogorov complexity, Solomonoff induction, and reinforcement learning. In 2006, Markus launched the 50,000 euro Hutter Prize for lossless compression of human knowledge. The idea behind this prize is that the ability to compress well is closely related to intelligence. This, to me, is a profound idea. Specifically, if you can compress the first 100 megabytes or one gigabyte of Wikipedia better than your predecessors, your compressor likely has to also be smarter. The intention of this prize is to encourage the development of intelligent compressors as a path to AGI. In conjunction with this podcast release, just a few days ago, Markus announced a 10X increase in several aspects of this prize, including the money, to 500,000 euros. The better your compressor works relative to the previous winners, the higher fraction of that prize money is awarded to you. You can learn more about it if you google simply Hutter Prize. I'm a big fan of benchmarks for developing AI systems and the Hutter Prize may indeed be one that will spark some good ideas for approaches that will make progress on the path of developing AGI systems. This is the Artificial Intelligence Podcast. If you enjoy it, subscribe on YouTube, give it five stars on Apple Podcasts, support it on Patreon, or simply connect with me on Twitter @LexFridman, spelled F-R-I-D-M-A-N. As usual, I'll do one or two minutes of ads now, and never any ads in the middle that can break the flow of the conversation. I hope that works for you and doesn't hurt the listening experience. This show is presented by Cash App, the number one finance app in the App Store. When you get it, use code LEXPODCAST. Cash App lets you send money to friends, buy Bitcoin, and invest in the stock market with as little as one dollar. Broker services are provided by Cash App Investing, a subsidiary of Square and member SIPC. Since Cash App allows you to send and receive money digitally, peer-to-peer, as security in all digital transactions is very important, let me mention the PCI Data Security Standard that Cash App is compliant with. I'm a big fan of standards for safety and security. PCI DSS is a good example of that, where a bunch of competitors got together and agreed that there needs to be a global standard around the security of transactions. Now we just need to do the same for autonomous vehicles and AI systems in general. So again, if you get Cash App from the App Store or Google Play and use the code LEXPODCAST, you'll get ten dollars, and Cash App will also donate ten dollars to FIRST, one of my favorite organizations that is helping to advance robotics and STEM education for young people around the world. And now here's my conversation with Markus Hutter.

2. 3:32 – 5:48

   Universe as a computer

   1. LFLex Fridman3:32

      Do you think of the universe as a computer or maybe an information processing system? Let's go with the big question first.

   2. MHMarcus Hutter3:39

      Okay. I'll go with the big question first, yeah. I think it's a very interesting hypothesis or idea. And, uh, I have a background in physics, so I know a little bit about physical theories, the standard model of particle physics and general relativity theory, and they are amazing and describe virtually everything in the universe, and they're all, in a sense, computable theories. I mean, they are very hard to compute. And, you know, it's very elegant, simple theories which describe virtually everything in the universe. So there's a strong indication that somehow the universe, um, is computable, but it's a plausible hypothesis.

   3. LFLex Fridman4:17

      So what- why do you think, just like you said, general relativity, quantum field theory, why do you think that the laws of physics are so nice and beautiful and simple and compressible? Do- do you think our universe was designed, uh, is naturally this way or are we just focusing on the parts that are especially compressible? Our human minds just enjoy something about that simplicity, and in fact, there's other things that are not so compressible.

   4. MHMarcus Hutter4:46

      No, I- I strongly believe, and I'm pretty convinced that the universe is inherently, um, beautiful, elegant, and simple, and described by these equations, and we're not just picking that, um. I mean, if there were some phenomena which cannot be neatly described, scientists would try that, right? And, you know, there's biology which is more messy, but we understand it, it's an emergent phenomena, and, you know, it's complex systems but they still follow the same rules, right, of quantum electrodynamics, um, all of chemistry follows that and we know that. I mean, we cannot compute everything because we have limited computational resources. No, I think it's not a bias of the humans, but it's objectively simple. I mean, of course you never know, you know, maybe there's some corners very far out in the universe or super, super tiny below the nucleus of- of- um, of atoms or, well, parallel universes where, which are not nice and simple, but, um, there's no evidence for that, and we should apply Occam's razor and, you know, choose the simplest three consistent with, uh... But also it's a little bit self-referential. (laughs)

3. 5:48 – 9:26

   Occam's razor

   1. MHMarcus Hutter5:48

   2. LFLex Fridman5:48

      So maybe a- a quick pause. What is Occam's razor?

   3. MHMarcus Hutter5:51

      So Occam's razor says that you should not multiply entities beyond necessity, which sort of if you translate (laughs) it to proper English-... means, uh, and, and, you know, in the scientific context means that if you have two theories or hypotheses or models which equally well describe the phenomenon of your study or the data, you should choose the more simple one.

   4. LFLex Fridman6:14

      So, that's just a principle?

   5. MHMarcus Hutter6:15

      Yes.

   6. LFLex Fridman6:16

      Or sort of that's not like a provable law, perhaps? Perhaps we'll, we'll, we'll, we'll, we'll kind of discuss it and think about it. But what's the intuition of why the simpler answer is the one that is likelier to be more correct descriptor of whatever we're talking about?

   7. MHMarcus Hutter6:35

      I believe that Occam's razor is probably the most important principle in science. I mean, of course, we

   8. NANarrator6:41

      (sighs)

   9. MHMarcus Hutter6:41

      ... do logical deduction and we do experimental design, yeah, but, um, science is about finding, uh, understanding the world, finding models of the world. And we can come up with crazy complex models which, you know, explain everything but predict nothing. But, um, the simple model seem to have predictive power, and it's a valid question why. Uh, and, um, there are two answers to that. You can just accept it. That is the principle of science. And we use this principle, and it seems to be successful. We don't know why, but-

   10. NANarrator7:14

       Yeah.

   11. MHMarcus Hutter7:14

       ... it just happens to be. Or you can try, you know, find another principle which explains Occam's razor. And if you start with the assumption that the world is governed by simple rules, then, um, there's a bias towards simplicity. And applying Occam's razor, um, is the mechanism to finding these rules. And actually, in a more quantitative sense, and we come back to that later in terms of Solomoff induction, you can rigorously prove that. If you assume that the world is simple, then Occam's razor is the best you can do in a certain sense.

   12. LFLex Fridman7:49

       So, apologize for the romanticized question, but, uh, why do you think, outside of its effectiveness, why do we... do you think we find simplicity so appealing as human beings? Why does it just, why does E equals mc2 seem so beautiful to us humans?

   13. MHMarcus Hutter8:08

       I guess mostly, in general, many things can be explained by an evolutionary argument.

   14. LFLex Fridman8:14

       (laughs)

   15. MHMarcus Hutter8:15

       And, you know, there are some artifacts in humans which, you know, are just artifacts and not e- evolutionary necessary.

   16. LFLex Fridman8:20

       Yeah.

   17. MHMarcus Hutter8:21

       But, um, with this beauty and simplicity, it's, I believe, at least-

   18. LFLex Fridman8:27

       You-

   19. MHMarcus Hutter8:27

       ... the core is, um, about, um, like science, um, finding regularities in the world, understanding the world, which is necessary for survival, right? You know, if I look, eh, at a bush, right, and I just see noise, and there is a tiger, right, and eats me, then I'm dead. But if I try to find a pattern, and we know that humans are prone to, um, find more patterns in data than there are, you know, like, you know, the Mars face and all these things. Um, but this bias towards finding patterns even if they are not... but, I mean, it's best of course if they are, yeah? Helps us for survival.

   20. LFLex Fridman9:06

       Yeah, that's fascinating. I haven't thought really about the... I thought I just loved science, but, uh, there, there, i- indeed, from, in terms of just (laughs) for survival purposes, there is an evolutionary argument for why, uh, why we find the work of Einstein so beautiful. Maybe a quick, uh,

4. 9:26 – 15:05

   Solomonoff induction

   1. LFLex Fridman9:26

      small tangent. Could you describe what Solomoff induction is?

   2. MHMarcus Hutter9:30

      Yeah. So that's a theory which I claim, and Ray Solomoff sort of claimed, you know, a long time ago, that this solves the big philosophical problem of induction. And I believe the claim is essentially true. And what it does is the following. So, okay, for the, um, picky listener, um, induction can be interpreted narrowly and widely. Uh, narrow means inferring models from data, and widely means also then using these models for doing predictions. So, prediction is also part of, of the induction. So I, I'm a little bit sloppy sort of with the terminology, and maybe that comes from Ray Solomoff, you know, being sloppy. Maybe I shouldn't say that (laughs) .

   3. LFLex Fridman10:16

      (laughs)

   4. MHMarcus Hutter10:16

      He can't complain anymore. Um, so let me explain a little bit this theory-

   5. LFLex Fridman10:20

      Yeah, okay.

   6. MHMarcus Hutter10:21

      ... um, in simple terms. So assuming you have a data sequence, um, make it very simple, the simplest one, say, 11111 and you see, well, 100 ones, yeah, what do you think comes next, yeah? The natural out- I'm gonna speed up a little bit. The natural answer is, of course, you know, one. Okay? And the question is why. Okay? Well, we see a pattern there. Yeah? Okay, there's a one, and we repeat it. And why should it suddenly after 100 ones be different? So what we're looking for is simple explanations or models for the data we have. And now the question is, our model has to be presented in a certain language. In which language do we use? Um, in science, we want formal languages. And we can use mathematics, or we can use programs on a computer. So, abstractly on a Turing machine, for instance, or it can be a general purpose computer. So, um, and there are of course lots of models of. You can say maybe it's 100 ones, and then 100 zeros, and 100 ones, that's a model, right? But there are simpler models. There's a model print one loop, yeah, that also explains the data. And if you push that to the extreme, um, you are looking for the shortest program which, if you run this program, reproduces the data you have. It will not stop. It will continue naturally. And this, you take for your prediction. And on the sequence of ones, it's very plausible, right, that print one loop is the shortest program. We can give some more complex examples like 12345. What comes next? The short program is again, you know, counter, um, and so that is roughly speaking how Solomoff induction works. Um, the extra twist is that it can also deal with noisy data. So, if you have for instance a coin flip, say a biased coin which comes up head with 60% probability, um-... then it will predict, um, it will learn and figure this out and after a while, it predict, "Oh, the next, um, coin flip will be head with probability 60%." So it's just a ƒ version of that.

   7. LFLex Fridman12:14

      But the goal is, the dream is always the search for the short program.

   8. MHMarcus Hutter12:18

      Yes. Yeah. Well, in Solomanov induction, precisely what you do is, so you combine... So looking for the shortest program is like applying a postulate, like looking for the simplest theory. There's also Epicurus' principle which says, "If you have multiple hypotheses which equally well describe your data, don't discard any of them. Keep all of them around, you never know." And you can put that together and say, "Okay, I have a bias towards simplicity, but I don't rule out the larger models." And technically what we do is we weigh the shorter models higher and the longer models lower, and you use a Bayesian techniques, you have a prior, and, uh, um, which is precisely 2 to the minus the complexity of the program, um, and you weigh all this hypotheses and take this mixture and then you get also the stochasticity in.

   9. LFLex Fridman13:07

      Yeah. Like many of your ideas, that, that's just a beautiful idea of weighing based on the simplicity of the program. I, I love that. That, that, that seems to me maybe very human-centric concept. Seems to me to be a very appealing way of discovering good programs in this world. You've used the term compression quite a bit, and I think that's a, that's a beautiful idea, sort of. We just talked about simplicity and maybe science or just all of our intellectual pursuits is basically the attempt to compress the complexity all around us into something simple. So, uh, what does this word mean to you, compression?

   10. MHMarcus Hutter13:50

       I essentially have already explained it, so it, compression means, for me, finding short programs, um, for the data or the phenomena at hand. You could interpret it more widely as, you know, finding, um, simple theories which can be mathematical theories or maybe even informal, you know, like, you know, just in words. Um, compression means finding short descriptions, explanations, programs for the data.

   11. LFLex Fridman14:15

       Do, do you see science as a kind of, uh, our human attempt at compression? So, so we're speaking more generally, 'cause when you say programs, you're kind of zooming in on a particular sort of almost like a computer science, artificial intelligence focus. But do you see all of human endeavor as a kind of compression?

   12. MHMarcus Hutter14:34

       Well, at least all of science I see as a endeavor of compression. (laughs) Not all of humanity maybe. Um, and, well, there are also some other aspects of science like experimental design, right? I mean, we, we, we create experiments specifically to get extra knowledge, and this is, um, that isn't part of the decision-making process, um, but, um, once we have the data, to understand the data is essentially compression. So I don't see any difference between compress- compression, understanding, and prediction.

5. 15:05 – 20:06

   Kolmogorov complexity

   1. MHMarcus Hutter15:05

   2. LFLex Fridman15:06

      So we're jumping around topics a little bit, but, uh, returning back to simplicity, uh, a fascinating concept of Kolmogorov complexity. So, eh, in your sense, do most objects in the, in our mathematical universe have high Kolmogorov complexity? And maybe what is... First of all, what is Kolmogorov complexity?

   3. MHMarcus Hutter15:26

      Okay. Kolmogorov complexity is a notion of, of simplicity or complexity, and, um, it takes the compression view to the extreme. So I explained before that, um, if you have some data sequence, just think about a file in a computer and, at best, sort of, you know, just a string of bits. And, um, if you... And we, we have data compressors like we compress big files into, say, zip files with certain compressors, and you can also produce self-extracting arXives. That means it's an executable. If you run it, it reproduces your original file without needing an extra decompressor. It's just the decompressor plus the arXiv together in one. And now there are better and worse compressors, and you can ask, what is the ultimate compressor? So what is the shortest possible self-extracting arXiv you could produce for a certain dataset, yeah, which reproduces the dataset? And the length of this is called the Kolmogorov complexity, and arguably, that is the information content in the dataset. I mean, if the dataset is very redundant or very boring, you can compress it very well so the information content should be low, and, you know, it is low according to this definition.

   4. LFLex Fridman16:36

      So it's the length of the shortest program that summarizes the data?

   5. MHMarcus Hutter16:41

      Yes. Yeah.

   6. LFLex Fridman16:42

      And w- what's your sense of, of our sort of universe wh- when we think about the different... the different objects in our universe that we tr- concepts or whatever, the, the, uh, at, at, at every level. Do they have high or low Kolmogorov complexity? So what's the hope? Uh, do we have a lot of hope and be able to summarize, um, much of our world?

   7. MHMarcus Hutter17:05

      Uh, that's a tricky and difficult question. So, as I said before, I believe that the whole universe, based on the evidence we have, is very simple, so has a very short description, the whole-

   8. LFLex Fridman17:19

      Sorry. To, to, to linger on that, the whole universe, what does that mean? Do you mean at the, at the very basic fundamental level in order to create the universe?

   9. MHMarcus Hutter17:28

      Yes. Yeah. So you need a very short program, and you run it on a-

   10. LFLex Fridman17:33

       To get the thing going.

   11. MHMarcus Hutter17:34

       To get the thing going, and then it will reproduce our universe. Um, there's a problem with noise. Um, we can come back to that later possibly, um-

   12. LFLex Fridman17:42

       Is noise a problem or a fea- is it a bug or a feature?

   13. MHMarcus Hutter17:46

       I would say it makes our life as a scientist really, really much harder. I mean, think about without noise, we wouldn't need all of the statistics.

   14. LFLex Fridman17:56

       But that may be, w- we wouldn't feel like there's a free will. Maybe we need that for the-

   15. MHMarcus Hutter18:00

       Yeah.

   16. LFLex Fridman18:00

       ... for the ex- for the (laughs)

   17. MHMarcus Hutter18:01

       This is an illusion that noise can give you free will, I think. (laughs)

   18. LFLex Fridman18:04

       Yeah. S- so that's, at least in that way, it's a feature.

   19. MHMarcus Hutter18:06

       But also, if you don't have noise, you have chaotic phenomena, which are effectively like noise. So we can't, you know, get away with statistics even then. I mean, think about rolling a dice and, you know, forget about quantum mechanics, and you know exactly how you, you throw it. But, I mean, it's still so hard to compute the trajectory that effectively it is best to model it, you know, as, you know, coming out with, um, a number with probability one over six. Um, but, um, from, from this set of philosophical Kolmogorov complexity perspective, if we didn't have noise, then arguably you could describe the whole universe as, um, well, as, uh, standard model plus generativity. I mean, we, we don't have a theory of everything yet, but sort of assuming we are close to it or have it, yeah? Plus the initial conditions, which may hopefully be simple. And then you just run it, and then you would reproduce the universe. Um, but that's spoiled by noise or by chaotic, um, systems, um, or by initial conditions, which, you know, may be complex. So now if we don't take the whole universe, but just, um, a subset, you know, just take planet Earth. Planet Earth cannot be compressed, you know, into a couple of equations. Um, this is a hugely complex system.

   20. LFLex Fridman19:19

       So interesting. So b- when you look at the window, like the whole thing might be simple, but when you just take a small window, then, um ...

   21. MHMarcus Hutter19:26

       It may become complex. And that may be counterintuitive, but, um, there's a very nice analogy, um, the, the book, uh, The Library of All Books. So imagine you have a normal library with interesting books, and you go there. Great, lots of information, and huge, quite complex, yeah? So now I create a library which contains all possible books, say, of 500 pages. So the first book just has A, A, A, A, A over all the pages.

   22. LFLex Fridman19:49

       Yes.

   23. MHMarcus Hutter19:49

       The next book, A, A, A and ends with B and so on. I create this library of all books. I can write a super short program which creates this library. So this library, which has all books, has zero information content. And you take a subset of this library, and suddenly you have a lot of information in there.

   24. LFLex Fridman20:05

       So that's fascinating.

6. 20:06 – 26:03

   Cellular automata

   1. LFLex Fridman20:06

      Uh, I think one of the most beautiful object, mathematical objects that at least today seems to be understudied or under-talked about is cellular automata. Uh, what lessons do you draw from sort of the game of life for cellular automata, where you start with the simple rules, just like you were describing with the universe, and somehow complexity emerges? Do you feel like you have an intuitive grasp on the behavior, the fascinating behavior of such systems, where some, like you said, some chaotic behavior could happen, some complexity could emerge? Some, it could die out in some very rigid structures. Do you have a sense about, uh, cellular automata that somehow transfers maybe to the bigger questions of our universe?

   2. MHMarcus Hutter20:51

      Yeah. The cellular automata, and especially the Conway's Game of Life, um, is really great because these rules are so simple. You can explain it to every child, and even by hand you can simulate a little bit, and you see these beautiful patterns emerge. And people have proven, you know, that it's even Turing-complete. You cannot just use a computer to simulate game of life, but you can also use game of life to simulate any computer.

   3. LFLex Fridman21:13

      Computer.

   4. MHMarcus Hutter21:13

      Um, that is (laughs) truly amazing. And it's, it's the prime example probably to demonstrate that very simple rules can lead to very rich phenomena. And people, you know, sometimes, you know, how can, how is chemistry and biology so rich? I mean, this can't be based on simple rules, yeah? But no, we know quantum electrodynamics describes all of chemistry. And, um, and we come later back to that. I claim intelligence can be explained or described in one single equation, this very rich phenomenon. Um, you asked also about whether, you know, I understand this phenomenon.

   5. LFLex Fridman21:49

      Mm-hmm.

   6. MHMarcus Hutter21:50

      And it's probably not, um, and there's this saying, "You never understand really things, you just get used to them." And (laughs) -

   7. LFLex Fridman21:59

      (laughs) Oh, wow.

   8. MHMarcus Hutter22:00

      ... I think I got pretty used-

   9. LFLex Fridman22:02

      That's a good line.

   10. MHMarcus Hutter22:02

       ... used to cellular automata. So you believe that you understand now why this phenomenon happens, but I give you a different example. I didn't play too much with, with Conway's Game of Life, but, um, a little bit more, um, with fractals and with the Mandelbrot set, uh, these beautiful, you know, patterns. Just, just look Mandelbrot set. Um, and well, when the computers were really slow and I just had a black and white monitor, and I programmed my own programs on a, in Assembler to- (laughs)

   11. LFLex Fridman22:29

       Assembler, wow. Wow, you're legit. (laughs)

   12. MHMarcus Hutter22:33

       (laughs) T- to, to get these fractals on the screen, and I was mesmerized. And much later ... So I returned to this, you know, every couple of years, and then I tried to understand what is going on. And you can understand a little bit. So I tried to derive the locations. You know, there are these, um, um, um, circles, and, uh, the apple shape, and then you have, um, smaller, uh, Mandelbrot sets recursively in this set. And there's a way to mathematically, by solving high order polynomials, to figure out where these centers are, and what size they are approximately. And by sort of un- mathematically approaching this problem, you slowly get a feeling of, um, why things are like they are, and that sort of is an, you know, first step to understanding why this rich phenomena appears.

   13. LFLex Fridman23:24

       Do you think it's, it's possible? What's your intuition? Do you think it's possible to reverse engineer and find the short program that generated the, these fractals, sort of by w- looking at the fractals?

   14. MHMarcus Hutter23:36

       Well, in principle, yes, yeah. Um, so I mean, in principle what you can do is, um, you take, you know, any data set. You know, you take these fractals or you take whatever, your data set, whatever you have. Uh, say a picture of Conway's Game of Life, and you run through all programs.

   15. LFLex Fridman23:53

       Mm-hmm.

   16. MHMarcus Hutter23:53

       You take programs as one, two, three, four, and all these programs, you run them all in parallel in so-called dovetailing fashion, give them...... computational resources, first one 50%, second one half resources, and so on, and let them run. Wait until they halt, give an output, compare it to your data, and if some of these programs produce the correct data, then you stop, and then you have already a sum program. It may be a long program because it's faster, and then you continue and you get shorter and shorter programs until you eventually find the shortest program. The interesting thing, you can never know whether it's the shortest program, because there could be an even shorter program which is just even slower, and we just have to wait, yeah?

   17. LFLex Fridman24:32

       (laughs)

   18. MHMarcus Hutter24:32

       But asymptotically, and actually after finite time, you have the shortest program. So, this is a theoretical but completely impractical way of finding, um, the underlying, um, uh, structure in every dataset, and that is what Solomonic induction does and Kolmogorov complexity. In practice, of course, we have to approach the problem more intelligently. And then, um, if you take resource limitations into account, there's, for instance, field of pseudo-random numbers, yeah? And, um, these are random numbers, so these are deterministic sequences, but no algorithm which is fast, fast means runs in polynomial time, can detect that it's actually deterministic. So we can produce interesting, I mean, random numbers are maybe not that interesting, but just an example, we can produce complex-looking data and we can then prove that no fast algorithm can detect the underlying pattern.

   19. LFLex Fridman25:31

       Which is, um, unfortunately, um, is, uh, that's a big challenge for our search for simple programs in the space of artificial intelligence perhaps.

   20. MHMarcus Hutter25:42

       Yes, it definitely is for artificial intelligence, and it's quite surprising that it's, I can't say easy, I mean, 'cause it's, (laughs) it's really hard to find-

   21. LFLex Fridman25:51

       (laughs)

   22. MHMarcus Hutter25:51

       ... these theories, but, um, apparently it was possible for human minds, uh, to find these simple rules in the universe. It could have been different, right?

   23. LFLex Fridman25:59

       It could have been different. It's, it's, uh, it's awe-inspiring.

7. 26:03 – 35:26

   What is intelligence?

   1. LFLex Fridman26:03

      So let me ask another absurdly big question.

   2. MHMarcus Hutter26:08

      (laughs)

   3. LFLex Fridman26:09

      What is intelligence, in your view?

   4. MHMarcus Hutter26:13

      So, I have, of course, a definition.

   5. LFLex Fridman26:15

      (laughs)

   6. MHMarcus Hutter26:17

      (laughs)

   7. LFLex Fridman26:17

      I wasn't sure what you were gonna say, 'cause you could've just as easily said, "I have no clue."

   8. MHMarcus Hutter26:21

      Which many people would say-

   9. LFLex Fridman26:23

      Yeah. (laughs)

   10. MHMarcus Hutter26:23

       ... but I'm not modest in this question.

   11. LFLex Fridman26:25

       (laughs)

   12. MHMarcus Hutter26:26

       Um, so the, the informal version, um, which I worked out together with Shane Leck, who co-founded DeepMind, is that intelligence measures an agent's ability to perform well in a wide range of environments. So, that doesn't sound very impressive, and, but it, these words have been very carefully chosen. And, um, there is a mathematical theory behind that, and we come back to that later. And if you look at this, this definition by itself, it seems like yeah, okay, but it seems a lot of things are missing. But if you think it through, then you realize that most, and I claim all of the other traits, at least of rational intelligence, which we usually associate with intelligence, are emergent phenomena from this definition. Like, you know, creativity, memorization, planning, knowledge, um, you all need that in order to perform well in a wide range of environments. So you don't have to explicitly mention that in a definition.

   13. LFLex Fridman27:30

       Interesting. So yeah, so the consciousness, abstract reasoning, all, all these kinds of things are just emerging for now and that help you in, uh, towards, s- well, can you say the definition again? So mul- multiple environments. Uh, did you mention the word goals?

   14. MHMarcus Hutter27:46

       No, but we have an alternative definition. Instead of performing well, you can just replace it by goals. So, uh, intelligence measures an agent's ability to achieve goals in a wide range of environments. That's more or less it.

   15. LFLex Fridman27:56

       Well, but it's interesting 'cause in there, there's an injection of the word goals. So w- we wanna specify, there, there should be a goal.

   16. MHMarcus Hutter28:03

       Yeah, but perform well is sort of, what does-

   17. LFLex Fridman28:05

       It's-

   18. MHMarcus Hutter28:05

       ... does it mean is the same problem.

   19. LFLex Fridman28:06

       Yeah.

   20. MHMarcus Hutter28:07

       Um-

   21. LFLex Fridman28:08

       Th- there's a little bit of gray area, but it's much closer to something that could be formalized. Are, in your view, are humans... Where do humans fit into that definition? Are they general intelligence systems that are able to perform m- uh, in a... Like, how good are they at fulfilling that definition, at performing well in multiple environments?

   22. MHMarcus Hutter28:31

       Yeah, that's a big question. I mean, the humans are performing best among all the-

   23. LFLex Fridman28:37

       Species on Earth?

   24. MHMarcus Hutter28:37

       ... species we know, we know of, yeah.

   25. LFLex Fridman28:40

       Depends. You could say that trees and plants are doing a better job. They'll probably outlast us, so.

   26. MHMarcus Hutter28:46

       Yeah, but they are in a much more narrow environment, right? I mean, you just, you know, have a little bit of air pollutions and these trees die, and we can adapt, right? We build houses, we build filters, we, we, um, we do-

   27. LFLex Fridman28:58

       Yeah.

   28. MHMarcus Hutter28:58

       ... geoengineering, so-

   29. LFLex Fridman28:59

       So the multiple environment part (soft music plays)

   30. MHMarcus Hutter29:01

       ... yeah, that is very important, yeah?
8. 35:26 – 1:05:24

   AIXI - Universal Artificial Intelligence

   1. LFLex Fridman35:27

      So you mentioned you're one of the, one of the only people who dared boldly to try to formalize ar- the idea of artificial general intelligence. To, to have a, um, a mathematical framework for intelligence, just like as we mentioned, termed AIXI, A-I-X-I. So let me ask the basic question. Uh, what is AIXI?

   2. MHMarcus Hutter35:54

      Okay, so let me first say what it stands for, because-

   3. LFLex Fridman35:57

      What it stands for, actually that's probably the more basic question.

   4. MHMarcus Hutter36:00

      Yeah. (laughs)

   5. LFLex Fridman36:01

      (laughs)

   6. MHMarcus Hutter36:01

      The first question is usually how, how it's pronounced, but finally I put it on the website how it's pronounced.

   7. LFLex Fridman36:06

      Yeah.

   8. MHMarcus Hutter36:06

      So people... (laughs) And you figured it out. (laughs)

   9. LFLex Fridman36:09

      (laughs) Yeah. Yeah.

   10. MHMarcus Hutter36:10

       The name comes from AI, artificial intelligence, and the XI is the Greek letter xi.... which I used for Solomanoff's distribution for quite stupid reasons, which I'm not willing to repeat here in front of camera.

   11. LFLex Fridman36:27

       Sure. (laughs) .

   12. MHMarcus Hutter36:27

       (laughs) . So, it just happened to be, uh, more or less arbitrary, I chose this xi.

   13. LFLex Fridman36:31

       Yeah.

   14. MHMarcus Hutter36:31

       Um, but it also has nice, um, other interpretations. So, um, there are actions and perceptions in this model, right? An agent has actions and perceptions. And over time, so this is A index i, X index i. So there's an action at time i and then followed by perception-

   15. LFLex Fridman36:47

       (laughs) . That's funny.

   16. MHMarcus Hutter36:48

       ... at time i.

   17. LFLex Fridman36:49

       Yeah.

   18. MHMarcus Hutter36:49

       Um-

   19. LFLex Fridman36:49

       We'll go with that. I'll edit out the first part. Mm.

   20. MHMarcus Hutter36:51

       Yeah. (laughs) .

   21. LFLex Fridman36:52

       I'm just kidding. (laughs) .

   22. MHMarcus Hutter36:53

       I have some more interpretations.

   23. LFLex Fridman36:54

       Yeah, go ahead.

   24. MHMarcus Hutter36:55

       So, uh, at some point, maybe five years ago or 10 years ago, I discovered in, uh, in Barcelona, it was in a big church, there was in, you know, in stone engraved some text and the word ikigai appeared there-

   25. LFLex Fridman37:10

       (laughs) .

   26. MHMarcus Hutter37:10

       (laughs) ... a couple of times. I was very surprised and, and, uh, and, and happy about it and I looked it up so it is in Catalan language and it means with some interpretation of that's it, that's the right thing to do. Yeah. Heureka.

   27. LFLex Fridman37:24

       Oh. So it's almost like destined, uh, somehow came-

   28. MHMarcus Hutter37:28

       Yeah.

   29. LFLex Fridman37:28

       ... came kind of like- (laughs)

   30. MHMarcus Hutter37:30

       Yeah. Yeah.
9. 1:05:24 – 1:12:14

   Where do rewards come from?

   1. LFLex Fridman1:05:24

      Right. So maybe le- let me ask about objective functions because that, uh, rewards, uh, it seems to be an important part. The rewards are kind of given to the system. Uh, for a lot of people, s- the, um, the specification of the objective function, uh, i- is a key part of intelligence. Like y- the in the agent itself figuring out what is important. Wh- what do you think about that? Is- is it possible within the AIXI framework to yourself discover the reward based on which you should operate?

   2. MHMarcus Hutter1:06:05

      Okay, that will be a long answer.

   3. LFLex Fridman1:06:07

      (laughs)

   4. MHMarcus Hutter1:06:07

      Um, so, um, and that is a very interesting question, and I, uh, I'm asked a lot about this question, where do the rewards come from?

   5. LFLex Fridman1:06:16

      (laughs)

   6. MHMarcus Hutter1:06:16

      And, uh, that depends, yeah? So, um, and then, you know, I give you now a couple of answers.

   7. LFLex Fridman1:06:22

      Yeah.

   8. MHMarcus Hutter1:06:22

      So if we want to build agents... Now let's start simple. So let's assume we want to build an agent based on the AIXI, um, model which performs a particular task. Let's start with something super simple like, I mean super simple, like playing chess, yeah, or Go or something. Yeah? Then-... you just, you know, the reward is, you know, winning the game is plus one, losing the game is minus one. Done. Uh, you apply this agent. If you have enough compute, you let it self-play, um, and it will learn the rules of the game. Will play perfect chess after some while. Problem solved, okay? So if you have more complicated problems, um, then, um, you may believe that you have the right reward, but it's not. So a nice cute example is elevator control that is also in Rich Sutton's book, which is a great book by the way. Um, so you control the elevator and you think, "Well, maybe the reward should be coupled to how long people wait in front of the elevator, you know. A long wait is bad."

   9. LFLex Fridman1:07:21

      Mm-hmm.

   10. MHMarcus Hutter1:07:22

       You program it and you do it, and what happens is, the elevator eagerly picks up all the people but never drops them off. (laughs)

   11. LFLex Fridman1:07:30

       (laughs) Yeah.

   12. MHMarcus Hutter1:07:31

       So then you realize, "Oh, maybe the time in the elevator also counts-"

   13. LFLex Fridman1:07:34

       Yeah.

   14. MHMarcus Hutter1:07:34

       ... so you minimize the sum, yeah?

   15. LFLex Fridman1:07:36

       Yeah.

   16. MHMarcus Hutter1:07:36

       And the elevator does that, but never picks up the people in the 10th floor and the top floor because, in expectation it's not worth it.

   17. LFLex Fridman1:07:42

       Yeah.

   18. MHMarcus Hutter1:07:42

       Just let them stay. (laughs)

   19. LFLex Fridman1:07:43

       Yeah. (laughs)

   20. MHMarcus Hutter1:07:46

       So, um, so even in apparently simple problems, you can make mistakes, yeah, and that's, um, um, what in, in more serious contexts, the AGI safety researchers consider. So now let's go back to, um, general agents. So assume we want to build an agent which is generally useful to humans, yes? We have a household robot, yeah, um, and it should do all kinds of tasks. So in this case, the human should give the reward on the fly. I mean, maybe it's pretrained in the factory and there, there's some sort of internal reward for, you know, the battery level or whatever, yeah? But, um, so it, you know, it does the dishes badly, you know, you punish the robot. It does it good, you reward the robot and then train it to a new task, kind of like a child, right? So, um, you need the human in the loop if you want a system which is useful to the human. And as long as this agent stays subhuman level, um, that should work reasonably well, um, apart from, you know, these examples. And it becomes critical if they become, you know, on a human level. It's the same as children. Small children, you have reasonably well under control. They become older, um, the reward technique doesn't work so well anymore. Um, so then finally, um, so this would be agents which are just, you could say, slaves to the humans, yeah?

   21. LFLex Fridman1:09:02

       Mm-hmm.

   22. MHMarcus Hutter1:09:02

       So if you are more ambitious and just say we want to build a new species of intelligent beings. We put them on a new planet and we want them to develop this planet or whatever. So we don't give them any reward. So what could we do? And you could try to, you know, come up with some reward functions like, you know, it should maintain itself, the robot. It should, um, maybe multiply, build more robots, right? Um, and, um, you know, maybe, well, all kinds of things which you find useful. But that's pretty hard, right? So, you know, what, what does self-maintenance mean, you know? What does it mean to build a copy? Should it be exact copy, an approximate copy? And so that's really hard. But, um, Laurent Assomme, also at DeepMind, uh, developed a beautiful model so it just took the axi model and coupled the rewards to information gain.

   23. LFLex Fridman1:09:55

       Mm-hmm. Yeah.

   24. MHMarcus Hutter1:09:55

       So he said the reward is proportional to how much the agent had learned about the world, and you can rigorously formally uniquely define that in terms of (inaudible) versions, okay? So if you put that in, you get a completely autonomous agent. And actually, interestingly for this agent, we can prove much stronger result than for the general agent, which is also nice. And if you let this agent loose, it will be, in a sense, the optimal scientist. It is absolutely curious to learn as much as possible about the world. And of course, it will also have a lot of instrumental goals, right? In order to learn, it needs to at least survive, right? A dead agent is not good for anything. So it needs to have self-preservation, and if it builds small helpers acquiring more, uh, information, it will do that, yeah? If exploration, space exploration or whatever is necessary, right, to gathering information, it develop it. So it has a lot of instrumental goals following on this information gain. And this agent is completely autonomous of us. No rewards necessary anymore.

   25. LFLex Fridman1:10:55

       Yeah. Of course, it could find a way to game the concept of information and get stuck in that, um, library that you mentioned beforehand with a, with a very large number of books.

   26. MHMarcus Hutter1:11:08

       The first agent had this problem. Um, it would get stuck, um, in front of an old TV screen which just had white noise.

   27. LFLex Fridman1:11:15

       Yeah, white noise. Yeah.

   28. MHMarcus Hutter1:11:15

       (laughs)

   29. LFLex Fridman1:11:16

       (laughs)

   30. MHMarcus Hutter1:11:16

       But, um, the second version can deal with at least, uh, stochasticity, um, well.
10. 1:12:14 – 1:13:32

    Reward function for human existence

    1. MHMarcus Hutter1:12:14

       better.

    2. LFLex Fridman1:12:14

       Well, okay. So if, uh, intelligent systems need to have this reward function, let me... You're an intelligent system currently passing the Turing test quite effectively.

    3. MHMarcus Hutter1:12:26

       (laughs)

    4. LFLex Fridman1:12:27

       What, uh, what's the reward function of, of our human intelligence existence? What's the reward function that Marcus Carter is operating under?

    5. MHMarcus Hutter1:12:37

       Okay, to the first question, the biological-... reward function is to survive and to spread-

    6. LFLex Fridman1:12:44

       Mm-hmm.

    7. MHMarcus Hutter1:12:44

       ... and very few humans sort of are able to overcome, uh, this biological reward function. Um, but we live in a, um, very nice world, uh, where we have lots of spare time and can still survive and spread. So we can develop, um, arbitrary other interests, which is quite interesting. Uh-

    8. LFLex Fridman1:13:03

       On top of that main one.

    9. MHMarcus Hutter1:13:04

       On, on top of that, yeah.

    10. LFLex Fridman1:13:05

        Yeah.

    11. MHMarcus Hutter1:13:06

        Um, but, um, the survival and spreading sort of is, um, I would say the, um, the goal or the reward function of humans, the, the, the core one.

    12. LFLex Fridman1:13:15

        I like how you avoided answering the second question, which a good intelligence system would.

    13. MHMarcus Hutter1:13:19

        (laughs)

    14. LFLex Fridman1:13:19

        So my, the, your own meaning of life and the reward function.

    15. MHMarcus Hutter1:13:24

        Uh, my own meaning of life and reward function is to find an AGI, to build it. (laughs)

    16. LFLex Fridman1:13:30

        (laughs) Beautifully put. Okay.

11. 1:13:32 – 1:16:07

    Bounded rationality

    1. LFLex Fridman1:13:32

       Let's dissect AIX even further. So, uh, one of the assumptions is kind of infinity keeps creeping up everywhere. Uh, so (laughs) which, uh, what are your thoughts on kind of bounded rationality and, uh, so the nature of our existence in intelligence systems is that we're operating always under constraints, under, you know, limited time, limited resources. How does that, how do you think about that within the AIXI framework, w- within trying to create an AGI system that operates under these constraints?

    2. MHMarcus Hutter1:14:06

       Yeah, that is one of the criticisms about AIXI, that it ignores computation and completely, and some people believe that intelligence is inherently tied towards bounded, um, resources.

    3. LFLex Fridman1:14:19

       What do you think on this one point?

    4. MHMarcus Hutter1:14:21

       Uh-

    5. LFLex Fridman1:14:21

       Do you think it's, uh, do you think the boundary resources are fundamental to intelligence?

    6. MHMarcus Hutter1:14:28

       I would say that an intelligence notion which ignores computational limits is extremely useful. A good intelligence notion which includes this res- resources would be even more useful, but we don't have that yet.

    7. LFLex Fridman1:14:42

       Mm-hmm.

    8. MHMarcus Hutter1:14:43

       Um, and so look at other fields outside of computer science. Computational aspects never play a fundamental role. You develop biological models for cells, something in physics. These theories, I mean, become more and more crazy and harder and harder to compute. Well, in the end, of course, we need to do something with this model, but that's more a nuisance than a feature. And, um, I'm sometimes wondering if artificial intelligence would not sit in a computer science department, but in a philosophy department, then this computational focus would be probably significantly less. I mean, think about the induction problem is more in the philosophy department. There's virtually no people who cares about, you know, how long it takes to compute the answer. That is completely secondary. Of course, once we have figured out the first problem, so intelligence without computational resources, then the next and very good question is, could we improve it by including computational resources? But nobody was able to do that so far in an even halfway satisfactory manner.

    9. LFLex Fridman1:15:49

       I like that, that, uh, in the long run, the right department to belong to is philosophy. (laughs) That's, uh, that's 'cause it's, it's actually quite a, a deep idea of, or even to, or at least to think about big picture philosophical questions, big picture questions w- even in the computer science department.

12. 1:16:07 – 1:18:01

    Approximation in AIXI

    1. LFLex Fridman1:16:07

       But you've mentioned approximation, sort of, uh, there's a lot of infinity, a lot of huge resources needed. Are there approximations to AIXI that, within the AIXI framework, that are useful or effective?

    2. MHMarcus Hutter1:16:19

       Yeah, we have developed a couple of approximations. Um, and, um, what we do there is that the Solomoff induction part, which was, you know, find the shortest program describing your data, we just replaced it by standard data compressors, right? And, um, the better compressors get, you know, the better this part will become. Uh, we focused on a particular compressor called context-free weighting which is, uh, pretty amazing, not so well-known. Uh, has beautiful theoretical properties, also works reasonably well in practice. So we use that for the approximation of the induction, um, and the learning and the prediction part. And, um, for the planning part, um, we essentially just took the ideas from a ComputerGo from 2006. It was Csaba Szepezvari, also now at DeepMind-

    3. LFLex Fridman1:17:11

       (laughs)

    4. MHMarcus Hutter1:17:11

       ... uh, who developed the so-called, um, UCT algorithm, Upper Confidence Bound for Trees algorithm, on top of the Monte Carlo tree search. So we approximate this planning part, um, by sampling. And, um, it's successful on some small toy problems. Um, we don't want to lose the generality, right?

    5. LFLex Fridman1:17:33

       Right.

    6. MHMarcus Hutter1:17:33

       And that's sort of the handicap, right? If you want to be general, you have to give up something. So, but this single agent was able to play, you know, small games like Kun Poker and Tic-Tac-Toe and, um, and, and even Pac-Man, um, um, into the same architecture, no change. The agent doesn't know the rules of the game, virtually nothing at all by self or by player with these environments.

    7. LFLex Fridman1:18:00

       So, um,

13. 1:18:01 – 1:21:51

    Godel machines

    1. LFLex Fridman1:18:01

       Jurgen Schmidhuber proposed something called, uh, Godel machines, which is a self-improving program that rewrites its own code. What... Sort of mathematically or philosophically, what's the relationship, in your eyes, if you're familiar with it, between AIXI and the Godel machines?

    2. MHMarcus Hutter1:18:18

       Yeah. Familiar with it. He developed it while I was in his lab. Yeah, so the Godel machine, uh, to explain it briefly, um, you give it a task. It could be a simple task, as you know, finding prime factors in numbers, right? You can formally write it down. There's a very slow algorithm to do that, just all, try all the factors, yeah? Or play chess, right? Optimally, you write the algorithm to minimax to the end of the game. So you write down what the Godel machine should do.... then it will take part of its resources to run this program, and other part of the resources to improve this program.

    3. LFLex Fridman1:18:53

       Mm-hmm.

    4. MHMarcus Hutter1:18:54

       And when it finds an improved version which provably computes the same answer, so that's the key part, yeah? It needs to prove by itself that this change of program still satisfies the original specification.

    5. LFLex Fridman1:19:08

       Mm-hmm.

    6. MHMarcus Hutter1:19:09

       And if it does so, then it replaces the original program by the improved program, and by definition, it does the same job, but just faster.

    7. LFLex Fridman1:19:16

       Mm-hmm.

    8. MHMarcus Hutter1:19:16

       Okay? And then, you know, it proves over it and over it, and it's, it's, it's developed in a way that, um, all parts of this Gödel machine can self-improve, but it stays provably consistent with the original specification. So, um, from this perspective, it has nothing to do with AIXI. But if you would now put AIXI as the starting axioms in, it would run AIXI. But, you know, that takes forever. But then if it finds a provable speed up of AIXI, it would replace it by this, and then this and this, and maybe eventually, it comes up with a model which is still the AIXI model. It cannot be ... I mean, uh, just for the, uh, knowledge of a reader, AIXI's imcomputable and I can prove that; therefore, there cannot be a computable exact, um, algorithm computers. There needs to be some approximations, and this is not dealt with the Gödel machine, so you have to do something about it. But there's the AIXI TL model which is finitely computable which we could put in.

    9. LFLex Fridman1:20:16

       Which part of AIXI is, um, uh, non-computable?

    10. MHMarcus Hutter1:20:19

        The Solomon of induction part.

    11. LFLex Fridman1:20:20

        The induction, okay. So-

    12. MHMarcus Hutter1:20:22

        But there is ways of getting computable approximations of the AIXI model, um, so then it's at least computable. It is still way beyond any resources anybody will ever have.

    13. LFLex Fridman1:20:33

        Yeah.

    14. MHMarcus Hutter1:20:33

        But then the Gödel machine could sort of improve it further and further in an exact way.

    15. LFLex Fridman1:20:37

        So wha- (laughs) So, so is it theoretically possible that the, the, the Gödel machine process could improve? Isn't, um, isn't, uh, isn't AIXI already optimal?

    16. MHMarcus Hutter1:20:51

        It is optimal in terms of the reward collected over its interaction cycles, but it takes infinite time to produce one action.

    17. LFLex Fridman1:21:03

        Mm.

    18. MHMarcus Hutter1:21:03

        And the world, you know, continues whether you want it or not, yeah? So the model is assuming here, then, oracle, which, you know, solve this problem, and then in the next 100 milliseconds or the reaction time you need gives the answer, then AIXI's optimal.

    19. LFLex Fridman1:21:17

        Oh, so-

    20. MHMarcus Hutter1:21:18

        It's optimal in sense of date, uh, also from learning efficiency and, um, data efficiency, but not in terms of computation time.

    21. LFLex Fridman1:21:25

        And then, the other Gödel machine, in theory, but probably not provably, could make it go faster.

    22. MHMarcus Hutter1:21:31

        Yes.

    23. LFLex Fridman1:21:31

        Okay. (laughs) This is interesting. Those two components are super interesting, the, sort of the, the perfect intelligence combined with, uh, self-improvement, sort of provable self-improvement in the sense you're always impro- like, uh, you're always getting the correct answer and you're improving.

    24. MHMarcus Hutter1:21:50

        Yep.

    25. LFLex Fridman1:21:50

        Beautiful ideas.

14. 1:21:51 – 1:27:15

    Consciousness

    1. LFLex Fridman1:21:51

       Okay, so you've also mentioned that, uh, different kinds of things in, in the chase of solving this, uh, reward f- sort of optimizing for the goal, interesting human things could emerge. So h- w- is there a place for consciousness within AIXI?

    2. MHMarcus Hutter1:22:08

       (laughs)

    3. LFLex Fridman1:22:10

       Where, where does, uh ... And maybe you can comment because I, I suppose we humans are just another instantiation of AIXI agents, only we seem to have consciousness.

    4. MHMarcus Hutter1:22:20

       You say humans are an instantiation of an AIXI agent?

    5. LFLex Fridman1:22:23

       Yes.

    6. MHMarcus Hutter1:22:23

       Well, that would be amazing, but I think that's-

    7. LFLex Fridman1:22:26

       It's not, right?

    8. MHMarcus Hutter1:22:27

       ... a dream for the smartest and most rational humans, I think. (laughs) Maybe we are very crude approximations. (laughs)

    9. LFLex Fridman1:22:33

       Interesting. I, I mean, I tend to believe, again, I'm Russian, so, um, I tend to believe our flaws are part of the optimal. So the, we s- tend to laugh off and criticize our flaws, and I, I tend to think that that's actually close to an optimal behavior.

    10. MHMarcus Hutter1:22:50

        Well, some flaws, if you think more carefully about it, are actually not flaws, yeah? But I think there are still enough flaws. (laughs)

    11. LFLex Fridman1:22:59

        I don't know.

    12. MHMarcus Hutter1:22:59

        But-

    13. LFLex Fridman1:22:59

        It's, it's unclear. Um, as a student of history, I think all the suffering that we've en- endured as a civilization, it's possible that that's the optimal amount of suffering we need to endure to minimize long-term suffering. (laughs)

    14. MHMarcus Hutter1:23:14

        That's- That's your Russian background, I think. (laughs)

    15. LFLex Fridman1:23:17

        Yes, that's, that's the Russian back... Whether w- humans are or not instantiations of an AIXI agent, do you think there is a consciousness of something that could emerge in the computational form or framework like AIXI?

    16. MHMarcus Hutter1:23:29

        Let me also ask you a question. Do you think I'm conscious?

    17. LFLex Fridman1:23:33

        (laughs) Uh, yeah, that's a good question. You, you're, you're-

    18. MHMarcus Hutter1:23:39

        (laughing)

    19. LFLex Fridman1:23:40

        (laughing) That, that, uh, that tie is confusing me. But, uh, I think so.

    20. MHMarcus Hutter1:23:44

        (laughs)

    21. LFLex Fridman1:23:44

        I think so.

    22. MHMarcus Hutter1:23:44

        You think it makes me unconscious because it strangles me, or?

    23. LFLex Fridman1:23:47

        If, if an agent were to solve the imitation game posed by Turing, I think they would be dressed similarly to you. That because there's, um, there's a kind of flamboyant interesting complex behavior pattern that sells that you're human and you're conscious. But, uh, why do you ask?

    24. MHMarcus Hutter1:24:06

        Was it a yes or was it a no?

    25. LFLex Fridman1:24:08

        Yes, I think you're-

    26. MHMarcus Hutter1:24:08

        Yes. (laughing)

    27. LFLex Fridman1:24:10

        (laughing) I thi- I think you're, uh, conscious, yes.

    28. MHMarcus Hutter1:24:12

        Yeah. So and you explained sort of somehow why. Um, but you infer that from my behavior, right?

    29. LFLex Fridman1:24:18

        Yes.

    30. MHMarcus Hutter1:24:18

        You can never be sure about that. And I think the same thing will happen with any intelligent agent we develop. If it behaves in a way sufficiently close to humans, or maybe even not humans. I mean, you know, maybe a dog is also sometimes a little bit self-conscious, right? So, so if it behaves in a way, um, where we attribute typically consciousness, we would attribute consciousness to these intelligent systems, and, you know, AIXI Pro in particular. That, of course, doesn't answer the question whether it's really conscious, and that's the, you know, the big, hard problem with consciousness, you know. Maybe I am a zombie. I mean, not the movie Zombie, but the philosophical zombie.

Episode duration: 1:39:55