Jeff Hawkins: Thousand Brains Theory of Intelligence | Lex Fridman Podcast #25

1. 0:00 – 15:00

   The following is a…

   1. LFLex Fridman0:00

      The following is a conversation with Jeff Hawkins. He's the founder of the Redwood Center for Theoretical Neuroscience in 2002, and Pneumenta in 2005. In his 2004 book, titled On Intelligence, and in the research before and after, he and his team have worked to reverse engineer the neocortex and propose artificial intelligence architectures, approaches, and ideas that are inspired by the human brain. These ideas include hierarchical temporal memory, HTM, from 2004, and new work, the Thousands Brains Theory of Intelligence from 2017, '18, and '19. Jeff's ideas have been an inspiration to many who have looked for progress beyond the current machine learning approaches, but they have also received criticism for lacking a body of empirical evidence supporting the models. This is always a challenge when seeking more than small incremental steps forward in AI. Jeff is a brilliant mind and many of the ideas he has developed and aggregated from neuroscience are worth understanding and thinking about. There are limits to deep learning as it is currently defined. Forward progress in AI is shrouded in mystery. My hope is that conversations like this can help provide an inspiring spark for new ideas. This is the Artificial Intelligence podcast. If you enjoy it, subscribe on YouTube, iTunes, or simply connect with me on Twitter at Lex Fridman, spelled F-R-I-D. And now here's my conversation with Jeff Hawkins. Are you more interested in understanding the human brain or in creating artificial systems that have many of the same qualities but don't necessarily require that you actually understand the underpinning workings of our mind?

   2. JHJeff Hawkins1:41

      So, there's a clear answer to that question: My primary interest is understanding the human brain. No question about it. But, um, I also firmly believe that we will not be able to create fully intelligent machines until we understand how the human brain works. So I don't see those as separate problems. Um, I think there's limits to what can be done with machine intelligence if you don't understand the principles by which the brain works, and so I actually believe that studying the brain is actually the frast- the fastest way (laughs) to get to machine intelligence.

   3. LFLex Fridman2:12

      And within that, let me ask the impossible question: How do you not define, but at least think about what it means to be intelligent?

   4. JHJeff Hawkins2:19

      So, I didn't try to answer that question first. We said, "Let's just talk about how the brain works and let's figure out how c- certain parts of the brain," mostly the neocortex, but some other parts too. The parts of the brain most associated with intelligence. And let's discover the principles by how they work, 'cause i- i- intelligence isn't just like some mechanism and it's not just some capabilities. It's like, okay, we don't even have, know where to begin on this stuff. And so now that we've made a lot of progress on this, after we've made a lot of progress on how the neocortex works, and we can talk about that, I now have a very good idea what's gonna be required to make intelligent machines. I g- I can tell you today, we know some of the things are gonna be necessary, I believe, to create intelligent machines.

   5. LFLex Fridman3:03

      Well, so, we'll, we'll get there. We'll get to the neocortex and some of the theories of how the whole thing works, and you're saying as we understand more and more, uh, about the neocortex, about our own human mind, we'll be able to start to more specifically define what it means to be intelligent. It's not useful to really talk about that until-

   6. JHJeff Hawkins3:22

      I don't know if it's not useful. You know, look, th- there's a long history of AI, as you know.

   7. LFLex Fridman3:25

      Right.

   8. JHJeff Hawkins3:26

      And there's been different approaches taken to it. And who knows? Maybe they're all useful.

   9. LFLex Fridman3:31

      Right.

   10. JHJeff Hawkins3:32

       Right? So-

   11. LFLex Fridman3:32

       In the end.

   12. JHJeff Hawkins3:32

       ... uh, you know, the good old-fashioned AI, the, uh, expert systems, uh, current convolutional neural networks, they all have their utility. They all have a value in the world. Uh, but I would think almost everyone agree that none of them are really intelligent in, in a sort of a deep way that, that humans are. And so, um, it's, it's just the question is how do you get from where those systems were or are today-

   13. LFLex Fridman3:56

       Mm-hmm.

   14. JHJeff Hawkins3:56

       ... to where a lot of people think we're gonna go?

   15. LFLex Fridman3:58

       Right.

   16. JHJeff Hawkins3:59

       And there's a big, big gap there, a huge gap. And I think the quickest way of, of bridging that gap is to figure out how the brain does that. And then we can sit back and look and say, "Oh, which of these principles that the brain works on are necessary and which ones are not?"

   17. LFLex Fridman4:14

       Right.

   18. JHJeff Hawkins4:15

       Clearly we don't have to build this in, in intelligent machines aren't gonna be built out of, uh, um, you know, organic living cells. Um, but there's a lot of stuff that goes on in the brain that's gonna be necessary.

   19. LFLex Fridman4:25

       So let me ask maybe before we get into the fun details, uh, let me ask maybe a depressing or difficult question. D- do you think it's possible that we will never be able to understand how our brain works, that maybe there's aspects to the human mind, like, we ourselves cannot introspectively get to the core, that there's a wall you eventually hit?

   20. JHJeff Hawkins4:48

       Yeah. Yeah. I don't believe that's the case. I have never believed that's the case. There's n- not been a single thing we've ever, humans have ever put their minds to that we've said, "Oh, we reached the wall. We can't go any further." It just people keep saying that. People used to believe that about life, you know, elan vital, right? There's like, what's the difference between living matter and nonliving matter? Something special we never understand. We no longer think that. So there's, there's no, uh, historical evidence to suggest this is the case, and I just never even consider that's a possibility. I would also say, uh, today, uh, we understand so much about the neocortex, we've made tremendous progress in the last few years, that I no longer think of it as, um, uh, uh, an open question. Um, the answers are very clear to me. Uh, the pieces we know, we don't know are clear to me, but the framework is all there and, and it's like, oh, okay, we're gonna be able to do this. Uh, this is not a problem anymore, just takes time and effort, but it- there's no mystery, uh, big mystery anymore.

   21. LFLex Fridman5:44

       So then let's get in- into it for, for people like myself who are not very well versed in the human brain, except my own.

   22. JHJeff Hawkins5:54

       (laughs)

   23. LFLex Fridman5:54

       Uh, can you describe to me at the highest level what are the different parts of the human brain, and then zooming in on the neocortex, the parts of the neocortex and so on-

   24. JHJeff Hawkins6:04

       Yeah.

   25. LFLex Fridman6:04

       ... a, a quick overview.

   26. JHJeff Hawkins6:05

       Yeah, sure.A human brain, uh, we can divide it roughly into two parts. There's the old parts, lots of pieces, and then there's the new part. The new part is the neocortex. Um, it's new because it didn't exist before mammals. Only mammals have a neocortex, and in humans, in primates, it's very large. In the human brain, the neocortex occupies about 70% to 75% of the volume of the brain. It's huge. And the old parts of the brain are, are... there's lots of pieces there. There's a spinal cord and there's the brain stem and the cerebellum and the different parts of the basal ganglia and so on. In the old parts of the brain, you have autonomic regulation, like breathing and heart rate. You have basic behaviors, so like walking and running are controlled by the old parts of the brain. All the emotional centers of the brain are in the old part of the brain, so when you feel anger or hungry, lust or things like that, those are all in the old parts of the brain. And, uh, and we associate with the neocortex all the things we think about as sort of high-level perception and cognitive functions. Anything from seeing and hearing and touching things, to language to mathematics and engineering and science and so on. Those are all associated with the neocortex, and they're certainly correlated. Uh, our abilities in those regards are correlated with the relative size of our neocortex compared to other mammals. So that's like the rough division. And you obviously can't understand the neocortex completely isolated, but you can understand a lot of it with just a few interfaces to the old parts of the brain, and so it, it gives you, um, a system to study. The other remarkable thing about the neocortex, uh, compared to the old parts of the brain, is the neocortex is extremely uniform. It's not visibly or anatomically or, uh, it's very, it's like a sh- I always like to say it's like the size of a dinner napkin, about two and a half millimeters thick, and it looks remarkably the same everywhere. Everywhere you look in that two and a half millimeters is this detailed architecture, and it looks remarkably the same everywhere. And that's across species, a mouse versus a cat and a dog and a human. Where if you look at the old parts of the brain, there's lots of little pieces do specific things. So it's, it's like the old parts of a brain involved, like, this is the part that controls heart rate, and this is the part that controls this, and this is this kind of thing, and that's this kind of thing. And these evolved for eons, a long, long time, and they have their specific functions, and all of a sudden, mammals come along and they got this thing called the neocortex, and it got large by just replicating the same thing over and over and over again. This is like, wow, this is incredible. Um, so all the evidence we have, um, and this is an idea that was first, uh, articulated, um, in a very cogent and beautiful argument by a guy named Vernon Mountcastle in 1978, I think it was, um, that the, the neocortex all works on the same principle. So language, hearing, touch, vision, engineering, all these things, are basically underlying or all built in the same computational substrate. They're really all the same problem.

   27. LFLex Fridman9:11

       So at the low level, the building blocks all look similar?

   28. JHJeff Hawkins9:14

       Yeah, and they're not even that low level. We're not talking about like, like neurons. We're talking about this very complex circuit that exists throughout the neocortex is remarkably similar. It is, it's like, yes, you see variations of it here and there, more of this cell, less, and that's not all, and so on. But, uh, what Mountcastle argued was, he says, you know, if you take a section of neocortex, why is one a visual area and one is a auditory area? Or why is... And his answer was, it's because one is connected to eyes and one is connected to ears.

   29. LFLex Fridman9:45

       Literally, you mean just it's most closest in terms of number of connections to-

   30. JHJeff Hawkins9:49

       Literally-
2. 15:00 – 30:00

   (laughs) …

   1. JHJeff Hawkins15:00

      would be changing on your ears about every 10 milliseconds or so, you'd have a change. Just constant flow. When you look at the world, your eyes are moving constantly, three to five times a second, and the inputs

   2. NANarrator15:10

      (laughs)

   3. JHJeff Hawkins15:10

      ... completely. If I were to touch something like a coffee cup, the- as I move my fingers, the input changes. So this idea that the brain works on time-changing patterns is almost completely, or was almost completely missing from a lot of the basic theories, like fears of vision and so on. It's like, oh no, we're gonna put this image in front of you and flash it and say, "What is it?"

   4. LFLex Fridman15:29

      Right.

   5. JHJeff Hawkins15:29

      Uh, convolutional neural networks work that way today, right? You know, classify this picture. But that's not what vision is like. Vision is this sort of crazy time-based pattern that's going all over the place, and so is touch and so is hearing.

   6. LFLex Fridman15:41

      Mm-hmm.

   7. JHJeff Hawkins15:41

      So the first part of, uh, hierarchical temporal memory was the temporal part.

   8. LFLex Fridman15:45

      Right.

   9. JHJeff Hawkins15:45

      It's, it's to say (laughs) you, you won't understand the brain, nor will you understand intelligent machines unless you're dealing with time-based patterns. The second thing was, uh, the memory component of it was, is to say that, um, we aren't just processing input. We learn a model of the world, that's- and I- and the memory stands for that model. We have to... The point of the brain, the part of the neocortex, it learns a model of the world. We have to store things that, that, that are experiences in a form that leads to a model of the world so we-

   10. LFLex Fridman16:14

       Mm-hmm.

   11. JHJeff Hawkins16:14

       ... can move around the world, we can pick things up and do things and navigate and know how what's going on. So that's, that's what the memory referred to and many people just, they were thinking about like certain processes without memory at all (laughs) . They're just like processing things. And then finally, the hierarchical component was a reflection to, uh, that the neocortex, although it's this uniform sheet of cells, um, different parts of it project to other parts, which project to other parts, and there is a sort of a rough hierarchy in terms of them. So the hierarchical temporal memory is just saying, look, we should be thinking about the brain as time-based, you know, model memory based-

   12. LFLex Fridman16:51

       Mm-hmm.

   13. JHJeff Hawkins16:52

       ... and hierarchical processing. Um, and, and that was a placeholder for a bunch of components that we would then plug into that. Uh, we still believe all those things I just said, but we now know so much more that, um, I'm stopping to use the word hierarchical temporal memory-

   14. LFLex Fridman17:08

       Mm-hmm.

   15. JHJeff Hawkins17:08

       ... because it's, it's insufficient to capture the stuff we know. So again, it's not incorrect, but it's... I now know more and I would rather describe it, uh, more accurately.

   16. LFLex Fridman17:16

       Yeah, so you're basically... We could think of HTM as, uh, emphasizing that there's three aspects of intelligence that are important to think about, whatever the, whatever the eventual theory it converges to.

   17. JHJeff Hawkins17:28

       Yeah.

   18. LFLex Fridman17:28

       So in terms of time, how do you think of, uh, nature of time across different time scales? You mentioned-

   19. JHJeff Hawkins17:35

       Yeah.

   20. LFLex Fridman17:35

       ... things changing, uh, sen- sensory inputs changing every 10, 20-

   21. JHJeff Hawkins17:38

       Yeah.

   22. LFLex Fridman17:39

       ... milliseconds.

   23. JHJeff Hawkins17:39

       Yeah.

   24. LFLex Fridman17:39

       What about every few minutes, every few-

   25. JHJeff Hawkins17:41

       Yeah.

   26. LFLex Fridman17:41

       ... months and years?

   27. JHJeff Hawkins17:42

       Well, if you think about a neuroscience problem, the brain problem-Neurons themselves, uh, can stay active for certain periods of time. Uh, they, there are parts of the brain where they stay active for minutes, you know, so you could hold a certain perception, uh, or an activity, um, uh, for, for a certain period of time. But not f- most of them don't last that long. Um, and so if you think about your thoughts are the activity of neurons, if you're gonna wanna involve something that happened a long time ago, I mean, even just this morning, for example, um, the neurons haven't been active throughout that time. So you have to store that. So if I ask you, "What did you have for breakfast today?" that is memory. That is, you've built that into your model of the world now. You remember that. And that memory is in the, uh, in the synapses, is basically in the formation of synapses. And so, um, uh, it's, it, you're sliding into what, you know, we used to do on timescales.

   28. LFLex Fridman18:36

       Mm-hmm.

   29. JHJeff Hawkins18:36

       There's time scales at which we are, like, understanding my language and moving about and seeing things rapidly and over time. That's the timescales of activities of neurons. But if you want to get in longer timescales, then it's more memory, and we have to invoke those memories to say, "Oh, yes, well, now I can remember what I had for breakfast because I stored that someplace." Um, I may forget it tomorrow, but I'd store it for d- for now.

   30. LFLex Fridman18:58

       So does memory also need to have... So the hierarchical aspect of reality is not just about concepts, it's also about time? Do you think of it that way?
3. 30:00 – 45:00

   Sure. In which, which,…

   1. JHJeff Hawkins30:00

      time to get the neocortex. And we've only had the neocortex for a few 100,000 years. So that's like nothing, (laughs) okay? So is it likely? Well, it certainly isn't something that happened right away o- on Earth, uh, and there were multiple steps to get there, so I would say it's probably not gonna be something that would happen instantaneously on other planets that might have life. Uh, it might take several billion years on average. Um, is it likely? I don't know, but you'd have to survive for several billion years to find out, probably. Um, is it a big leap? Yeah, I think it's, um, uh, it is a, a, a qualitative difference in all other evolutionary steps. Uh, I can try to describe that if you'd like.

   2. LFLex Fridman30:39

      Sure. In which, which, in which way?

   3. JHJeff Hawkins30:41

      Uh, yeah, I can tell you how. Um, pretty much, uh, I'll, let's start with a little preface. Many of the things that humans are able to do do not have obvious, um, uh, survival advantages, uh, precedent. Uh, you know, w- we c- create music, is that, is there a really survival advantage to that? Uh, maybe, maybe not. Uh, what about mathematics? Is there a real survival advantage to mathematics? Well, maybe. Uh, well, uh, you could stretch it, you could try to figure these things out, right? Um, but up, but most of evolutionary history, everything had immediate survival advantages to them. So I'll tell you a story, um, which I like, may or may not be true, um, but the story goes as follows. Um, organisms have been evolving for since the beginning of life here on Earth, um, and adding this sort of complexity onto that, and this sort of complexity onto that. And the brain itself is evolved this way. In fact-

   4. LFLex Fridman31:40

      Mm-hmm.

   5. JHJeff Hawkins31:40

      ... there's an old part, an older part, an older, older part to the brain that kind of just keeps calming on new things and we keep adding capabilities. When we got to the neocortex, initially it had, uh, a very clear survival advantage in that it produced better vision and better hearing and better touch and maybe, and so on. But what, what I think happens is, is that evolution discovered, it took, it took a mechanism, um, and this is in our theor- recent theories, but it took a mechanism that evolved a long time ago for navigating in the world, for knowing where you are. These are, um, the so-called grid cells and place cells of an old part of the brain. And it, it took that mechanism for building maps of the world, um, and knowing where you are on those maps and how to navigate those maps, and turns it into a sort of a, a slimmed down, idealized version of it.

   6. LFLex Fridman32:28

      Mm-hmm.

   7. JHJeff Hawkins32:29

      And that idealized version could now apply to building maps of other things. Maps of coffee cups and maps of phones, maps of, you know-

   8. LFLex Fridman32:36

      Concepts almost.

   9. JHJeff Hawkins32:37

      ... mathematics, and concepts, yes. And, and not just almost, exactly.

   10. LFLex Fridman32:40

       Right.

   11. JHJeff Hawkins32:40

       And, and so you, and it just started replicating this stuff, right? It's just like more and more and more. And so we went from being sort of, uh, dedicated purpose neural hardware-

   12. LFLex Fridman32:49

       Mm-hmm.

   13. JHJeff Hawkins32:50

       ... to solve certain problems that are important to survival, to a general purpose neural hardware that could be applied to all problems, and now it's, it's, uh, it's escaped the orbit of survival. It's, we are now able to apply it to things which we find enjoyment, um, you know, but aren't really clearly, um, survival characteristics. Uh, and that seems to only have happened in humans to, to, to the large extent. Um, and so that's what's going on. We're, we sort of have, um, we've sort of escaped the gravity of evolutionary pressure in some sense in the neocortex, and it now does things which are not, that are really interesting, discovering models of the universe, which may not really help us. It doesn't matter.

   14. LFLex Fridman33:36

       Yeah.

   15. JHJeff Hawkins33:37

       How does it help us surviving knowing that there might be multiverses, or th- there might be, you know, the age of the universe, or wh- how do, you know, various stellar things occur? It doesn't really help us survive at all, but we enjoy it, and that's what happened.

   16. LFLex Fridman33:50

       Or at least not in the obvious way perhaps it, it is required if, if you look at the entire universe in an evolutionary way, it's required for us to do interplanetary travel and therefore survive past our own sun-

   17. JHJeff Hawkins34:03

       Yeah.

   18. LFLex Fridman34:03

       ... but you know, let's not get too crazy. (laughs)

   19. JHJeff Hawkins34:04

       Yeah, but, but, you know, evolution works at one timeframe and-

   20. LFLex Fridman34:07

       Yes, yeah.

   21. JHJeff Hawkins34:07

       ... well, it's, it, it's survival. If you think of survival of the phenotype, survival of the individual-

   22. LFLex Fridman34:12

       Right, exactly.

   23. JHJeff Hawkins34:13

       ... it, it, this, that, what you're talking about there, it spans well beyond that.

   24. LFLex Fridman34:16

       Yeah.

   25. JHJeff Hawkins34:16

       So there's no genetic... I'm not transferring any genetic, um, traits to my children, uh, that are gonna help them survive better-

   26. LFLex Fridman34:25

       Right.

   27. JHJeff Hawkins34:25

       ... on Mars.

   28. LFLex Fridman34:25

       Right. Totally different mechanism, that's right.

   29. JHJeff Hawkins34:27

       Yeah.

   30. LFLex Fridman34:28

       So let's get into the, the new, as, as you've mentioned this i- idea of the... I don't know if you have a nice name, thousand...
4. 45:00 – 1:00:00

   Yeah, so there, there's,…

   1. JHJeff Hawkins45:00

      talked a little bit about that. I've been working on it more since. I have, uh, more, better ideas about it now. Um, I'm sitting here very confident that that's what's happening, and I can give you some examples to help you think about that. Um, it's not that we understand it completely, but I understand it better than I've described it in any paper so far. So, uh, but we did put that idea out there, says, okay, this is, it's, it's, it's a good place to start, you know? And the evidence would suggest it's how it's happening, and then we can start tackling that problem one piece at a time. Like, what does it mean to do high level thought? What does it mean to do language? How would that fit into a reference frame framework?

   2. LFLex Fridman45:34

      Yeah, so there, there's, uh, I don't know if you could tell me if there's a connection, but there's an app called Anki that helps you remember different concepts, and they, they talk about like a memory palace that helps you remember completely random concepts by sort of trying to put them in a physical space in your mind.

   3. JHJeff Hawkins45:51

      Yeah.

   4. LFLex Fridman45:51

      And putting them next to each other.

   5. JHJeff Hawkins45:52

      It's called the method of loci.

   6. LFLex Fridman45:53

      Loci, yeah.

   7. JHJeff Hawkins45:54

      Yes, it is.

   8. LFLex Fridman45:54

      For some reason, that seems to work really well.

   9. JHJeff Hawkins45:56

      Yeah.

   10. LFLex Fridman45:57

       Uh, now that's a very narrow kind of application of just remembering some facts and so on.

   11. JHJeff Hawkins46:00

       But that's not, but that's a very, very telling one. Okay? So...

   12. LFLex Fridman46:03

       Yes, exactly. So this seems like you're describing a mechanism why this seems to work.

   13. JHJeff Hawkins46:09

       Yes. So, so basically the way, what we think is going on is all c- things you know, all concepts, all ideas, words, everything you know, um, are stored in reference frames. And, and so, uh, if you want to remember something, you have to basically navigate through a reference frame the same way a rat navigates through a maze and the same way my finger ra- navigates to this coffee cup. You are moving through some space. And so what you can, if you have a random list of things you were asked to remember, by assigning them to a reference frame you already know very well, let's say your house, right? An idea of the method of loci is you can say, "Okay, in my lobby, I'm gonna put this thing, and then, and then the bedroom, I put this one. I go down the hall, I put this thing." And then you want to recall those facts or recall those things, you just walk mentally, you walk through your house. You're mentally moving through a reference frame that you already had, and that tells you, uh, there's two things that are really important about that. It tells us the brain prefers to store things in reference frames, and that the method of recalling things or thinking, if you will, is to move mentally through those reference frames. You could move physically through some reference frames, like I could physically move through the reference frame of this coffee cup. I can also mentally move through the reference frame of the coffee cup, imagining me touching it. But I can also mentally move my house, and, and so now we can ask yourself, are all concepts stored this way? There was some recent, uh, research, um, using human subjects and fMRI, and I'm gonna apologize for not knowing the name of the scientist who did this. Um, but, um, what they did is they, they put humans in this fMRI machine, which is one of these imaging machines, and they w- they gave the humans tasks to think about birds. So they had different types of birds and birds that looked big and small and long necks and long legs, things like that. And what they could tell from the fMRI, it was a very clever experiment, um, they get to tell when humans were thinking about the birds, that the birds, the, the knowledge of birds was arranged in a reference frame similar to the, the ones that are used when you navigate in a room. That, these are called grid cells, and there, there are grid cell-like patterns of activity in the neocortex when they do this. So that-... it's a very clever experiment, you know. And what it basically says that even when you're thinking about something abstract and you're not really thinking about it as a reference frame, it tells us the brain is actually using a reference frame. And it's using the same neural mechanisms. These grid cells are the basic same neural mechanisms that we- we propose that grid cells, which exist in the, in the old part of the brain, the entorhinal cortex, that that mechanism is now... Similar mechanism is used throughout the neocortex. It's the same. Nature preserved this interesting way of creating reference frames. And so now they have empirical evidence that, uh, when you think about concepts like birds, that you're using reference frames that are built on grid cells. So this, that's similar to the method of loci, but in this case, the birds are related so it makes-

   14. LFLex Fridman48:57

       Hm.

   15. JHJeff Hawkins48:57

       ... they create their own reference frame which is consistent with bird space. And when you think about something, you go through that. You, I, you can make these same examples. Let's take a math- mathematics, right? Let's say you wanna prove a conjecture, okay? What is a conjecture? A conjecture is a, uh, a statement you believe to be true, but you haven't proven it. And so it might be an equation. "I, I, I want to show that this is equal to that." And you have a p- you have some places you start with. You say, "Well, I know this is true and I know this is true, and I think that maybe to get to the final proof, I need to go through some intermediate results."

   16. LFLex Fridman49:28

       Mm-hmm.

   17. JHJeff Hawkins49:28

       What I believe is happening is literally these equations or these points are assigned to a reference frame, a mathematical reference frame. And when you do mathematical operations, a simple one might be multiply or divide, but you might be a Laplace transform or something else. That is like a movement in the reference frame of, of the math. And so you're literally trying to discover a path from one location to another location in a space of mathematics. And if you can get to these intermediate results, then you know y- your map is pretty good and you know you're using the right operations. Uh, much of what we think about as solving hard problems is designing the correct reference frame for that problem, figuring out how to organize the information, and what behaviors I want to use in that space to get me there. (laughs)

   18. LFLex Fridman50:16

       Yeah, so if you dig in o- on the idea of this reference frame, whether it's the math you start, a set of axioms to try to get to proving the conjecture, uh, can you try to describe, maybe take a step back, w- how you think of the reference frame in that context? Is, is it the reference frame that the axioms are happy in? Is it the reference frame that might contain everything? Is it a changing thing-

   19. JHJeff Hawkins50:40

       So the, it, it, it-

   20. LFLex Fridman50:41

       ... as you...

   21. JHJeff Hawkins50:41

       You have many, many reference frames. I mean, in fact, the way the theory, the thousand-brain theory of intelligence does it, every single thing in the world has its own reference frame. So, um, every word has its own reference frame. And we can talk about this ma- uh, the mathematics workout, this is no problem for neurons to do this.

   22. LFLex Fridman50:56

       But how many reference frames does a coffee cup have?

   23. JHJeff Hawkins50:58

       Well-

   24. LFLex Fridman50:59

       Like, the, it's on a table.

   25. JHJeff Hawkins51:00

       Remember, uh, uh, let's say you ask how many reference frames could the column in my finger that's touching the coffee cup have? Because there are many, many cop-

   26. LFLex Fridman51:09

       Okay.

   27. JHJeff Hawkins51:09

       There are many, many models of the coffee cup. So the coffee... There is no one model of a coffee cup. There are many models of a coffee cup. And you could say, "Well, how many different things can my finger learn?" is this, is this the question you wanna ask.

   28. LFLex Fridman51:19

       Mm-hmm.

   29. JHJeff Hawkins51:19

       Imagine, I say every concept, every idea, everything you've ever know about that you can say, "I know that thing," uh, has a, a reference frame associated with it. And what we do when we build composite objects, we com- we up- assign reference frames to point another reference frame. So my coffee cup has multiple components to it. It's got a rim, it's got a cylinder, it's got a handle. Um, and those things have their own reference frames and they're assigned to a master reference frame, which is called this cup. And now I have this Numenta logo on it. Well, that's something that exists elsewhere in the world.

   30. LFLex Fridman51:50

       Mm-hmm.
5. 1:00:00 – 1:10:30

   Mm-hmm. …

   1. JHJeff Hawkins1:00:00

      Or one way to think about it, is we have all these models of the world, okay? And we model, we model everything. And as I said earlier, I, I kind of snuck it in there. Our models are actually, we, we build composite structures, so, uh, every object is composed of other objects, which are composed of other objects, and they become members of other objects.

   2. LFLex Fridman1:00:18

      Mm-hmm.

   3. JHJeff Hawkins1:00:18

      So this room has chairs and a table and a room and walls and so on. Now we can just r- arrange them in a c- these things in a certain way and go, "Oh, that's the Numenta conference room." So-... so, and what we do is when we go around the world and we experience the world, we, if I walk into a room for example, the first thing I do is I could say, "Oh, I'm in this room. Do I recognize the room?" Then I could say, "Oh, look, there's a, there's a table here." Um, and I, by attending to the table, I'm then assigning this table in the context of the room. Then I could, "Oh, on the table, there's a coffee cup. Oh, and on the table, there's a logo. And in the logo, there's the word Numenta. Oh, and look in the logo, there's a, the letter E. Oh, and look, it has an unusual serif." Uh, doesn't actually, but, uh, pretend it does. (laughs) So the point is you, your a- a- attention is kind of drilling deep in and out of these nested structures. And I can pop back up and I can pop back down. I can pop back up and I can pop back down. So I, when I attend to the coffee cup, I haven't lost the context of everything else. But, uh, but it's sort of, there's this sort of nested structure.

   4. LFLex Fridman1:01:18

      So the attention filters the reference frame information for that particular period of time?

   5. JHJeff Hawkins1:01:24

      It ... Yes. It basically, moment to moment, you attend to sub-components-

   6. LFLex Fridman1:01:28

      Mm-hmm.

   7. JHJeff Hawkins1:01:28

      ... and then you can attend the sub-components, the sub-components, and so on.

   8. LFLex Fridman1:01:30

      And you can move up and down the hierarchy.

   9. JHJeff Hawkins1:01:31

      You can move up and down, and we do that all the time. You're not even, uh, uh, uh, now that I'm aware of it, I'm very conscious of it. But until-

   10. LFLex Fridman1:01:37

       (laughs)

   11. JHJeff Hawkins1:01:37

       (laughs) But, but most people don't, don't even think about this. You know, you don't, you just walk in a room and you don't say, "Oh, I looked at the chair and I looked at the board, and I looked at that word on the board, and I looked over here. What's going on?"

   12. LFLex Fridman1:01:46

       Mm-hmm.

   13. JHJeff Hawkins1:01:46

       Right?

   14. LFLex Fridman1:01:47

       So what percentage of your day are you deeply aware (laughs) of this, and what part can you actually relax and just be, uh, Jeff?

   15. JHJeff Hawkins1:01:52

       Me personally, like my personal day?

   16. LFLex Fridman1:01:54

       Yeah. (laughs)

   17. JHJeff Hawkins1:01:55

       Uh, unfortunately, I'm afflicted with too much of the former. Um ...

   18. LFLex Fridman1:01:59

       (laughs)

   19. JHJeff Hawkins1:01:59

       (laughs) I, I think-

   20. LFLex Fridman1:02:01

       Well, unfortunately or unfortunately? Okay.

   21. JHJeff Hawkins1:02:02

       Yeah. So I think-

   22. LFLex Fridman1:02:03

       You don't think it's useful?

   23. JHJeff Hawkins1:02:04

       Oh, I think it's useful, totally useful. I think about this stuff almost all the time. And, and I'm, uh, one of my primary ways of thinking is when I'm in sleep at night, I always wake up in the middle of the night, and then I stay awake for at least an hour with my eyes shut in sort of a half-sleep state thinking about these things, and I come up with answers to problems very often in that sort of half-sleeping state. I think about it on my bike ride, I think about it on walks. I'm just constantly thinking about this. I have to almost, uh, schedule time to not think about this stuff because it's very, it- it's mentally taxing.

   24. LFLex Fridman1:02:37

       Are you, uh, when you're thinking about this stuff, are you, are you thinking introspectively, like almost taking a step outside of yourself and trying to figure out what is your mind doing right now?

   25. JHJeff Hawkins1:02:45

       I do that all the time, but that's not all I do. I'm constantly observing myself. So as soon as I started thinking about grid cells, for example, uh, and getting into that, I started saying, "Oh, well, grid cells can have my place a sense in the world." You know, that's where you know where you are. And, and it's interesting, you know, we always have a sense of where we are, unless we're lost. And so I started at night when I got up to go to the bathroom, I would start trying to do it completely with my eyes closed all the time, and I would test my sense of grid cells.

   26. LFLex Fridman1:03:10

       (laughs)

   27. JHJeff Hawkins1:03:10

       I would, I would walk f- you know, five feet and say, "Okay, I think I'm here. Am I really there? How, what's my error?"

   28. LFLex Fridman1:03:15

       Yeah.

   29. JHJeff Hawkins1:03:15

       And then I would calculate my error again and see how the errors c- accumulate. So even something as simple as getting up in the middle of the night to go to the bathroom, I'm testing these theories out. Um, it's kind of fun. I mean, the coffee cup is an example of that too. So I, I think, uh, I, I find that these sort of everyday introspections are actually quite helpful. Um, it doesn't mean you can ignore the science. I mean, I spend hours every day reading ridiculously complex papers. Um, that's not nearly as much fun, but you have to sort of build up those constraints, um, and the knowledge about the field and who's doing what, and what exactly they think is happening here. And then you can sit back and say, "Okay, let's try to piece this all together." Um, let's come up with some, you know ... I, I, I'm, I'm very, in this group, uh, here, people, they know that je- I do this all those time. I come in with-

   30. LFLex Fridman1:03:59

       (laughs)

Episode duration: 2:09:41