Skip to content
Lex Fridman PodcastLex Fridman Podcast

Jeff Hawkins: Thousand Brains Theory of Intelligence | Lex Fridman Podcast #25

Lex Fridman and Jeff Hawkins on jeff Hawkins maps how brain’s thousand models could reinvent intelligence.

Lex FridmanhostJeff Hawkinsguest
Jul 1, 20192h 9mWatch on YouTube ↗

EVERY SPOKEN WORD

  1. 0:00 – 4:25

    Brain understanding vs. engineering AI: why Hawkins focuses on the neocortex

    1. LF

      The following is a conversation with Jeff Hawkins. He's the founder of the Redwood Center for Theoretical Neuroscience in 2002, and Pneumenta in 2005. In his 2004 book, titled On Intelligence, and in the research before and after, he and his team have worked to reverse engineer the neocortex and propose artificial intelligence architectures, approaches, and ideas that are inspired by the human brain. These ideas include hierarchical temporal memory, HTM, from 2004, and new work, the Thousands Brains Theory of Intelligence from 2017, '18, and '19. Jeff's ideas have been an inspiration to many who have looked for progress beyond the current machine learning approaches, but they have also received criticism for lacking a body of empirical evidence supporting the models. This is always a challenge when seeking more than small incremental steps forward in AI. Jeff is a brilliant mind and many of the ideas he has developed and aggregated from neuroscience are worth understanding and thinking about. There are limits to deep learning as it is currently defined. Forward progress in AI is shrouded in mystery. My hope is that conversations like this can help provide an inspiring spark for new ideas. This is the Artificial Intelligence podcast. If you enjoy it, subscribe on YouTube, iTunes, or simply connect with me on Twitter at Lex Fridman, spelled F-R-I-D. And now here's my conversation with Jeff Hawkins. Are you more interested in understanding the human brain or in creating artificial systems that have many of the same qualities but don't necessarily require that you actually understand the underpinning workings of our mind?

    2. JH

      So, there's a clear answer to that question: My primary interest is understanding the human brain. No question about it. But, um, I also firmly believe that we will not be able to create fully intelligent machines until we understand how the human brain works. So I don't see those as separate problems. Um, I think there's limits to what can be done with machine intelligence if you don't understand the principles by which the brain works, and so I actually believe that studying the brain is actually the frast- the fastest way (laughs) to get to machine intelligence.

    3. LF

      And within that, let me ask the impossible question: How do you not define, but at least think about what it means to be intelligent?

    4. JH

      So, I didn't try to answer that question first. We said, "Let's just talk about how the brain works and let's figure out how c- certain parts of the brain," mostly the neocortex, but some other parts too. The parts of the brain most associated with intelligence. And let's discover the principles by how they work, 'cause i- i- intelligence isn't just like some mechanism and it's not just some capabilities. It's like, okay, we don't even have, know where to begin on this stuff. And so now that we've made a lot of progress on this, after we've made a lot of progress on how the neocortex works, and we can talk about that, I now have a very good idea what's gonna be required to make intelligent machines. I g- I can tell you today, we know some of the things are gonna be necessary, I believe, to create intelligent machines.

    5. LF

      Well, so, we'll, we'll get there. We'll get to the neocortex and some of the theories of how the whole thing works, and you're saying as we understand more and more, uh, about the neocortex, about our own human mind, we'll be able to start to more specifically define what it means to be intelligent. It's not useful to really talk about that until-

    6. JH

      I don't know if it's not useful. You know, look, th- there's a long history of AI, as you know.

    7. LF

      Right.

    8. JH

      And there's been different approaches taken to it. And who knows? Maybe they're all useful.

    9. LF

      Right.

    10. JH

      Right? So-

    11. LF

      In the end.

    12. JH

      ... uh, you know, the good old-fashioned AI, the, uh, expert systems, uh, current convolutional neural networks, they all have their utility. They all have a value in the world. Uh, but I would think almost everyone agree that none of them are really intelligent in, in a sort of a deep way that, that humans are. And so, um, it's, it's just the question is how do you get from where those systems were or are today-

    13. LF

      Mm-hmm.

    14. JH

      ... to where a lot of people think we're gonna go?

    15. LF

      Right.

    16. JH

      And there's a big, big gap there, a huge gap. And I think the quickest way of, of bridging that gap is to figure out how the brain does that. And then we can sit back and look and say, "Oh, which of these principles that the brain works on are necessary and which ones are not?"

    17. LF

      Right.

    18. JH

      Clearly we don't have to build this in, in intelligent machines aren't gonna be built out of, uh, um, you know, organic living cells. Um, but there's a lot of stuff that goes on in the brain that's gonna be necessary.

  2. 4:25 – 6:05

    Can we ever understand the brain? Data-rich neuroscience and paradigm shifts

    1. LF

      So let me ask maybe before we get into the fun details, uh, let me ask maybe a depressing or difficult question. D- do you think it's possible that we will never be able to understand how our brain works, that maybe there's aspects to the human mind, like, we ourselves cannot introspectively get to the core, that there's a wall you eventually hit?

    2. JH

      Yeah. Yeah. I don't believe that's the case. I have never believed that's the case. There's n- not been a single thing we've ever, humans have ever put their minds to that we've said, "Oh, we reached the wall. We can't go any further." It just people keep saying that. People used to believe that about life, you know, elan vital, right? There's like, what's the difference between living matter and nonliving matter? Something special we never understand. We no longer think that. So there's, there's no, uh, historical evidence to suggest this is the case, and I just never even consider that's a possibility. I would also say, uh, today, uh, we understand so much about the neocortex, we've made tremendous progress in the last few years, that I no longer think of it as, um, uh, uh, an open question. Um, the answers are very clear to me. Uh, the pieces we know, we don't know are clear to me, but the framework is all there and, and it's like, oh, okay, we're gonna be able to do this. Uh, this is not a problem anymore, just takes time and effort, but it- there's no mystery, uh, big mystery anymore.

    3. LF

      So then let's get in- into it for, for people like myself who are not very well versed in the human brain, except my own.

    4. JH

      (laughs)

    5. LF

      Uh, can you describe to me at the highest level what are the different parts of the human brain, and then zooming in on the neocortex, the parts of the neocortex and so on-

    6. JH

      Yeah.

    7. LF

      ... a, a quick overview.

  3. 6:05 – 14:15

    Brain architecture overview: old brain vs. neocortex and the common cortical algorithm

    1. JH

      Yeah, sure.A human brain, uh, we can divide it roughly into two parts. There's the old parts, lots of pieces, and then there's the new part. The new part is the neocortex. Um, it's new because it didn't exist before mammals. Only mammals have a neocortex, and in humans, in primates, it's very large. In the human brain, the neocortex occupies about 70% to 75% of the volume of the brain. It's huge. And the old parts of the brain are, are... there's lots of pieces there. There's a spinal cord and there's the brain stem and the cerebellum and the different parts of the basal ganglia and so on. In the old parts of the brain, you have autonomic regulation, like breathing and heart rate. You have basic behaviors, so like walking and running are controlled by the old parts of the brain. All the emotional centers of the brain are in the old part of the brain, so when you feel anger or hungry, lust or things like that, those are all in the old parts of the brain. And, uh, and we associate with the neocortex all the things we think about as sort of high-level perception and cognitive functions. Anything from seeing and hearing and touching things, to language to mathematics and engineering and science and so on. Those are all associated with the neocortex, and they're certainly correlated. Uh, our abilities in those regards are correlated with the relative size of our neocortex compared to other mammals. So that's like the rough division. And you obviously can't understand the neocortex completely isolated, but you can understand a lot of it with just a few interfaces to the old parts of the brain, and so it, it gives you, um, a system to study. The other remarkable thing about the neocortex, uh, compared to the old parts of the brain, is the neocortex is extremely uniform. It's not visibly or anatomically or, uh, it's very, it's like a sh- I always like to say it's like the size of a dinner napkin, about two and a half millimeters thick, and it looks remarkably the same everywhere. Everywhere you look in that two and a half millimeters is this detailed architecture, and it looks remarkably the same everywhere. And that's across species, a mouse versus a cat and a dog and a human. Where if you look at the old parts of the brain, there's lots of little pieces do specific things. So it's, it's like the old parts of a brain involved, like, this is the part that controls heart rate, and this is the part that controls this, and this is this kind of thing, and that's this kind of thing. And these evolved for eons, a long, long time, and they have their specific functions, and all of a sudden, mammals come along and they got this thing called the neocortex, and it got large by just replicating the same thing over and over and over again. This is like, wow, this is incredible. Um, so all the evidence we have, um, and this is an idea that was first, uh, articulated, um, in a very cogent and beautiful argument by a guy named Vernon Mountcastle in 1978, I think it was, um, that the, the neocortex all works on the same principle. So language, hearing, touch, vision, engineering, all these things, are basically underlying or all built in the same computational substrate. They're really all the same problem.

    2. LF

      So at the low level, the building blocks all look similar?

    3. JH

      Yeah, and they're not even that low level. We're not talking about like, like neurons. We're talking about this very complex circuit that exists throughout the neocortex is remarkably similar. It is, it's like, yes, you see variations of it here and there, more of this cell, less, and that's not all, and so on. But, uh, what Mountcastle argued was, he says, you know, if you take a section of neocortex, why is one a visual area and one is a auditory area? Or why is... And his answer was, it's because one is connected to eyes and one is connected to ears.

    4. LF

      Literally, you mean just it's most closest in terms of number of connections to-

    5. JH

      Literally-

    6. LF

      ... the sensor?

    7. JH

      Literally, if you took the optic nerve and attached it to a different part of the neocortex, that part would become a visual region. This actually, this experiment was actually done by Mriganka Sur-

    8. LF

      Oh, boy.

    9. JH

      ... in, uh, in, in developing, I think it was lemurs. I can't remember what it was, some animal. And, and there's a lot of evidence to this. You, you know, if you take a blind person, a person who's born blind at birth, they, they're born with a visual neocortex. Um, it doesn't, may not get any input from the eyes because of some congenital defect or something, and that region becomes, does something else. It picks up another task. So, uh, and it's, so it's this, it's this very complex thing. It's not like, oh, they're all built on neurons. No, they're all built on this very complex circuit, and, uh, and somehow that circuit underlies everything.

    10. LF

      Mm-hmm.

    11. JH

      And so this is, uh, the, it's called the common cortical algorithm, if you will. Some scientists just find it hard to believe, and they just, "I can't believe that's true." But the evidence is overwhelming in this case. And so a large part of what it means to figure out how the brain creates intelligence and what is intelligence in the, in the brain, is to understand what that circuit does. If you can figure out what that circuit does, um, as amazing as it is, then you can, then you understand what all these other cognitive functions are.

    12. LF

      So if you were to sort of put neocortex outside of your book on intelligence, you look... If you wrote a giant tome, a textbook on the neocortex, and you look, uh, maybe a couple centuries from now, how much of what we know now would still be accurate two centuries from now? So how close are we in terms of-

    13. JH

      So, uh-

    14. LF

      ... understanding the neocortex?

    15. JH

      ... I'm gonna, I have to speak from my own particular experience here. So I run a, a small research lab here, it's like-

    16. LF

      Okay.

    17. JH

      ... it's, it's like any other research lab. I'm sort of the principal investigator. There's actually two of us, and there's a bunch of other people. And this is what we do. Uh, we study the neocortex, and we publish our results and so on. So about three years ago, uh, we had a real breakthrough in this, in this field, just tremendous breakthrough. We've started, we've now published, I think, three papers on it. Um, and so I have, I have a pretty good pers- exp- understanding of all the pieces and what we're missing. I would say that almost all the empirical data we've collected about the brain, which is enormous. If you don't know the neuroscience literature, it's just incredibly...... big. And, um, it's, it's for the most part all correct. It's facts and, and experimental results and measurements and all kinds of stuff, but it, none, none of that has been really assimilated into a theoretical framework.

    18. LF

      Right.

    19. JH

      It's, it's data without... It's in the, in the language of Thomas Kuhn, the historian, would be a, sort of a pre-paradigm science. Lots of data, but no way to fit it in together. I think almost all of that's correct. There's gonna be some mistakes in there. Um, and for the most part, there aren't really good cogent theories about how to put it together.

    20. LF

      Right.

    21. JH

      It's not like we have two or three competing good theories, which ones are right and which ones are wrong. It's like, nah, people just like scratching their heads, throwing things, you know. Some people have given up on trying to like figure out what the whole thing does.

    22. LF

      Mm-hmm.

    23. JH

      In fact, there's very, very few labs that we- that- that we do that focus really on theory and all this unassimilated data and trying to explain it. So it's not like we have- we've got it wrong, it's just that we haven't got it at all.

    24. LF

      So it's really, I would say, pretty early days in terms of understanding the fundamental theories, forces of the way our mind works.

    25. JH

      I don't think so. That... I would have said that's true five years ago. So, uh, as I said, we had some really big breakthroughs on this recently and we started publishing papers on this, so, um, uh, you could-

    26. LF

      So we'll get, we'll get to the-

    27. JH

      ... get to that. But, so I don't think it's... I, you know, I'm, I'm an optimist and from where I sit today, most people would disagree with this, but from where I sit today, from what I know, um, uh, it's not super early days anymore. We are... It's, it's, you know, the way these things go is it's not a linear path, right? You don't just start accumulating and get better and better and better. No, you've got all this stuff you've collected, none of it makes sense, all these different things are just sort of surrounding it, and then you're gonna have some breaking points where all of a sudden, oh my God, now we got it right. It's a- that's how it goes in science. And I- I personally feel like we passed that little thing about a couple years ago. Um, well, that big thing a couple years ago. So we can talk about that. Time will tell if I'm right, but, uh, I feel very confident about it. That's why I'm willing to say it on tape like this.

  4. 14:15 – 20:25

    Hierarchical Temporal Memory (HTM): time, memory, and hierarchy as core constraints

    1. LF

      (laughs) Uh, at least very optimistic. So let, let's, before those few years ago, let's take, take a step back to HTM, the hierarchical temporal memory theory, which you first proposed on intelligence and went through a few, uh, different generations. Can you describe what it is, how it evolved through the three generations-

    2. JH

      Yeah.

    3. LF

      ... since you first put it on paper?

    4. JH

      Yeah. So one of the things that, uh, neuroscientists just sort of missed, uh, for many, many years, and I s- and especially people who are thinking about theory, was the nature of time in the brain. Brains process information through time. The information coming into the brain is constantly changing. Um, the, the patterns from my speech right now, if you're listening to it at normal speed, uh, would be changing on your ears about every 10 milliseconds or so, you'd have a change. Just constant flow. When you look at the world, your eyes are moving constantly, three to five times a second, and the inputs

    5. NA

      (laughs)

    6. JH

      ... completely. If I were to touch something like a coffee cup, the- as I move my fingers, the input changes. So this idea that the brain works on time-changing patterns is almost completely, or was almost completely missing from a lot of the basic theories, like fears of vision and so on. It's like, oh no, we're gonna put this image in front of you and flash it and say, "What is it?"

    7. LF

      Right.

    8. JH

      Uh, convolutional neural networks work that way today, right? You know, classify this picture. But that's not what vision is like. Vision is this sort of crazy time-based pattern that's going all over the place, and so is touch and so is hearing.

    9. LF

      Mm-hmm.

    10. JH

      So the first part of, uh, hierarchical temporal memory was the temporal part.

    11. LF

      Right.

    12. JH

      It's, it's to say (laughs) you, you won't understand the brain, nor will you understand intelligent machines unless you're dealing with time-based patterns. The second thing was, uh, the memory component of it was, is to say that, um, we aren't just processing input. We learn a model of the world, that's- and I- and the memory stands for that model. We have to... The point of the brain, the part of the neocortex, it learns a model of the world. We have to store things that, that, that are experiences in a form that leads to a model of the world so we-

    13. LF

      Mm-hmm.

    14. JH

      ... can move around the world, we can pick things up and do things and navigate and know how what's going on. So that's, that's what the memory referred to and many people just, they were thinking about like certain processes without memory at all (laughs) . They're just like processing things. And then finally, the hierarchical component was a reflection to, uh, that the neocortex, although it's this uniform sheet of cells, um, different parts of it project to other parts, which project to other parts, and there is a sort of a rough hierarchy in terms of them. So the hierarchical temporal memory is just saying, look, we should be thinking about the brain as time-based, you know, model memory based-

    15. LF

      Mm-hmm.

    16. JH

      ... and hierarchical processing. Um, and, and that was a placeholder for a bunch of components that we would then plug into that. Uh, we still believe all those things I just said, but we now know so much more that, um, I'm stopping to use the word hierarchical temporal memory-

    17. LF

      Mm-hmm.

    18. JH

      ... because it's, it's insufficient to capture the stuff we know. So again, it's not incorrect, but it's... I now know more and I would rather describe it, uh, more accurately.

    19. LF

      Yeah, so you're basically... We could think of HTM as, uh, emphasizing that there's three aspects of intelligence that are important to think about, whatever the, whatever the eventual theory it converges to.

    20. JH

      Yeah.

    21. LF

      So in terms of time, how do you think of, uh, nature of time across different time scales? You mentioned-

    22. JH

      Yeah.

    23. LF

      ... things changing, uh, sen- sensory inputs changing every 10, 20-

    24. JH

      Yeah.

    25. LF

      ... milliseconds.

    26. JH

      Yeah.

    27. LF

      What about every few minutes, every few-

    28. JH

      Yeah.

    29. LF

      ... months and years?

    30. JH

      Well, if you think about a neuroscience problem, the brain problem-Neurons themselves, uh, can stay active for certain periods of time. Uh, they, there are parts of the brain where they stay active for minutes, you know, so you could hold a certain perception, uh, or an activity, um, uh, for, for a certain period of time. But not f- most of them don't last that long. Um, and so if you think about your thoughts are the activity of neurons, if you're gonna wanna involve something that happened a long time ago, I mean, even just this morning, for example, um, the neurons haven't been active throughout that time. So you have to store that. So if I ask you, "What did you have for breakfast today?" that is memory. That is, you've built that into your model of the world now. You remember that. And that memory is in the, uh, in the synapses, is basically in the formation of synapses. And so, um, uh, it's, it, you're sliding into what, you know, we used to do on timescales.

  5. 20:25 – 28:54

    How Hawkins builds theory: empirical constraints, prediction, and falsification

    1. LF

      And you have to make sure that your models have an intelligence, uh, incorporate it. So, uh, like you mentioned, the state of neuroscience is deeply empirical. A lot of data collection. It's, uh, you know, that's, that's where it is. Uh, as you mentioned, Thomas Kuhn, right?

    2. JH

      Yeah.

    3. LF

      Um, uh, and then you're proposing a theory of intelligence and, uh, which is really the next step, the really i- important step to take. But why, why is HTM or what we'll talk about soon, uh, the right, uh, theory? So is it more in the s- it, wha- is it backed by intuition? Is it backed by evidence? Is it backed by a mixture of both? Is it kind of closer to where string theory is in physics where there's, uh, mathematical components which show that, you know what? It, it seems that this, it fits together too well for it not to be true, which is what, where string theory is.

    4. JH

      Yeah.

    5. LF

      Is that where you're kinda saying-

    6. JH

      It's, it's a mixture of all those things, although, uh, definitely where we are right now, it's definitely much more on the empirical side than, let's say, string theory. The way this goes about, we're theorists, right? So we look at all this data and we're trying to come up with some sort of model that simp- explains it, basically. And there's, uh, uh, unlike string theory, there, there's, there's ju- vast more amounts of empirical data here that I think, uh, than most phys- physicists deal with. And so, um, our challenge is to sort through that and figure out what kind of, um, uh, constructs would explain this. And, uh, when we have an idea, um, you come up with a theory of some sort, you have lots of ways of testing it. First of all, I am, you know, there are a, a hundred years of assimilated, unassimilated empirical data from neuroscience. So we go back and read papers and we say, "Oh, well, did someone find this already?" We, we can predict X, Y, and Z.

    7. LF

      Right.

    8. JH

      And maybe no one's even talked about it since 1972 or something, but we go back and find that and we say, "Oh, either it can support the theory or it can, uh, invalidate the theory." And then we say, "Okay, we have to start over again. Oh, oh, no, it's supportive. Let's keep going with that one." Um, so the way I kind of view it, when we do our work, we come up... We, we look at all this empirical data and it's, it's what I call is a set of constraints. We're not interested in something that's biologically inspired. We're trying to figure out how the actual brain works.

    9. LF

      Mm-hmm.

    10. JH

      So every piece of empirical data is a constraint on a theory.

    11. LF

      Mm-hmm.

    12. JH

      In theory, if you have the correct theory, it needs to explain e- everything, right? So we have this huge number of constraints on the problem, which initially makes it very, very difficult. If you don't have many constraints, you can make up stuff all the day. You can say, "Oh, uh, here's an answer for how you can do this, you can do that, you can do this." But if you consider all biology as a set of constraints, all neuroscience as a set of constraints, and even if you're working on one little part of the neocortex, for example, there are hundreds and hundreds of constraints, these are empirical constraints, that it, it's very, very difficult initially to come up with a theoretical framework for that. But when you do, and it solves all those constraints at once...

    13. LF

      Mm-hmm.

    14. JH

      You have a high confidence that you got something close to correct.

    15. LF

      Yeah.

    16. JH

      It's just im- im- mathematically almost impossible not to be.

    17. LF

      Right.

    18. JH

      So, it, that- that's the- the curse and the advantage of what we have. The- the- the curse is we have to solve, we have to con- meet all these constraints (laughs) -

    19. LF

      (laughs)

    20. JH

      ... which is really hard. But when you do meet them, uh, then you have a- a- a great confidence that you've discovered something. In addition, then, we work with, uh, scientific labs. So we'll say, "Oh, there's something we can't find. We can predict something, but, w- that we can't find it anywhere in the literature."

    21. LF

      Mm-hmm.

    22. JH

      So, we will then, we have s- people we've collaborated with, we'll say, th- sometimes they'll say, "You know what? I have some collected data which I didn't publish, but we can go back and look at it-"

    23. LF

      Mm-hmm.

    24. JH

      "... and see if we can find that." Um, which is much easier than designing a new experiment, you know?

    25. LF

      Yeah.

    26. JH

      New neuroscience experiments take a long time, years. So, uh, although some people are doing that now too. So, uh, but between all of these things, uh, eh, I think it's a reasonable, um, it's actually a very, very good approach. We- we are blessed with the fact that we can test our theories out to Yin Yang here, because there's so much unassimilated data. And we can also falsify our theories very easily, which we do often.

    27. LF

      That's kind of reminiscent to whenever- whenever that was with Copernicus, uh, you know, when you figure out that the sun's at the center of the, um, the solar system, as opposed to Earth, the pieces just fall into place (inaudible)

    28. JH

      Yeah, I think that's the general, um, nature of aha moments, uh, is, and- and it's Copernicus. It could be, you could say the same- same thing about Darwin. Um, you could say the same thing about, you know, um, about the double helix, uh. That- that people have been working on a problem for so long, and they have all this data, and they can't make sense of it, they can't make sense of it. But when the answer comes to you and everything falls into place, it's like, oh my gosh (snaps fingers) , that's it. Um, that's got to be right. I asked both Jim Watson and- and, um, Francis Crick about this. Um, I asked them, you know, "When you were working on trying to discover the structure of the double helix, um, and when you came up with the- the sort of, um, the- the structure that ended up being correct, um, but it was sort of a guess, you know, it wasn't really verified yet." I said, "Did you know that it was right?"

    29. LF

      Mm-hmm.

    30. JH

      And they both said, "Absolutely. We absolutely knew it was right. And, uh, it doesn't matter if other people didn't believe it or not, we knew it was right. They'd get around to thinking it and agree with it eventually anyway."

  6. 28:54 – 34:28

    Evolutionary leap: grid/place-cell navigation reused for general-purpose “concept maps”

    1. LF

      Do you think, uh, in terms of evolutionary timeline in history, the development of the neocortex was a big leap? Or is it just a, uh, small step, um ... So like, if we ran the whole thing over again from the- Yeah. ... from the birth of hum- of life on Earth, how likely would we develop the mechanism of the neocortex?

    2. JH

      Okay, well those are two separate questions. Uh, one is was it a big leap, and one was how likely it is. Okay?

    3. LF

      (laughs) Right.

    4. JH

      They're- they're- they're not necessarily related.

    5. LF

      Maybe correlated? (laughs) I don't know.

    6. JH

      Yeah, maybe correlated, maybe not. And we don't really have enough data to make a judgment about that.

    7. LF

      Okay.

    8. JH

      Um, I would say definitely it was a big leag- leap, and I can tell you why I think, I don't think it was just another incremental step. I'll get to that in a moment. Um, I don't really have any idea how likely it is. Um, if we look at evolution, we have one data point, which is Earth, right? Life formed on Earth billions of years ago, whether it was introduced here or i- created here or someone introduced it, we don't really know, but it was here early. It took a long, long time to get to multicellular life.And then for multi (inaudible) life, um, uh, it took a long, long time to get the neocortex. And we've only had the neocortex for a few 100,000 years. So that's like nothing, (laughs) okay? So is it likely? Well, it certainly isn't something that happened right away o- on Earth, uh, and there were multiple steps to get there, so I would say it's probably not gonna be something that would happen instantaneously on other planets that might have life. Uh, it might take several billion years on average. Um, is it likely? I don't know, but you'd have to survive for several billion years to find out, probably. Um, is it a big leap? Yeah, I think it's, um, uh, it is a, a, a qualitative difference in all other evolutionary steps. Uh, I can try to describe that if you'd like.

    9. LF

      Sure. In which, which, in which way?

    10. JH

      Uh, yeah, I can tell you how. Um, pretty much, uh, I'll, let's start with a little preface. Many of the things that humans are able to do do not have obvious, um, uh, survival advantages, uh, precedent. Uh, you know, w- we c- create music, is that, is there a really survival advantage to that? Uh, maybe, maybe not. Uh, what about mathematics? Is there a real survival advantage to mathematics? Well, maybe. Uh, well, uh, you could stretch it, you could try to figure these things out, right? Um, but up, but most of evolutionary history, everything had immediate survival advantages to them. So I'll tell you a story, um, which I like, may or may not be true, um, but the story goes as follows. Um, organisms have been evolving for since the beginning of life here on Earth, um, and adding this sort of complexity onto that, and this sort of complexity onto that. And the brain itself is evolved this way. In fact-

    11. LF

      Mm-hmm.

    12. JH

      ... there's an old part, an older part, an older, older part to the brain that kind of just keeps calming on new things and we keep adding capabilities. When we got to the neocortex, initially it had, uh, a very clear survival advantage in that it produced better vision and better hearing and better touch and maybe, and so on. But what, what I think happens is, is that evolution discovered, it took, it took a mechanism, um, and this is in our theor- recent theories, but it took a mechanism that evolved a long time ago for navigating in the world, for knowing where you are. These are, um, the so-called grid cells and place cells of an old part of the brain. And it, it took that mechanism for building maps of the world, um, and knowing where you are on those maps and how to navigate those maps, and turns it into a sort of a, a slimmed down, idealized version of it.

    13. LF

      Mm-hmm.

    14. JH

      And that idealized version could now apply to building maps of other things. Maps of coffee cups and maps of phones, maps of, you know-

    15. LF

      Concepts almost.

    16. JH

      ... mathematics, and concepts, yes. And, and not just almost, exactly.

    17. LF

      Right.

    18. JH

      And, and so you, and it just started replicating this stuff, right? It's just like more and more and more. And so we went from being sort of, uh, dedicated purpose neural hardware-

    19. LF

      Mm-hmm.

    20. JH

      ... to solve certain problems that are important to survival, to a general purpose neural hardware that could be applied to all problems, and now it's, it's, uh, it's escaped the orbit of survival. It's, we are now able to apply it to things which we find enjoyment, um, you know, but aren't really clearly, um, survival characteristics. Uh, and that seems to only have happened in humans to, to, to the large extent. Um, and so that's what's going on. We're, we sort of have, um, we've sort of escaped the gravity of evolutionary pressure in some sense in the neocortex, and it now does things which are not, that are really interesting, discovering models of the universe, which may not really help us. It doesn't matter.

    21. LF

      Yeah.

    22. JH

      How does it help us surviving knowing that there might be multiverses, or th- there might be, you know, the age of the universe, or wh- how do, you know, various stellar things occur? It doesn't really help us survive at all, but we enjoy it, and that's what happened.

    23. LF

      Or at least not in the obvious way perhaps it, it is required if, if you look at the entire universe in an evolutionary way, it's required for us to do interplanetary travel and therefore survive past our own sun-

    24. JH

      Yeah.

    25. LF

      ... but you know, let's not get too crazy. (laughs)

    26. JH

      Yeah, but, but, you know, evolution works at one timeframe and-

    27. LF

      Yes, yeah.

    28. JH

      ... well, it's, it, it's survival. If you think of survival of the phenotype, survival of the individual-

    29. LF

      Right, exactly.

    30. JH

      ... it, it, this, that, what you're talking about there, it spans well beyond that.

  7. 34:28 – 40:05

    Thousand Brains Theory: reference frames and location-based prediction (coffee cup insight)

    1. LF

      So let's get into the, the new, as, as you've mentioned this i- idea of the... I don't know if you have a nice name, thousand...

    2. JH

      We call it the Thousand Brain Theory of Intelligence.

    3. LF

      I like it. So can you, can you talk about the, the, this idea of, uh, s- spatial view of concepts and so on?

    4. JH

      Yeah.

    5. LF

      That, that, that-

    6. JH

      So can I just describe sort of the, there's an underlying core discovery-

    7. LF

      Mm-hmm.

    8. JH

      ... which then everything comes from that. That's a very simple, uh, this is really what happened, um, uh, we were deep into problems about understanding how we build models of stuff in the world and how we make predictions about things, and I was holding a coffee cup just like this in my hand, and I, my finger was touching the side, my index finger, and then I moved it to the top, and I was gonna feel the, the rim at the top of the cup. And I asked myself a very simple question, I said, "Well, first of all, I have to say, I know that my brain predicts what it's gonna feel before it touches it." You can just-

    9. LF

      Mm-hmm.

    10. JH

      ... think about it and imagine it, um, and so we know that the brain's making predictions all the time. So the question is, what does it take to predict that, right?

    11. LF

      Mm-hmm.

    12. JH

      And there's a very interesting answer. Uh, it, it, first of all it says the brain has to know it's touching a coffee cup, it has to have a model of a coffee cup. It needs to know where the finger currently is on the cup, relative to the cup, because when I make a movement, it needs to know where it's going to be on the cup after the movement is completed relative to the cup, and then it can make a prediction about what it's gonna sense. So this told me that the neocortex, which is making this prediction, needs to know that it's sensing it's touching a cup, and it needs to know the location of my finger relative to that cup in a reference frame of the cup. It doesn't matter where the cup is relative to my body, it doesn't matter its orientation.None of that matters. It's where my finger is relative to the cup, which tells me then that the neocortex is, has a reference frame that's anchored to the cup. 'Cause otherwise, I wouldn't, uh, be able to say the location and I wouldn't be able to predict my new location. And then we quickly, very instantly- instantly you can say, well, every part of my skin could touch this cup, and therefore every part of my skin's making predictions, and every part of my skin must have a reference frame, um, that it's using to make predictions. So, uh, the- the big idea is that throughout the neocortex, there are ... everything is being re-, uh, is being, um, stored and referenced in reference frames. You can think of them like X, Y, Z reference frames, but they're not like that. Uh, we know a lot about the neuromechanism for this, but the brain thinks in reference frames. And as an engineer, if you're an engineer, this is not surprising. You'd say, if I wanted to build a, uh, a CAD model of the coffee cup, well, I would bring it up in some CAD software and I would assign some reference frame and say, this feature's at these locations, and so on. But the fact that this, uh, the idea that this is occurring throughout the neocortex everywhere, it was a novel idea. And, um, and then a- a zillion things fell into place after that.

    13. LF

      Mm-hmm.

    14. JH

      A zillion. So now, we think about the neocortex as processing information quite differently than we used to do it. We used to think about the neocortex as processing sensory data and extracting features from that sensory data, and then extracting features from the features-

    15. LF

      Mm-hmm.

    16. JH

      ... very much like a- a deep learning network does today.

    17. LF

      Yeah.

    18. JH

      But that's not how the brain works at all. The brain works by, uh, assigning everything, every input, everything, to reference frames. And there are thousands, hundreds of thousands of them active at once in your neocortex. Um, it's a surprising thing to think about, but once you sort of internalize this, you understand that, um, it explains almost every, all the, almost all the mysteries we've had (laughs) about this sy- about this structure. So, um, one of the consequences of that is that every small part of the neocortex, say, a millimeter square, and there's 150,000 of those, so it's about 150,000 square millimeters. If you take every little square millimeter of the cortex, it's got some input coming into it and it's gonna have reference frames where it's assigned that input to. And each square millimeter can learn complete models of objects. So, what do I mean by that? If I'm touching the coffee cup, well, if I just touch it in one place, I can't learn what this coffee cup is because I'm just feeling one part. But if I move it around the cup and touch it at different areas, I can build up a complete model of the cup because I'm now filling in that three-dimensional map, which is the coffee cup. I can say, "Oh, what am I feeling at all these different locations?" That's the basic idea. It's more complicated than that. Um, but so through time, and we talked about time earlier, through time, even a single column which is only looking at, or a single part of the cortex which is only looking at a small part of the world, can build up a complete model of an object. And so if you think about the part of the brain which is getting input from all my fingers-

    19. LF

      Mm-hmm.

    20. JH

      ... so there's, they're spread across the top of your head here, this is the somatosensory cortex, um, there's columns associated of all the different areas of my skin. And what we believe is happening is that all of them are building models of this cup, every one of them. Uh, or things. Not, they're not all building, all ... Not every column or every part of the cortex builds models of everything.

    21. LF

      Mm-hmm.

    22. JH

      But they're all building models of something. And- and so you have ... And so when I, when I touch this cup with my hand, there are multiple models of the cup being invoked. If I look at it with my eyes, there are, again, many models of the cup being invoked 'cause each part of the visual system, it, the brain doesn't process an image. That's mis- that's a misleading idea. Uh, it's just like your fingers touching the cup, so different parts of my retina are looking at different parts of the cup. And thousands and thousands of models of the cup are being invoked at once.

    23. LF

      Mm-hmm.

    24. JH

      And they're all voting with each other, trying to figure out what's going on. So that's why we call it the thousand brains theory of intelligence, because there isn't one model of a cup. There are thousands of models of this cup. There are thousands of models of your cellphone and about cameras and microphones and so on. Um, it's a distributed modeling system, which is very different than what people have thought about it.

  8. 40:05 – 43:49

    Voting instead of sensor fusion: how thousands of models settle on one interpretation

    1. LF

      And so that's a really compelling and interesting idea. I- I have two first questions. So one, on the ensemble part of everything coming together, you have these thousand brains, uh, how- how do you know which one has done the best job of forming the cup?

    2. JH

      Great question. Let me try to explain it. There- there's a problem that's, uh, known in neuroscience called the sensor fusion problem.

    3. LF

      Yes.

    4. JH

      And so it's, the idea is just something like, oh, the image comes from the eye. There's a picture on the retina-

    5. LF

      Mm-hmm.

    6. JH

      ... and it gets projected through the neocortex. Oh, by now, it's all spread out all over the place and it's kind of squirrely and distorted and pieces are all over the, you know, it doesn't look like a picture anymore.

    7. LF

      Yeah.

    8. JH

      When does it all come back together again? Right? Or you might say, well, yes, but I also, I also have sounds or touches associated with the cup, so I'm seeing the cup and touching the cup. How do they get combined together again? So this, it's called the sensor fusion problem, as if all these disparate parts have to be brought together into one model some place. That's the wrong idea. The right idea is that you got all these guys voting. There's auditory models of the cup, there's visual models of the cup, there's tactile models of the cup. Um, there are one ... in the vision system, there might be ones that are more focused on black and white and ones focusing on color. It doesn't really matter. There's just thousands and thousands of models of this cup. And they vote. They don't actually come together in one spot. It- just literally, think of it this way. Imagine you have ... these columns are like th- about the size of a little piece of spaghetti, okay? Like, uh, two and a half millimeters tall and about a millimeter in width. They're not physical, like, uh ... but you can think of them that way. And each one's trying to guess what this thing is they're touching. Now, they can g- they can do a pretty good job if they're allowed to move over time. So I could reach my hand into a black box and move my finger around an object, and if I touch enough places, I go, "Okay, now I know what it is." But often, we don't do that. Often, I can just reach and grab something with my hand all at once and I get it, or if I had to look through the world through a straw, so I'm only invoking one little column, I can only see part of something 'cause I have to move the straw around. But if I open my eyes, I see the whole thing at once. So what we think is going on is all these little pieces of spaghetti, if you will, all these little columns in the cortex are all trying to guess what it is that they're sensing. Um, they'll do a better guess if they have time.... and can move over time, so if I move my eyes, I move my fingers. But if they don't, they have a, they have a, a, a poor guess. It's a, it's a probabilistic guess of what they might be touching. Now imagine they can post their probability-

    9. LF

      Mm-hmm.

    10. JH

      ... at the top of the little piece of spaghetti. Each one of them says, "I think it..." and it's not really a probability distribution. It's more like a set of possibilities. In the brain, it doesn't work as a probability distribution, it works as more like a, what we call a union. So you could say, uh, in one column it says, "I think it could be a coffee cup, a soda can, or a water bottle." And another column says, "I think it could be a coffee cup or, um, you know, a telephone, or a camera," or whatever, right?

    11. LF

      Mm-hmm.

    12. JH

      Um, and all these guys are saying what they think it might be. And there's these long-range connections in certain layers in the cortex. So there's, in some layers, in some cell types in each column, send the projections across the brain, and that's the voting occurs. And so there's a, a simple associative memory mechanism, we've, we've described this in a recent paper and we've modeled this, um, that says they can all quickly settle on the only, or the, the one best answer for all of them.

    13. LF

      Mm-hmm.

    14. JH

      If there is a single best answer, they all vote and say, "Yep, it's got to be the coffee cup." And at that point, they all know it's a coffee cup. And at that point, everyone acts as if it's the coffee cup. They're, "Yep, we know it's a coffee, even though I've only seen one little piece of this world, I know it's a coffee cup I'm touching or I'm seeing or whatever." And so you can think of all these columns are looking at different parts and different places, different sensory input, different locations. They're all different. But this layer that's doing the voting, um, that's, it solidifies. It's just like, it crystallizes and says, "Oh, we all know what we're doing." And so you don't bring these models together in one model, you just vote, and there's a crystallization of the vote.

  9. 43:49 – 55:56

    Reference frames for abstract thought: method of loci, “bird space,” and mathematics as navigation

    1. LF

      Great. That's a, at least a compelling way to think about, uh, about the way you, uh, form a model of the world. Now, you, you talk about a coffee cup. Do you see this, as far as I understand, you were proposing this as well, that this extends to much more than coffee cups?

    2. JH

      Yeah.

    3. LF

      (laughs) That-

    4. JH

      It does. (laughs)

    5. LF

      Or at least the physical world, it expands to, uh, the world of concepts.

    6. JH

      Yeah. It does. And well, the first, uh, the prima face, ev- evidence for that is that the regions of the neocortex that are associated with language or high level thought or mathematics or things like that, they look like the regions of the neocortex that process vision, hearing, and touch. They're, they don't look any different.

    7. LF

      Right.

    8. JH

      Or they look only marginally different. Um, and so one would say, "Well, if Vernon Mountcastle, who proposed that all the co- all the parts of the neocortex are doing the same thing, if he's right, then the parts that are doing language or mathematics or physics are working on the same principle. They must be working on the principle of reference frames." So that's a little odd thought. Um, said, "Hmm." But of course, we had no i- we had no prior idea how these things happened, so let's, let's go with that. Um, and, uh, we, in our recent paper, we talked a little bit about that. I've been working on it more since. I have, uh, more, better ideas about it now. Um, I'm sitting here very confident that that's what's happening, and I can give you some examples to help you think about that. Um, it's not that we understand it completely, but I understand it better than I've described it in any paper so far. So, uh, but we did put that idea out there, says, okay, this is, it's, it's, it's a good place to start, you know? And the evidence would suggest it's how it's happening, and then we can start tackling that problem one piece at a time. Like, what does it mean to do high level thought? What does it mean to do language? How would that fit into a reference frame framework?

    9. LF

      Yeah, so there, there's, uh, I don't know if you could tell me if there's a connection, but there's an app called Anki that helps you remember different concepts, and they, they talk about like a memory palace that helps you remember completely random concepts by sort of trying to put them in a physical space in your mind.

    10. JH

      Yeah.

    11. LF

      And putting them next to each other.

    12. JH

      It's called the method of loci.

    13. LF

      Loci, yeah.

    14. JH

      Yes, it is.

    15. LF

      For some reason, that seems to work really well.

    16. JH

      Yeah.

    17. LF

      Uh, now that's a very narrow kind of application of just remembering some facts and so on.

    18. JH

      But that's not, but that's a very, very telling one. Okay? So...

    19. LF

      Yes, exactly. So this seems like you're describing a mechanism why this seems to work.

    20. JH

      Yes. So, so basically the way, what we think is going on is all c- things you know, all concepts, all ideas, words, everything you know, um, are stored in reference frames. And, and so, uh, if you want to remember something, you have to basically navigate through a reference frame the same way a rat navigates through a maze and the same way my finger ra- navigates to this coffee cup. You are moving through some space. And so what you can, if you have a random list of things you were asked to remember, by assigning them to a reference frame you already know very well, let's say your house, right? An idea of the method of loci is you can say, "Okay, in my lobby, I'm gonna put this thing, and then, and then the bedroom, I put this one. I go down the hall, I put this thing." And then you want to recall those facts or recall those things, you just walk mentally, you walk through your house. You're mentally moving through a reference frame that you already had, and that tells you, uh, there's two things that are really important about that. It tells us the brain prefers to store things in reference frames, and that the method of recalling things or thinking, if you will, is to move mentally through those reference frames. You could move physically through some reference frames, like I could physically move through the reference frame of this coffee cup. I can also mentally move through the reference frame of the coffee cup, imagining me touching it. But I can also mentally move my house, and, and so now we can ask yourself, are all concepts stored this way? There was some recent, uh, research, um, using human subjects and fMRI, and I'm gonna apologize for not knowing the name of the scientist who did this. Um, but, um, what they did is they, they put humans in this fMRI machine, which is one of these imaging machines, and they w- they gave the humans tasks to think about birds. So they had different types of birds and birds that looked big and small and long necks and long legs, things like that. And what they could tell from the fMRI, it was a very clever experiment, um, they get to tell when humans were thinking about the birds, that the birds, the, the knowledge of birds was arranged in a reference frame similar to the, the ones that are used when you navigate in a room. That, these are called grid cells, and there, there are grid cell-like patterns of activity in the neocortex when they do this. So that-... it's a very clever experiment, you know. And what it basically says that even when you're thinking about something abstract and you're not really thinking about it as a reference frame, it tells us the brain is actually using a reference frame. And it's using the same neural mechanisms. These grid cells are the basic same neural mechanisms that we- we propose that grid cells, which exist in the, in the old part of the brain, the entorhinal cortex, that that mechanism is now... Similar mechanism is used throughout the neocortex. It's the same. Nature preserved this interesting way of creating reference frames. And so now they have empirical evidence that, uh, when you think about concepts like birds, that you're using reference frames that are built on grid cells. So this, that's similar to the method of loci, but in this case, the birds are related so it makes-

    21. LF

      Hm.

    22. JH

      ... they create their own reference frame which is consistent with bird space. And when you think about something, you go through that. You, I, you can make these same examples. Let's take a math- mathematics, right? Let's say you wanna prove a conjecture, okay? What is a conjecture? A conjecture is a, uh, a statement you believe to be true, but you haven't proven it. And so it might be an equation. "I, I, I want to show that this is equal to that." And you have a p- you have some places you start with. You say, "Well, I know this is true and I know this is true, and I think that maybe to get to the final proof, I need to go through some intermediate results."

    23. LF

      Mm-hmm.

    24. JH

      What I believe is happening is literally these equations or these points are assigned to a reference frame, a mathematical reference frame. And when you do mathematical operations, a simple one might be multiply or divide, but you might be a Laplace transform or something else. That is like a movement in the reference frame of, of the math. And so you're literally trying to discover a path from one location to another location in a space of mathematics. And if you can get to these intermediate results, then you know y- your map is pretty good and you know you're using the right operations. Uh, much of what we think about as solving hard problems is designing the correct reference frame for that problem, figuring out how to organize the information, and what behaviors I want to use in that space to get me there. (laughs)

    25. LF

      Yeah, so if you dig in o- on the idea of this reference frame, whether it's the math you start, a set of axioms to try to get to proving the conjecture, uh, can you try to describe, maybe take a step back, w- how you think of the reference frame in that context? Is, is it the reference frame that the axioms are happy in? Is it the reference frame that might contain everything? Is it a changing thing-

    26. JH

      So the, it, it, it-

    27. LF

      ... as you...

    28. JH

      You have many, many reference frames. I mean, in fact, the way the theory, the thousand-brain theory of intelligence does it, every single thing in the world has its own reference frame. So, um, every word has its own reference frame. And we can talk about this ma- uh, the mathematics workout, this is no problem for neurons to do this.

    29. LF

      But how many reference frames does a coffee cup have?

    30. JH

      Well-

  10. 55:56 – 1:04:17

    Open problems and attention: orientations, nested object composition, and focus control

    1. LF

      So in terms of each individual column building up more and more information over time, do you think that mechanism is well-understood? In, in your mind, you've proposed a lot of architectures there. Is that a key piece or is it, um, is the big piece the thousand-brain, uh, theory of intelligence, the ensemble of it all?

    2. JH

      Well, I think they're both big. I mean, clearly the concept, as a theorist, the concept is most exciting, right? We want-

    3. LF

      A high-level concept 000c-

    4. JH

      A high-level concept. This is a totally new way of thinking about how the neocortex works, so that is appealing. It has all these ramifications. And with that as a framework for how the brain works, you can make all kinds of predictions and solve all kinds of problems. Now we're trying to work through many of these details right now. Okay, how do the neurons actually do this? Well, it turns out, if you think about grid cells and place cells in the old parts of the brain, there's a lot that's known about 'em, but there's still some mysteries. There's a lot of debate about exactly the details, how these work and what are 0000. And we have that still, that same level of detail, that same level of concern. What we spend here most of our time doing is trying to, um, make a, a very good list of the things we don't understand yet. Uh, that's the key part here. What are the constraints? It's not like, "Oh, this thing seems to work. We're done." No, it's like, "Okay, it kinda works, but these are other things we know it has to do, and it's not doing those yet."

    5. LF

      Mm-hmm.

    6. JH

      Um, I would say we're well on the way here. I'm, we're not done yet. Um, there's a lot of trickiness to this system, but the basic principles about how different layers in the neocortex are doing much of this, we understand. But there's some fundamental parts that we don't understand as well.

    7. LF

      S- so what would you say is, uh, one of the harder open problems or one of the ones that have been bothering you?

    8. JH

      Oh. Oh, yeah.

    9. LF

      Uh, uh, keeping you up at night the most? (laughs)

    10. JH

      Oh, well, uh, right now, this is a detailed thing that wouldn't apply to most people, okay? (laughs)

    11. LF

      (laughs) Sure.

    12. JH

      But you want me to answer that question?

    13. LF

      Yeah, please. (laughs)

    14. JH

      Um, we've talked about, uh, as if, oh, to predict what you're going to sense from this coffee cup, I need to know where my finger's gonna be on the coffee cup. That is true, but it's insufficient. Um, think about my finger touches the edge of the coffee cup. My finger can, uh, touch it at different orientations.

    15. LF

      Right.

    16. JH

      I can rotate my finger around here, um, and that doesn't change. I s- I can make that prediction e- and somehow, so it's not just the location. There's an orientation component of this as well. Um, this is known in the old parts of the brain too. There's things called head direction cells, which which way the rat is facing. It's the same kind of basic idea. Um, so if my finger were a rat, you know, in three dimensions, I have a three-dimensional orientation and I have a three-dimensional location. If I was a rat, I would have a, you might think of it as a two-dimensional location, a two-dimensional orienta- a one-dimensional orientation, like just which way is it facing. So how the, the two components work together, how it is that I, I combine orientation, right, the orientation of my sensor, um, as well as the, the, the location, um, is a tricky problem. And, uh, I think I've made progress on it. (laughs)

    17. LF

      So, uh, at a bigger version of that, so perspective's super interesting, uh, but super specific.

    18. JH

      Yeah, you, I warned you. It was 0004-

    19. LF

      No, no.

    20. JH

      Yeah. (laughs)

    21. LF

      That's really good, but is there, uh, there's a more general version of that. Do you think, um, context matters, the fact that we are in a building in North America, uh, that, uh, that we l- in the day and age where we have mugs? I mean, um-

    22. JH

      Well-

    23. LF

      ... there's all this extra information that you bring to the table about everything else in the room that's outside of just the coffee cup.

    24. JH

      Of course, it is. Yeah.

    25. LF

      How, how does it get-

    26. JH

      Yeah. So-

    27. LF

      ... connected, do you think?

    28. JH

      Um, yeah, and that, and that is a, another really interesting question. I'm gonna throw that under the, the rubric or the, the name of attentional problems.

    29. LF

      Mm-hmm.

    30. JH

      First of all, we have this model. I have many, many models of the world.

  11. 1:04:17 – 1:15:20

    Deep learning critique and neuron realism: dendrites, synapses, and sparse predictive computation

    1. LF

      So let's talk a little bit about deep learning and, uh, the successes, uh, in, in the applied space of, uh, neural networks. And, uh, ideas of training model on, on data and these, these simple computational units, neuron, uh, artificial neurons that, uh, with back propagation, there's statistical ways of being able to, uh, generalize from the training set onto data that's similar to that training set.

    2. JH

      Yeah.

    3. LF

      So where do you think are the limitations of those approaches? What do you think are its strengths-

    4. JH

      Yeah.

    5. LF

      ... relative to your major efforts of constructing a theory of human intelligence?

    6. JH

      Yeah. Well, I'm not an expert in this field. I'm somewhat knowledgeable, so, uh, but I'm not-

    7. LF

      Some of it is in just your intuition. What are your-

    8. JH

      Well, I ha- I have a little bit more than intuition, but, uh, but I just wanna say, like, you know, uh, one of the things that you asked me, do I spend all my time thinking about neuroscience? I do. That's to the exclusion of thinking about things like convolutional neural networks.

    9. LF

      Yeah.

    10. JH

      But I try to stay current. So look, I think it's great, the progress they've made. It's fantastic. And as I mentioned earlier, it's very highly useful for many things. Uh, the models that we have today are actually derived from a lot of neuroscience principles. Uh, they are distributed processing systems and distributed memory systems, and that's how the brain works. Um, they use things that we, we might call them neurons, but they're really not neurons at all. So we can just, they're not really neurons. They're, they're distributed processing systems. Um, and, uh, and the nature of hierarchy, that came also from, uh, neuroscience. And so there's a lot of things, the, the learning rules basically, so not back prop, but other, you know, .......................... learner.

    11. LF

      But I'd be, I'd be curious to say they're not neurons at all. Can you describe in which way? I mean, it's, some of it is obvious, but I'd be curious if, if you have specific ways-

    12. JH

      Yes.

    13. LF

      ... in which you think are the biggest differences.

    14. JH

      Yeah. Yeah. We had a paper in 2016 called Why Neurons Have Thousands of Synapses, and it p- p- and if you read that paper, you'll s- you'll know what I'm talking about here. A real neuron in the brain is a complex thing. It, uh, let's just start with the synapses on it, which is a connection between neurons. Uh, real neurons can everywhere from 5 to 30,000 synapses on 'em. Um, the ones near the cell body, the ones that are close to the, the soma or the cell body, those are like the ones who people model in artificial neurons.

    15. LF

      Mm-hmm.

    16. JH

      There is a few hundred of those, maybe. They can affect the cell. They can make the cell become active.... 95% of the synapses can't do that. They're too far away. So if you activate one of those synapses, it just doesn't affect the cell body enough to make any difference.

    17. LF

      Any one of them individually?

    18. JH

      Any one of them individually, or even if you do a mass of 'em.

    19. LF

      Hm.

    20. JH

      But what we n- what, what real neurons do is the following. If you activate or they, you know, you, you get 10 to 20 of 'em active at the same time, meaning they're all receiving an input at the same time. And those 10 to 20 synapses, or 40 synapses, within a very short distance on the dendrite, like 40 microns, so a very small area. So if you activate a bunch of these right next to each other at some distant place, what happens is it creates what's called the dendritic spike. And the dendritic spike travels through the dendrites and can reach the soma, or the cell body. Now when it gets there, it changes the voltage, which is sort of like gonna make the cell fire, but never enough to make the cell fire. It's sort of, what we call, it says we depolarize the cell. You raise the voltage a little bit, but not enough to do anything. It's like, "Well what good is that?" And then it goes back down again. So, uh, we proposed a, a theory which I'm very confident in, uh, basics are, is that what's happening there is those 95% of those synapses are recognizing dozens to hundreds of unique patterns. They can rec- you know, about 10, 20 neur- synapses at a time, and they're acting like predictions. So the neuron actually is a predictive engine on its own. It, it can fire when it gets enough what they call proximal input from the ones near the cell fire, but it can get ready to fire from dozens to hundreds of patterns that are recognized just from the other guys. And the advantage of this to the neuron is that when it actually does produce a spike, an action potential, it does so slightly sooner than it would've otherwise. And so what good is slightly sooner? Well, the slightly sooner part is it- there's a- all the neurons in the ... the excitatory neurons in the brain are surrounded by these inhibitory neurons, and they're very fast, the inhibitory neurons, these basket cells. And if I get my spike out a little bit sooner than someone else, I inhibit all my neighbors around me.

    21. LF

      Mm-hmm.

    22. JH

      Right? And what you end up with is a different representation. You end up with a representation that matches your prediction. It's a, it's a sparser representation, meaning there's fewer s- neur- neurons are active, but it's much more specific. And so we showed how networks of these neurons can do very sophisticated, uh, temporal prediction basically. So, so this, uh, summarize this. Real neurons in the brain are time-based prediction engines and, and they, and there's no concept of this at all-

    23. LF

      Right.

    24. JH

      ... in, um, artificial, uh, what we call point neurons. I don't think you can build a brain without 'em. I don't think you can build intelligence without 'em because you- it's, it's the en- it's where l- large part of the time comes from. It's, it's ... these are predictive models and the time is in- is ... there's a prior and a, and a, you know, a prediction and an action and it's inherent to every neuron in the neocortex.

    25. LF

      Yeah.

    26. JH

      So, so I would say that point neurons sort of model a piece of that.

    27. LF

      Mm-hmm.

    28. JH

      And not very well at that either. Uh, but you know.

    29. LF

      (laughs) Yeah.

    30. JH

      Like, like for example, synapses, um, are very unreliable and you cannot assign any precision to them.

Episode duration: 2:09:41

Install uListen for AI-powered chat & search across the full episode — Get Full Transcript

Transcript of episode -EVqrDlAqYo

Get more out of YouTube videos.

High quality summaries for YouTube videos. Accurate transcripts to search & find moments. Powered by ChatGPT & Claude AI.

iOSAndroidClaudeChrome