Judea Pearl: Causal Reasoning, Counterfactuals, and the Path to AGI | Lex Fridman Podcast #56

1. 0:00 – 5:58

   Judea Pearl’s origins: analytic geometry and the power of translating “languages”

   1. LFLex Fridman0:00

      The following is a conversation with Judea Pearl, professor at UCLA, and the winner of the Turing Award that's generally recognized as the Nobel Prize of computing. He's one of the seminal figures in the field of artificial intelligence, computer science, and statistics. He has developed and championed probabilistic approaches to AI, including Beijing Networks, and profound ideas in causality in general. These ideas are important not just to AI, but to our understanding and practice of science. But in the field of AI, the idea of causality, cause and effect, to many, lie at the core of what is currently missing and what must be developed in order to build truly intelligent systems. For this reason, and many others, his work is worth returning to often. I recommend his most recent book, called Book of Why, that presents key ideas from a lifetime of work in a way that is accessible to the general public. This is the Artificial Intelligence podcast. If you enjoy it, subscribe on YouTube, give it five stars on Apple Podcast, support it on Patreon, or simply connect with me on Twitter, @lexfridman, spelled F-R-I-D-M-A-N. If you leave a review on Apple Podcast especially, but also Castbox, or comment on YouTube, consider mentioning topics, people, ideas, questions, quotes in science, tech, and philosophy that you find interesting, and I'll read them on this podcast. I won't call out names, but I love comments with kindness and thoughtfulness in them, so I thought I'd share them with you. Someone on YouTube highlighted a quote from the conversation with Noam Chomsky, where he said that, "The significance of your life is something you create." I like this line as well. On most days, the existentialist approach to life is one I find liberating and fulfilling. I recently started doing ads at the end of the introduction. I'll do one or two minutes after introducing the episode, and never any ads in the middle that break the flow of the conversation. I hope that works for you and doesn't hurt the listening experience. This show is presented by Cash App, the number one finance app in the App Store. I personally use Cash App to send money to friends, but you can also use it to buy, sell, and deposit Bitcoin in just seconds. Cash App also has a new investing feature. You can buy fractions of a stock, say $1 worth, no matter what the stock price is. Brokerage services are provided by Cash App Investing, a subsidiary of Square and member SIPC. I'm excited to be working with Cash App to support one of my favorite organizations called FIRST, best known for their FIRST robotics and LEGO competitions. They educate and inspire hundreds of thousands of students in over 110 countries, and have a perfect rating on Charity Navigator, which means the donated money is used to the maximum effectiveness. When you get Cash App from the App Store or Google Play, and use code LEXPODCAST, you'll get $10, and Cash App will also donate $10 to FIRST, which again is an organization that I've personally seen inspire girls and boys to dream of engineering a better world. And now, here's my conversation with Judea Pearl. You mentioned in an interview that science is not a collection of facts, but a constant human struggle with the mysteries of nature. What was the first mystery that you can recall that hooked you, that captivated your curiosity?

   2. JPJudea Pearl3:30

      Oh, the first mystery. That's a good one. Yeah, I remember that.

   3. LFLex Fridman3:37

      What was it?

   4. JPJudea Pearl3:38

      I had a fever for three days, uh, when I learned about Descartes' analytic geometry, and I found out that you can do all the construction in geometry using algebra. And I couldn't get over it. I simply couldn't get out of bed. (laughs)

   5. LFLex Fridman3:58

      So, what- what kind of world does analytic geometry unlock?

   6. JPJudea Pearl4:02

      Well, it connects algebra with geo- geometry. Okay, so Descartes had the idea that, um, (clears throat) geometrical construction and geometrical theorems and the assumptions can be articulated in the language of algebra, which means that all the proof that we did in high school in trying to prove that the three bisectors meet at one point, and that... (laughs) Okay, uh, all this can be proven by just-

   7. LFLex Fridman4:36

      Through algebra.

   8. JPJudea Pearl4:36

      ... shuffling around notation. Uh, that was a-

   9. LFLex Fridman4:41

      The connection-

   10. JPJudea Pearl4:42

       ... traumatic experience.

   11. LFLex Fridman4:42

       (laughs) The tr- traumatic experience.

   12. JPJudea Pearl4:44

       For me, it was. I'm telling you, right?

   13. LFLex Fridman4:45

       So, it's the connection between the different mathematical disciplines, that they all-

   14. JPJudea Pearl4:50

       No, in between two diff- two different languages.

   15. LFLex Fridman4:51

       Just even... Languages?

   16. JPJudea Pearl4:53

       Yeah.

   17. LFLex Fridman4:54

       So, which mathematic discipline is the most beautiful? Is geometry it for you?

   18. JPJudea Pearl4:58

       Both are beautiful. They have, uh, almost the same power.

   19. LFLex Fridman5:02

       But there's a visual element to geometry, being a-

   20. JPJudea Pearl5:05

       Visually, (laughs) it's more transparent, but, uh, once you get over to algebra, then, uh, y- a linear equation is a straight line. This translation is easily absorbed, uh, and, um, the f- to pass a tangent to a circle, uh, you know, (laughs) you have the basic theorems and you can do it with algebra. So, but, uh, the transition from one to another was really ... I thought that Descartes was the greatest mathematician of all times. (laughs)

   21. LFLex Fridman5:37

       So, you have been at the ... if you think of engineering and mathematics as a spectrum-

   22. JPJudea Pearl5:43

       Yes.

   23. LFLex Fridman5:43

       ... uh, you have been ... you have walked casually along this spectrum throughout your- throughout your life. You know, you had a little bit of engineering, and then, you know, uh, you have b- done a little bit of mathematics here and there. (laughs)

2. 5:58 – 7:09

   Learning math “chronologically”: teachers, history, and the people behind theorems

   1. JPJudea Pearl5:58

      Well, not a little bit. I mean, we got a very solid background in mathematics because our teachers were geniuses.

   2. LFLex Fridman6:06

      Yeah.

   3. JPJudea Pearl6:07

      Our teachers came from Germany in the 1930s, running away from Hitler. Uh, they left their careers in Heidelberg and Berlin, and came to teach high school in Israel. And we were the beneficiary of that experiment. So, I al- when they taught us math, the good way.

   4. LFLex Fridman6:25

      What's the good way to teach math?

   5. JPJudea Pearl6:26

      Chronologically.

   6. LFLex Fridman6:29

      The people?

   7. JPJudea Pearl6:29

      The people behind the theorems, yeah. Their cousins, and their nieces (laughs) and their faces, and how they jumped from the bathtub when they scream, "Eureka!" (laughs) and ran naked in town. (laughs)

   8. LFLex Fridman6:46

      So, you were almost educated as a historian of math.

   9. JPJudea Pearl6:49

      No, we just got a glimpse of that history together with the theorem. So, every, um, exercise in math was connected with a person-

   10. LFLex Fridman6:59

       S-

   11. JPJudea Pearl6:59

       ... and the time of the person, the period.

   12. LFLex Fridman7:03

       The period also mathematically speaking?

   13. JPJudea Pearl7:05

       Mathematically speaking, yes. Not the politics.

   14. LFLex Fridman7:08

       Yeah.

   15. JPJudea Pearl7:08

       No.

3. 7:09 – 9:14

   From engineering and physics to AI: superconductivity and the “Pearl vortex”

   1. LFLex Fridman7:09

      So, and then in, uh, in university, you have, you have gone on to do engineering.

   2. JPJudea Pearl7:16

      Yeah. I got a BS in engineering at Technion.

   3. LFLex Fridman7:19

      Mm-hmm.

   4. JPJudea Pearl7:20

      Right? And then, uh, I moved here for graduate work, and I got to, I did engineering, uh, in addition to physics in Rutgers.

   5. LFLex Fridman7:31

      Mm-hmm.

   6. JPJudea Pearl7:32

      And it would combine very nicely with my thesis, which I did in RCA Laboratories in superconductivity.

   7. LFLex Fridman7:40

      And then somehow thought to switch to, uh, almost computer science software even, even the, the, uh, not switch, but long to become, uh, to get into software engineering a little bit-

   8. JPJudea Pearl7:53

      Yes.

   9. LFLex Fridman7:53

      ... almost even programming, if you can call it that in the '70s. So, though there's all these disciplines.

   10. JPJudea Pearl7:58

       Yeah.

   11. LFLex Fridman7:58

       If you were to pick a favorite, what, uh, in terms of engineering and mathematics, which path do you think has more beauty? Which path has more power?

   12. JPJudea Pearl8:08

       It's hard to choose now. I enjoy doing physics, and even have a vortex named on my name. So, I have a (laughs) investment in immortality. (laughs)

   13. LFLex Fridman8:22

       Do you, uh, so what- what is a vortex?

   14. JPJudea Pearl8:25

       Vortex is in superconductivity.

   15. LFLex Fridman8:27

       In the superconductivity, yeah.

   16. JPJudea Pearl8:27

       You have permanent current, uh, swirling around. One way or the other, you can have it store one or zero for computer. That was we worked on in, in the 1960 in RCA. And, uh, I discovered a few nice phenomena with the vortices.

   17. LFLex Fridman8:44

       (laughs)

   18. JPJudea Pearl8:44

       You push current-

   19. LFLex Fridman8:44

       So, there's a pearl vortex.

   20. JPJudea Pearl8:44

       ... and they move. Pearl vortex, right? You can Google it.

   21. LFLex Fridman8:48

       (laughs)

   22. JPJudea Pearl8:49

       Right? I, I didn't know about it, but the physicists, they picked up on my thesis, on my, uh, PhD thesis. And, uh, they, um, it becomes popular when thin film superconductors became important for high temperature superconductors. So, they called it, uh, pearl vortex without my knowledge. (laughs) I discovered it only about 15 years ago.

4. 9:14 – 11:59

   Determinism, quantum mechanics, and free will as an AI-solvable illusion

   1. LFLex Fridman9:14

      You have footprints in all of the sciences. So, let's talk about the universe a little bit. Is the universe at the lowest level deterministic or stochastic, in your amateur philosophy view? Put another way, does God play dice?

   2. JPJudea Pearl9:29

      Well, we, we know it is stochastic, right? Because-

   3. LFLex Fridman9:33

      Today. Today, we think it is stochastic.

   4. JPJudea Pearl9:35

      Yes. We think, because we have the Heisenberg uncertainty principle, and we have some, uh, experiments to, um, confirm that.

   5. LFLex Fridman9:45

      All we have is experiments to confirm it. We don't understand why.

   6. JPJudea Pearl9:50

      Why is already-

   7. LFLex Fridman9:51

      You wrote a book about why. (laughs)

   8. JPJudea Pearl9:55

      Yes. (laughs) Yeah. It- it's a puzzle. It's a puzzle that you have the, uh, dice-flipping machine or god, and the, and the, uh, result of the flipping propagate with a speed faster than the speed of light (laughs) .

   9. LFLex Fridman10:11

      Right.

   10. JPJudea Pearl10:12

       We can't explain it, okay? So, um, but it's, it only governs microscopic phenomena.

   11. LFLex Fridman10:21

       So, you don't think of quantum mechanics as useful-

   12. JPJudea Pearl10:25

       No.

   13. LFLex Fridman10:25

       ... uh, for understanding-

   14. JPJudea Pearl10:26

       It's just a diversion.

   15. LFLex Fridman10:26

       ... the nature of reality?

   16. JPJudea Pearl10:28

       No. Diversionary.

   17. LFLex Fridman10:30

       So, in your thinking, the world might as well be deterministic?

   18. JPJudea Pearl10:36

       The world is deterministic. And as far as the neural fir- neuron firing is concerned, it, it's a deterministic to first approximation.

   19. LFLex Fridman10:47

       What about free will?

   20. JPJudea Pearl10:49

       Free will is also a nice e- exercise. Free will is an illusion-

   21. LFLex Fridman10:54

       Illusion.

   22. JPJudea Pearl10:54

       ... that we AI people are going to solve.

   23. LFLex Fridman10:59

       So, what do you think once we solve it, that solution will look like? Once we put it in the page.

   24. JPJudea Pearl11:04

       The solution will look like... First of all, it will look like a machine, a machine that act as though it had free will. It communicates with other machines as though they have free will. And you wouldn't be able to tell the difference between a machine that does and machine that doesn't have free will. Okay?

   25. LFLex Fridman11:24

       So, the illusion, it propagates the illusion of free will amongst the other machines.

   26. JPJudea Pearl11:29

       And, and faking it is having it. Okay? That's what Turing test is all about.

   27. LFLex Fridman11:34

       Yeah.

   28. JPJudea Pearl11:35

       Faking intelligence is intelligent, because it's not easy to fake. It's very hard to fake, and you can only fake if you have it.

   29. LFLex Fridman11:46

       (laughs) Yeah, that-

   30. JPJudea Pearl11:48

       So-
5. 11:59 – 14:48

   What probability and correlation really mean (and why we reach for causality anyway)

   1. LFLex Fridman11:59

      So, uh, let- let's begin at the beginning with probability, uh, both philosophically and mathematically. What- what does it mean to say the probability of something happening is 50%? What is probability?

   2. JPJudea Pearl12:16

      It's the degree of uncertainty that an agent has about the world.

   3. LFLex Fridman12:22

      You're still expressing some knowledge in that statement.

   4. JPJudea Pearl12:24

      Of course. If the probability is 90%, it's absolutely different kind of knowledge than if it is 10%. (clears throat)

   5. LFLex Fridman12:32

      But it's still not solid knowledge. It's-

   6. JPJudea Pearl12:36

      It is solid knowledge, boy. If you tell me that (laughs) the 90% assurance smoking will, um, give you lung cancer in five years, versus 10%, it's a piece of useful knowledge.

   7. LFLex Fridman12:52

      So, this statistical view of the universe-

   8. JPJudea Pearl12:55

      Yeah.

   9. LFLex Fridman12:56

      ... why is it useful? So, we're swimming in complete uncertainty.

   10. JPJudea Pearl13:00

       Yeah.

   11. LFLex Fridman13:00

       Most of everything around us is-

   12. JPJudea Pearl13:02

       It allows you to predict things with a certain probability, and computing those probabilities are very useful. That's, uh, the whole idea of, uh, (sighs) of prediction, and you need prediction to be able to survive. If you cannot predict the future, then you're just a ... crossing the street will be extremely, uh, fearful.

   13. LFLex Fridman13:26

       And so you've done a lot of work in causation, and so let's, let's think about correlation.

   14. JPJudea Pearl13:32

       I started with the probability.

   15. LFLex Fridman13:34

       You started with probability.

   16. JPJudea Pearl13:35

       Yes.

   17. LFLex Fridman13:35

       You've invented the Bayesian networks.

   18. JPJudea Pearl13:38

       Yeah.

   19. LFLex Fridman13:39

       And so, you know, we'll, we'll, we'll, we'll dance back and forth between these levels of, uh-

   20. JPJudea Pearl13:46

       Okay.

   21. LFLex Fridman13:46

       ... uh, uncertainty. But what, what is correlation? What is it? So, probability of something happening is something, but then there's a bunch of things happening, and, uh, sometimes they happen together, sometimes not. They're independent or not. So, how do you think about correlation of things?

   22. JPJudea Pearl14:03

       Correlation occurs when two things vary together over a very long time, is one way of measuring it, or when you have a bunch of variables that they all vary cohesively. Um, then we call we have a correlation here. And usually, when we think about correlation, we really think causally. Things can not be correlated unless there is a reason for them to vary together. Why should they vary together if they don't see each other? Why should they vary together?

   23. LFLex Fridman14:35

       So, underlying it somewhere is causation.

   24. JPJudea Pearl14:38

       Yes. Hidden in our intuition, there is a notion of causation, because we cannot grasp any other logic except causation.

6. 14:48 – 23:11

   Conditioning pitfalls: selection effects, Simpson’s paradox, and observational studies

   1. LFLex Fridman14:48

      And how does conditional probability differ from causation? So, what is conditional probability?

   2. JPJudea Pearl14:57

      Conditional probability, how things vary when one of them, uh, stays the same. Now, staying the same means that I have chosen to look only at those incidents where the guy has the same value as previous one. It's my choice as an experimenter. So, things that are not correlated before could become correlated. Like, for instance, if I have two coins which are uncorrelated, okay, and I choose only those flippings experiments in which a bell rings, and the bell rings when at least one of them is a tail, okay, then suddenly, I see correlation between the two coins, because I only look at the cases where the bell rang. You see, it's my design with my ignorance, essentially, with my, uh, audacity to ignore certain incidents, I suddenly create a correlation where it doesn't exist physically.

   3. LFLex Fridman16:07

      Right. So that's ... You just outlined one of the flaws of observing the world and, and trying to infer something fundamental about the world from looking at the correlation.

   4. JPJudea Pearl16:17

      I don't look at it as a flaw. The world works like that. Which mean- but the flaws comes if you try to impose-

   5. LFLex Fridman16:25

      Hmm.

   6. JPJudea Pearl16:25

      ... um, causal logic on correlation. It doesn't work too well.

   7. LFLex Fridman16:34

      I mean, but that's exactly what we do. That's what ... That has been the majority of science, is you're r-

   8. JPJudea Pearl16:40

      No, no, no. Majority of, of naive science. Statisticians know it. The statisticians know that if you condition on a third variable, then you can destroy or create correlations among two other variables.

   9. LFLex Fridman16:55

      Right.

   10. JPJudea Pearl16:55

       They know it. It's in the data.

   11. LFLex Fridman16:58

       Right.

   12. JPJudea Pearl16:58

       There's nothing to surprise them. That's why they all dismiss the Simpson paradox. "Ah, we know it." They don't know anything about it. (laughs)

   13. LFLex Fridman17:07

       Well, there's, uh, there's disciplines, like psychology, where all the variables are hard to ge- to account for. And so, uh, oftentimes, there is a leap between correlation to causation. You're, you're imposing-

   14. JPJudea Pearl17:18

       What do you mean a leap?

   15. LFLex Fridman17:21

       Uh-

   16. JPJudea Pearl17:21

       Who, who is trying to get causation from correlation? No one.

   17. LFLex Fridman17:25

       Not, not ... You're not proving causation-

   18. JPJudea Pearl17:27

       (laughs)

   19. LFLex Fridman17:28

       ... but you're sort of, uh, um, discussing it, implying, sort of hypothesizing without ability to prove.

   20. JPJudea Pearl17:35

       Well, which discipline we have in mind? I'll tell you if they are obsolete-

   21. LFLex Fridman17:40

       (laughs)

   22. JPJudea Pearl17:40

       ... or if they are outdated, or they're about to get outdated.

   23. LFLex Fridman17:44

       Yes.

   24. JPJudea Pearl17:44

       Or (laughs) -

   25. LFLex Fridman17:45

       Yes.

   26. JPJudea Pearl17:45

       Yeah, tell me which one you have in mind.

   27. LFLex Fridman17:46

       Well, psychology, you know? Uh-

   28. JPJudea Pearl17:48

       Psychology, what is it? SEM? Structural equation modeling?

   29. LFLex Fridman17:50

       No, no. I was thinking of applied psychology, studying, um ... for example, we work with human behavior in semi-autonomous vehicles, how people behave. And you have to conduct these studies of people driving cars.

   30. JPJudea Pearl18:04

       Everything starts with a question. What is a research question?
7. 23:11 – 29:23

   Causality needs a language: models first, discovery second

   1. LFLex Fridman23:11

      Let's, let's, let's even take a step back. You invented, uh, Bayesian networks ... that look awfully a lot like they express something like causation, but they don't, not necessarily. So how do we turn Bayesian networks into expressing causation? How do we build causal networks? This A causes B, B causes C, how do we start to infer that kind of thing?

   2. JPJudea Pearl23:38

      We start asking ourself question, what are the factors that would determine the value of X? X could be blood pressure, uh, death, um, hungry, hunger.

   3. LFLex Fridman23:53

      But these are hypotheses that we propose-

   4. JPJudea Pearl23:56

      Hypotheses, everything which has to do with causality comes from a theory, okay? The difference is only what kind ... how you interrogate the theory that you have in your mind.

   5. LFLex Fridman24:11

      So, it still needs the human expert to propose-

   6. JPJudea Pearl24:13

      Right. You need the human expert to specify-

   7. LFLex Fridman24:18

      Yeah.

   8. JPJudea Pearl24:18

      ... the initial model.... initial model could be very qualitative, just who listens to whom, by whom listen to, I mean one variable listen to the other. So, I say, "Okay, the tide is listening to the moon, and, and not to the rooster crow." Okay? And so far, this is our understanding of the world in which we live, scientific understanding of reality. We have to start there, because if we don't know how to handle cause and effect relationship when we do have a model, then we certainly do not know how to handle it when we don't have a model. So, let's start first, an AI slogan is, "Representation first, discovery second."

   9. LFLex Fridman25:13

      Mm-hmm.

   10. JPJudea Pearl25:13

       Right? If I give you all the information that you need, can you do anything useful with it? That is the first, representation. How do you represent it? I give you all the knowledge in the world. How do you represent it? When you represent it, I ask you, "Can you infer X or Y or Z? Can you answer certain queries? Is it, uh, complex? Is it polynomial?" It... All the computer science exercises-

   11. LFLex Fridman25:41

       Mm-hmm.

   12. JPJudea Pearl25:41

       ... we do, once you give me a representation for my knowledge, then you can ask me, "Now I understand how to represent things, how do I discover them?" It's a secondary thing.

   13. LFLex Fridman25:54

       First of all, we... I- I should echo the statement that mathematics and the current m- m- much of the machine learning world has not considered causation, that A causes B.

   14. JPJudea Pearl26:06

       Correct.

   15. LFLex Fridman26:06

       Just in anything. So that- that seems like a s- uh, that seems like a non-obvious thing that you- you'd think we would've really acknowledged it, but we haven't. So, we- we have to put that on the table. So, uh, knowledge. How hard is it to create a knowledge from which t- to work?

   16. JPJudea Pearl26:28

       In certain area-

   17. LFLex Fridman26:30

       S-

   18. JPJudea Pearl26:30

       ... it's easy, because we have only four or five major variables, okay, and- and an epidemiologist or an economist can put them down, um, what, the- the minimum wage, uh, un- unemployment, p- policy, X, Y, Z, and- and start collecting data, and quantify the parameter that were left unquantified with the initial knowledge.

   19. LFLex Fridman27:01

       Yeah.

   20. JPJudea Pearl27:01

       Okay? That's the routine work that you find in experimental psychology-

   21. LFLex Fridman27:09

       Yes.

   22. JPJudea Pearl27:10

       ... in economics, ev- everywhere, any- any health science. That's a routine things. But I should emphasize, you should start with a research question: What do you want to estimate? Once you have that, you have to have a language of expressing what you want to estimate. You think it's easy? No.

   23. LFLex Fridman27:32

       So, we can talk about two things. I think, b- one is, um, how the science of causation is, uh, very useful for a- uh, answering certain questions, and then the other is, how do we create intelligent systems, uh, that need to reason with causation? So, if my research question is, "How do I pick up this water bottle from the table?" (laughs) Uh, the w- all the knowledge that is required to be able to do that, how do we construct that knowledge base? Does it- does it- do we return back to the problem that we didn't solve in the '80s with expert systems? Do we have to solve that problem of automated construction of knowledge?

   24. JPJudea Pearl28:18

       Hmm. You're talking about the, uh, task of eliciting knowledge from an expert.

   25. LFLex Fridman28:26

       Task of eliciting knowledge from an expert, or just self-discovery of more knowledge, uh, more and more knowledge, so automating the building of knowledge as much as possible.

   26. JPJudea Pearl28:38

       It's a different game in the causal domain, because... It- it's essentially the same thing. You have to start with some knowledge, and you're trying to enrich it.

   27. LFLex Fridman28:51

       Mm-hmm.

   28. JPJudea Pearl28:51

       But you don't enrich it-

   29. LFLex Fridman28:53

       Mm-hmm.

   30. JPJudea Pearl28:53

       ... by asking for more rules. You enrich it by asking for the data, for them to look at the data and quantifying, and ask queries that you couldn't answer when you started. You couldn't, because it- it- it's... the- the question is quite complex, and it's not within the, um, capability of ordinary cognition, of ordinary person, or ordinary expert even, to answer.

8. 29:23 – 34:47

   Interventions and the do-operator: asking “what if we do X?”

   1. LFLex Fridman29:23

      So, what kind of questions do you think we can s- start to answer? What's-

   2. JPJudea Pearl29:27

      Even a simple one. Suppose... uh, yeah (laughs) . I start with easy one.

   3. LFLex Fridman29:31

      Let's do it.

   4. JPJudea Pearl29:31

      Okay. What's the effect of a drug on my recovery? Uh, was it the aspirin that caused my headache to be cured, or was it the television program, or the good news I received? Um, this is already... you see, it's a difficult question, because it's find the cause from effect. The easy one is find effect from cause.

   5. LFLex Fridman29:57

      That's right. So, first you construct a model saying that this is an important research question.

   6. JPJudea Pearl30:01

      Yeah.

   7. LFLex Fridman30:01

      This is an important question. Then you- you d-

   8. JPJudea Pearl30:04

      No, no, I didn't construct a model yet. I just said it's important question.

   9. LFLex Fridman30:07

      It's important question.

   10. JPJudea Pearl30:07

       And the first exercise is express it mathematically. What do you want to prove? Like, if I tell you, "What's the eff- what will be the effect of taking this drug?" Okay? You have to say that in mathematics. How do you say that?

   11. LFLex Fridman30:22

       Yes.

   12. JPJudea Pearl30:23

       Can you write down the question? Not the answer.I want to find the effect of the drug on my headache.

   13. LFLex Fridman30:32

       Right.

   14. JPJudea Pearl30:32

       W- write down, right? Write it down.

   15. LFLex Fridman30:34

       That's where the do calculus comes in. (laughs)

   16. JPJudea Pearl30:35

       Yes. Do operator, what is do operator?

   17. LFLex Fridman30:38

       Do operator, yeah.

   18. JPJudea Pearl30:39

       Yeah. You have to have a-

   19. LFLex Fridman30:40

       Which is nice. It's the difference between association and intervention.

   20. JPJudea Pearl30:43

       Correct.

   21. LFLex Fridman30:43

       Very beautifully, sort of constructed.

   22. JPJudea Pearl30:45

       Yeah. So, we co- we have a do operator, so the do calculus connected on the do operator itself, connects the operation of doing to something that we can see.

   23. LFLex Fridman30:57

       Right. So, as opposed to the purely observing, you're making the choice to change a variable.

   24. JPJudea Pearl31:05

       Yeah. That's what it, it expresses.

   25. LFLex Fridman31:07

       Hm. Gotcha.

   26. JPJudea Pearl31:08

       And then, the way that we interpret it, and the mechan- mechanism by which we take your query, and we translate it into something that we can work with, is by giving it semantics. Saying that you have a model of the world, and you cut off all the incoming arrow into X, and you're looking now in the modified, mutilated model, you ask for the probability of Y. That is interpretation of doing X, because by doing things, you've liberated them from all influences that acted upon them earlier, and you subject them to the tyranny of your muscles.

   27. LFLex Fridman31:50

       So you... (laughs) You remove all the questions about causality by doing them.

   28. JPJudea Pearl31:55

       So, because-

   29. LFLex Fridman31:55

       You're now-

   30. JPJudea Pearl31:56

       ... it's one level of questions.
9. 34:47 – 37:09

   Why adding arrows can hurt: assumptions, identifiability, and when experiments are necessary

   1. JPJudea Pearl34:47

      I construct a model, I can still cannot answer it.

   2. LFLex Fridman34:50

      Right.

   3. JPJudea Pearl34:50

      I have to see if I have enough information in the model that would allow me to find out the effects of intervention from a non-interventional study. From observation- hands-off study.

   4. LFLex Fridman35:03

      Right. So, what's needed to make that-

   5. JPJudea Pearl35:06

      You need to have assumptions about who affects whom. If the, if the graph had a certain property, the answer is yes, you can get it from observational study.

   6. LFLex Fridman35:20

      Mm-hmm.

   7. JPJudea Pearl35:20

      If the graph is too meshy, bushy, bushy, the answer is no, you cannot. Then, you need to find either different kind of observation that you haven't considered or one experiment.

   8. LFLex Fridman35:35

      So basically, does that put... That puts a lot of pressure on you to encode wisdom into that graph.

   9. JPJudea Pearl35:42

      Correct. But you don't have to encode more than what you know. God forbid. (laughs) If you put... like economists are doing that, they call it identifying assumption. They put assumptions, even they don't prevail in the world, they put assumptions so they can identify things. Let no-

   10. LFLex Fridman35:59

       But the problem is... Yes, beautifully put, but the problem is you don't know what you don't know. So...

   11. JPJudea Pearl36:06

       You know what you don't know, because if you don't know, you say, "It's possible. It's possible that X affect the, uh, traffic tomorrow." And I'm...

   12. LFLex Fridman36:17

       Yeah.

   13. JPJudea Pearl36:17

       It's possible. You put down a arrow which says it's pos- every arrow in the graph says it's possible.

   14. LFLex Fridman36:24

       So, there's not a significant cost to adding arrows that-

   15. JPJudea Pearl36:28

       The more arrow you add-

   16. LFLex Fridman36:30

       The better.

   17. JPJudea Pearl36:30

       ... the less likely you are to identify things from purely observational data.So, if the whole world is bushy, and every- everybody ac- affect everybody else, the answer is, you can answer it ahead of time. I cannot answer my query from observational data. I have to go to experiments.

   18. LFLex Fridman36:56

       Right. So, you talk about machine learning as essentially learning by s- association, or reasoning by association. And this new calculus is allowing for intervention. I like that word. Uh, but-

   19. JPJudea Pearl37:08

       Intervention.

10. 37:09 – 40:46

    Counterfactuals as explanations: responsibility, regret, and the limits of today’s ML

    1. LFLex Fridman37:09

       ... action. So, you also talk about counterfactuals.

    2. JPJudea Pearl37:12

       Yeah.

    3. LFLex Fridman37:13

       And I'm trying to sort of understand the difference in counterfactuals and in- in- intervention. Uh, what's the... well, first of all, what is counterfactuals, and why are they useful? Wh- wh- why are they, eh, eh, especially, uh, useful as opposed to just reasoning what- what effect actions have?

    4. JPJudea Pearl37:34

       Well, conf- counterfactual contains what we normally call explanations.

    5. LFLex Fridman37:40

       Can you give an example of a counterfactual?

    6. JPJudea Pearl37:41

       If I tell you that acting one way affects something else, I didn't explain anything yet. But if I, if I ask you, uh, "Was it the aspirin that cure my headache?" I'm asking for explanation. What cure my headache?

    7. LFLex Fridman37:58

       Mm-hmm.

    8. JPJudea Pearl37:58

       And putting a finger on aspirin provide information. It was aspirin. It was responsible for your headache going away. If y- if you didn't take the aspirin, you will still have a headache.

    9. LFLex Fridman38:16

       So by s- by saying, "If I didn't take aspirin, I would have a headache." You're thereby saying that aspirin is the thing that removes the headache.

    10. JPJudea Pearl38:26

        Yeah, but you have to have another important information. "I took the aspirin, and my headache is gone." Okay? It's very important information. Now I'm reasoning backward, and I said, "Was it the aspirin?"

    11. LFLex Fridman38:40

        Yeah.

    12. JPJudea Pearl38:42

        Okay.

    13. LFLex Fridman38:42

        By considering what would have happened if everything else is the same, but I didn't take aspirin.

    14. JPJudea Pearl38:47

        That's right.

    15. LFLex Fridman38:47

        Counterfactual.

    16. JPJudea Pearl38:47

        So, in order, things took place, you know. Joe killed Schmo, okay?

    17. LFLex Fridman38:53

        Yeah.

    18. JPJudea Pearl38:53

        And Schmo would, uh, would be alive had Joe not used his gun.

    19. LFLex Fridman38:59

        Right.

    20. JPJudea Pearl38:59

        Okay? So, that is the counterfactual. It, it, it conf- it had a confliction, it had a conflict here or clash between observed fact that he, he did shoot, okay, and the hypothetical predicate, which says, "Had he not shot." You have a clash, a logical clash. They cannot exist together. That's a counterfactual, and that is the source of our explanation of our idea of responsibility, regret, and free will.

    21. LFLex Fridman39:34

        Yeah, so it certainly seems, uh, that's the highest level of reasoning, right, is counterfactual?

    22. JPJudea Pearl39:40

        Yes, and physicists do it all the time.

    23. LFLex Fridman39:41

        Who does it all the time?

    24. JPJudea Pearl39:42

        Physicists.

    25. LFLex Fridman39:43

        Physicists.

    26. JPJudea Pearl39:45

        In every equation of physics, let's say you have a Hooke's law, and you put, uh, one kilogram on the spring, and the spring is one meter, and you say, "Had this weight been two kilogram, the spring would have been twice as long." It's no problem in, for a physicist-

    27. LFLex Fridman40:04

        Hmm.

    28. JPJudea Pearl40:04

        ... to say that. Accept that mathematics is only i- i- is in the form of equation. Okay? Equating the weight, proportionality constant, and the length of the string. So, you don't have the asymmetry in the equation of physics, although every physicist thinks counterfactually. Ask the high school kids.

    29. LFLex Fridman40:28

        Mm-hmm.

    30. JPJudea Pearl40:29

        Had the weight been three kilograms, what would be the length of the spring? They can answer it immediately, because they do the counterfactual processing in their mind, and then they put it into equation, algebra equation, and they solve it. Okay? But a robot cannot do that.

11. 40:46 – 44:37

    How could machines learn causality? Manipulation, noise, and inferring the “strings behind the facts”

    1. LFLex Fridman40:46

       How do you make a robot learn these relationships?

    2. JPJudea Pearl40:51

       Uh, well, and, and why you would learn?

    3. LFLex Fridman40:53

       Or not learn.

    4. JPJudea Pearl40:53

       Suppose you tell him. Can you do it? See, before you go learning-

    5. LFLex Fridman40:57

       Yeah.

    6. JPJudea Pearl40:57

       ... you have to ask yourself, suppose I give him all the information, okay? How... can the, can the robot perform a task that I ask him to perform? Can he reason and say, "No, it wasn't the aspirin. It was the good news you received on the phone."

    7. LFLex Fridman41:15

       Right, because, well, unless the robot had a model, uh, a, a causal, um, model of the world.

    8. JPJudea Pearl41:23

       Right, right. That's right.

    9. LFLex Fridman41:24

       Look, I'm sorry I have to linger on this.

    10. JPJudea Pearl41:26

        But now we have to linger, and we have to say how do we, how do we do it?

    11. LFLex Fridman41:29

        How do we build it?

    12. JPJudea Pearl41:29

        Yes.

    13. LFLex Fridman41:30

        How do we build a causal model without a h- a team of human experts-

    14. JPJudea Pearl41:35

        No, put- eh- eh-

    15. LFLex Fridman41:36

        ... running around?

    16. JPJudea Pearl41:36

        ... why don't, why don't you go to learning right away? You're too much involved with learning.

    17. LFLex Fridman41:41

        'Cause I like babies. Babies learn fast.

    18. JPJudea Pearl41:43

        Oh, yeah.

    19. LFLex Fridman41:43

        I'm trying to figure out how they do it.

    20. JPJudea Pearl41:45

        Oh, yeah. Okay, good.

    21. LFLex Fridman41:45

        So, yeah.

    22. JPJudea Pearl41:46

        That's another question. How do the babies come out with a counterfactual model of the world?

    23. LFLex Fridman41:51

        Yeah.

    24. JPJudea Pearl41:51

        And babies do that.

    25. LFLex Fridman41:53

        Yeah.

    26. JPJudea Pearl41:53

        They know how to play with the... in the crib. They know which balls hits another one.

    27. LFLex Fridman41:59

        Mm-hmm.

    28. JPJudea Pearl41:59

        And, but they learn it by, um, playful manipulation of the world.

    29. LFLex Fridman42:06

        Yes.

    30. JPJudea Pearl42:07

        In a simple world involve only toys, and balls, and chimes (laughs) and bells. But it's a v- if you think about it, it's a complex world.
12. 44:37 – 59:13

    Metaphor as intelligence: mapping the unfamiliar to the familiar (and why curve-fitting isn’t enough)

    1. JPJudea Pearl44:37

       I think it's a matter of combining simple models from many, many sources-

    2. LFLex Fridman44:42

       Mm-hmm.

    3. JPJudea Pearl44:42

       ... from many, many disciplines, and many metaphors. Metaphors are the basics of human intelligence, basis.

    4. LFLex Fridman44:51

       Yeah. So h- how do you think of, about a metaphor, in terms of its use in human intelligence?

    5. JPJudea Pearl44:56

       Metaphors is an expert system.

    6. LFLex Fridman45:00

       Mm-hmm.

    7. JPJudea Pearl45:01

       An expert... it's, it's mapping problem with which you are not familiar to a problem with which you are familiar.

    8. LFLex Fridman45:12

       Mm-hmm.

    9. JPJudea Pearl45:13

       Like, I'll give you a good example. The Greek believed that the sky is an opaque shell. It's not really outsp- in infinite space. It's an opaque shell, and the stars are holes-

    10. LFLex Fridman45:30

        Yeah.

    11. JPJudea Pearl45:30

        ... poked in the shell through which you see the eternal light. Okay? That was a metaphor. Why? Because they u- they understand how you poke holes in shells, okay? They're not f- they were not familiar with infinite space, okay? And it's, so, a- and, and we are walking on a shell of a turtle, and if you get too close to the edge, you're going to fall down to Hades, or wherever.

    12. LFLex Fridman45:56

        Yeah.

    13. JPJudea Pearl45:57

        Yeah? Um, that's a metaphor. It's not true. But this kind of metaphor enabled Aristarchus to measure the radius of the Earth.

    14. LFLex Fridman46:10

        Hm.

    15. JPJudea Pearl46:10

        Because he said, "Come on. If the w- we are walking on a turtle shell, then the ray of light coming through this angle will be different, um, this place, will be a different angle that's coming to this place. I know the distance. I'll measure the two, uh, angles, and then I have the radius of the shell of the, of the turtle."

    16. LFLex Fridman46:32

        Mm-hmm.

    17. JPJudea Pearl46:32

        Okay? And he did. And he found his measurement very close to the measurements we have today through the, uh, what, 6,000 and 700, 700 kilometers (laughs) of the Earth. That's something that would not h- occur to Babylonian astronomer, even though the Babylonian experiments were the machine learning people of the time. They fit curves, and they could predict the, um, eclipse of the moon much more accurately than the Greek, because they fit curve. Okay? Uh, so that's the difference in metaphor.

    18. LFLex Fridman47:18

        Mm-hmm.

    19. JPJudea Pearl47:19

        Something that you're familiar with. Again, a turtle shell. Okay? What does it mean if you are familiar? Familiar means that answers to certain questions are explicit. You don't have to derive them.

    20. LFLex Fridman47:35

        And they were made explicit, because somewhere in the past, you've constructed a model of that. Uh-

    21. JPJudea Pearl47:43

        Before, yeah, you- you're familiar with-

    22. LFLex Fridman47:44

        Yeah.

    23. JPJudea Pearl47:45

        So the child is familiar with billiard balls.

    24. LFLex Fridman47:47

        Yes.

    25. JPJudea Pearl47:48

        So the child could predict that if you let loose of one ball, the other one will bounce off. These are... You, you obtain that by, um, familiarity. Familiarity is answering questions, and you store the answer explicitly. You don't have to derive them. So, this is ideal for metaphor. All our life, all our intelligence is built around metaphors. Mapping from the unfamiliar to the familiar, but the, um, marriage between the two is a tough thing, which I, which we haven't yet been able to algorithmatize.

    26. LFLex Fridman48:24

        So, you think of that process of cas- of using metaphor to leap from one place to another as, we can call it reasoning? Is it a kind of reasoning?

    27. JPJudea Pearl48:35

        It is, uh, reasoning by metaphor, metaphorical reason-

    28. LFLex Fridman48:38

        Reasoning by metaphor.

    29. JPJudea Pearl48:38

        ... yeah.

    30. LFLex Fridman48:39

        Do you think of that as learning?So, learning is a popular terminology today in a narrow sense.
13. 59:13 – 1:04:31

    Toward human-level AI: communication, ethics, self-models, and consciousness as a ‘software blueprint’

    1. LFLex Fridman59:13

       Got it. So, I know you're not a futurist, but are you excited? Have you, when you look back at your life, longed for the idea of creating a human level intelligence system?

    2. JPJudea Pearl59:26

       Yeah. I'm, uh, I'm driven by that. All my life, I'm driven just by one thing (laughs) . But I go slowly. I go from what I know, to the next step, incrementally.

    3. LFLex Fridman59:39

       So, without imagining what the end goal looks like, do you imagine what an-

    4. JPJudea Pearl59:44

       Yes, end- the end goal is going to be a machine that can answer sophisticated questions-

    5. LFLex Fridman59:50

       Questions.

    6. JPJudea Pearl59:50

       ... counterfactuals, regret, compassion, um, responsibility, and free will.

    7. LFLex Fridman59:59

       So, what is a good test? Is a Turing test a reasonable test?

    8. JPJudea Pearl1:00:04

       If there is test of free will, doesn't exist yet, uh, there's no-

    9. LFLex Fridman1:00:08

       How would you test free will? Uh, that's a-

    10. JPJudea Pearl1:00:11

        So far, we know only one thing. I mean (laughs) , if robots can communicate with reward and punishment among themselves, and hitting each other on the wrists and say, "You shouldn't have done that."

    11. LFLex Fridman1:00:27

        Mm-hmm.

    12. JPJudea Pearl1:00:27

        Okay? Um, playing better soccer because they can do that.

    13. LFLex Fridman1:00:34

        What do you mean, because they can do that?

    14. JPJudea Pearl1:00:36

        Because they can communicate among themselves. It's ƒ-

    15. LFLex Fridman1:00:38

        Because of the communication, they can do the soccer?

    16. JPJudea Pearl1:00:40

        Because they communicate like us.

    17. LFLex Fridman1:00:42

        Yeah.

    18. JPJudea Pearl1:00:42

        Reward and punishment. Yes, you didn't pass the ball the right, the right time and so forth, therefore you're going to sit on the bench for the next two... If they start communicating like that, the question is, will they play better soccer?As opposed to what? As opposed to what they do now, without this ability to reason about, uh, uh, reward and punishment, responsibility.

    19. LFLex Fridman1:01:06

        And a lot of factions.

    20. JPJudea Pearl1:01:08

        So far, I can only think about communication.

    21. LFLex Fridman1:01:11

        Communication is... And, and, and not necessarily natural language, but just communication.

    22. JPJudea Pearl1:01:16

        Just communication. And that's important to have a quick and effective means of communicating knowledge. If the coach tells you, "You should have passed the ball." Ping. He conveys so much knowledge to you, as opposed to, what? Go down and change your software. Right. That's the alternative. But the coach doesn't know your software.

    23. LFLex Fridman1:01:37

        Right.

    24. JPJudea Pearl1:01:37

        So, how can the coach tell you, "You should have passed the ball?" But that... (laughs) Our language is very effective. "You should have passed the ball." You know your software, you tweak the right module, okay? And next time, you don't do it.

    25. LFLex Fridman1:01:52

        Now, that's for playing soccer where the rules are well defined.

    26. JPJudea Pearl1:01:55

        No, no, no. Well, they're not well defined. When you should pass the ball and when you should stop-

    27. LFLex Fridman1:01:58

        Is not well defined.

    28. JPJudea Pearl1:02:00

        No, it's a-

    29. LFLex Fridman1:02:01

        Yeah.

    30. JPJudea Pearl1:02:02

        It's very soft world, very noisy.
14. 1:04:31 – 1:19:08

    Risk, society, and personal tragedy: AI as a new species; Israel, religion, and the story of Daniel Pearl

    1. LFLex Fridman1:04:31

       Do you have concerns about the future of AI, all the different trajectories of all of our research?

    2. JPJudea Pearl1:04:39

       Yes.

    3. LFLex Fridman1:04:40

       Um, where's your hope, where the movement heads? Where are your concerns?

    4. JPJudea Pearl1:04:44

       I- I'm concerned, because I know we are building a new species that has the capability of exceeding our... Exceeding us, uh, uh, exceeding our capabilities, and can breed itself and take over the world. Absolutely. It's a new species. It's uncontrolled. We don't know the degree to which we control it. We don't even understand what it means to be able to control this new species. Uh, so, I'm concerned. I don't have anything to add to that, because it's such a gray area, an unknown. It never happened in history.

    5. LFLex Fridman1:05:27

       Yeah.

    6. JPJudea Pearl1:05:28

       Okay? The only, the only time it happened in history was evolution with a human being.

    7. LFLex Fridman1:05:36

       Right.

    8. JPJudea Pearl1:05:37

       And it wasn't very successful, was it? (laughs)

    9. LFLex Fridman1:05:40

       Uh-

    10. JPJudea Pearl1:05:40

        Some people says it was a great success.

    11. LFLex Fridman1:05:42

        For us, it was, but a few people along the way, uh, or a few creatures along the way would not agree. So, uh, so, it's just because it's such a gray area, there's nothing else to say.

    12. JPJudea Pearl1:05:55

        Uh, we have a sample of one.

    13. LFLex Fridman1:05:56

        Sample of one.

    14. JPJudea Pearl1:05:58

        That's us.

    15. LFLex Fridman1:06:00

        But-

    16. JPJudea Pearl1:06:00

        We dominantly

    17. NANarrator1:06:00

        (murmuring)

    18. LFLex Fridman1:06:00

        ... some people would look at you and say, "Yeah, but we were looking to you to help us make sure that-

    19. JPJudea Pearl1:06:10

        Correct.

    20. LFLex Fridman1:06:10

        "... sample two works out okay."

    21. JPJudea Pearl1:06:13

        Actually, we have more than a sample of one. We have theory of, theories.

    22. LFLex Fridman1:06:16

        Yeah.

    23. JPJudea Pearl1:06:17

        And that's a good... And s- we don't need to be statisticians. So, sample of one doesn't mean any, uh, uh, poverty of knowledge.

    24. LFLex Fridman1:06:25

        Yeah.

    25. JPJudea Pearl1:06:25

        It's not. Sample of one plus theory, conjectural theory of what could happen.

    26. LFLex Fridman1:06:32

        Yeah.

    27. JPJudea Pearl1:06:33

        That, we do have. But I- I really feel helpless in contributing to this argument, because I know so little, and, uh, and my imagination is limited. And, uh, I know how much I don't know, and, uh-

    28. LFLex Fridman1:06:52

        Hmm.

    29. JPJudea Pearl1:06:52

        ... I... But I'm concerned.

    30. LFLex Fridman1:06:55

        You were born and raised in Israel.
15. 1:19:08 – 1:23:00

    Advice, rebellion, and legacy: ask questions you can’t yet name

    1. LFLex Fridman1:19:08

       (laughs) Do you have advice for young minds today dreaming about creating, as you have dreamt, creating intelligent systems? What is the best way to arrive at new breakthrough ideas and carry them through the fire of criticism and, uh, and past conventional ideas?

    2. JPJudea Pearl1:19:28

       Ask your questions (laughs) freely. Your questions are never dumb.

    3. LFLex Fridman1:19:37

       (laughs)

    4. JPJudea Pearl1:19:37

       And solve them your own way, (laughs) okay? And don't take no for an answer. Look, if they are really dumb, you will find out quickly by trying an arrow to see that they're not leading any place. But follow them and try to understand things your way. That is my, uh, advice. I don't know if it- it's going to help anyone.

    5. LFLex Fridman1:20:04

       No, that brilliantly put.

    6. JPJudea Pearl1:20:05

       But I-- there is a lot of, uh, I would say, inertia in science, in academia. It is slowing down science.

    7. LFLex Fridman1:20:18

       Yeah, those two words: "your way." That's a powerful thing.

    8. JPJudea Pearl1:20:22

       Um, yes.

    9. LFLex Fridman1:20:23

       It's against inertia, potentially, against the flow.

    10. JPJudea Pearl1:20:26

        Against your professor.

    11. LFLex Fridman1:20:28

        (laughs) Against your professor.

    12. JPJudea Pearl1:20:29

        It is, uh... I- I wrote The Book of Why-

    13. LFLex Fridman1:20:32

        Yeah.

    14. JPJudea Pearl1:20:32

        ... in order to democratize common sense, yeah. (laughs)

    15. LFLex Fridman1:20:38

        (laughs)

    16. JPJudea Pearl1:20:38

        In order to instill rebellious spirits in students so they wouldn't wait until the professor get things right.

    17. LFLex Fridman1:20:51

        (laughs) Thus, you wrote the manifesto of the rebellion against the professor. (laughs)

    18. JPJudea Pearl1:20:58

        Against the professor, yes.

    19. LFLex Fridman1:21:00

        So, looking back at your life of research, what ideas do you hope ripple through the next many decades? What, what do you hope your legacy will be?

    20. JPJudea Pearl1:21:10

        I already have a tombstone, uh, carved. (laughs)

    21. LFLex Fridman1:21:19

        (laughs) Oh boy, yeah.

    22. JPJudea Pearl1:21:20

        The fundamen- the fundamental law of counterfactuals, that what, uh, it said. (laughs)

    23. LFLex Fridman1:21:27

        (laughs)

    24. JPJudea Pearl1:21:28

        It's a simple equation. What a counterfactual in terms of a model surgery.

    25. LFLex Fridman1:21:35

        Mm-hmm.

    26. JPJudea Pearl1:21:35

        That's it, because everything follows from that. If you get that, all the rest, I can die in peace, and my student can derive all my knowledge (laughs) by mathematical means.

    27. LFLex Fridman1:21:51

        The rest follows.

    28. JPJudea Pearl1:21:53

        Yeah.

    29. LFLex Fridman1:21:54

        Judea, thank you so much for talking today. I really appreciate it.

    30. JPJudea Pearl1:21:57

        Well, thank you for being so attentive and, uh, instigating. (laughs)

Episode duration: 1:23:01