Stuart Russell: Long-Term Future of Artificial Intelligence | Lex Fridman Podcast #9

Lex Fridman and Stuart Russell on stuart Russell on Controlling Superhuman AI and Humanity’s Future Choices.

Lex FridmanhostStuart Russellguest

Dec 9, 20181h 26mWatch on YouTube ↗

EVERY SPOKEN WORD

140 min read · 27,773 words

0:00 – 1:59
Early chess programming: punch cards, tight compute budgets, and alpha-beta search
1. LFLex Fridman
  The following is a conversation with Stuart Russell. He's a professor of computer science at UC Berkeley and a co-author of a book that introduced me and millions of other people to the amazing world of AI, called Artificial Intelligence: The Modern Approach. So, it was an honor for me to have this conversation as part of MIT course on Artificial General Intelligence and The Artificial Intelligence podcast. If you enjoy it, please subscribe on YouTube, iTunes, or your podcast provider of choice, or simply connect with me on Twitter @lexfridman, spelled F-R-I-D. And now, here's my conversation with Stuart Russell. So you've mentioned in 1975, in high school, you've created one of your first AI programs that played chess.
2. SRStuart Russell
  Yeah.
3. LFLex Fridman
  Were you ever able to build a program that beat you at chess or another board game?
4. SRStuart Russell
  Uh, so my program never beat me at chess. I actually wrote the program at Imperial College, so I used to take the bus every Wednesday with a box of cards this big, uh, and shove them into the card reader, and they gave us eight seconds of CPU time. It took about five seconds to read the cards in and compile the code, so we had three seconds of CPU time, which was enough to make one move, you know, with a not very deep search, and then we would print that move out, and then we'd have to go to the back of the queue and wait to feed the cards in again.
5. LFLex Fridman
  How deep was the search?
6. SRStuart Russell
  (laughs)
7. LFLex Fridman
  What, are we talking about one move, two moves, three moves?
8. SRStuart Russell
  Uh, so, no, I think we got, uh, we got an eight-move, uh, eight, you know, depth eight, um, with alpha-beta, and we had some tricks of our own about, um, move ordering and some pruning of the tree, and...
9. LFLex Fridman
  But you were still able to beat that program?
10. SRStuart Russell
  Yeah, yeah. I, I was a reasonable chess player in my youth.
11. LFLex Fridman
  (laughs)
1:59 – 3:57
Meta-reasoning: deciding what to think about in enormous search trees
1. SRStuart Russell
  I did an Othello program, uh, and a backgammon program. So when I got to Berkeley, I worked a lot on what we call meta-reasoning, which really means reasoning about reasoning. In, in the case of a game-playing program, you need to reason about what parts of the search tree you're actually going to explore, because the search tree is enormous, uh, you know, bigger than the number of atoms in the universe, and, and, uh, the way programs succeed and the way humans succeed is by only looking at a small fraction of the search tree. And if you look at the right fraction, you play really well. If you look at the wrong fraction, if you waste your time thinking about things that are never gonna happen, the moves that no one's ever gonna make, then you're gonna lose 'cause you, you won't be able to figure out the right decision. So that question of how machines can manage their own computation, how they, how they decide what to think about is the meta-reasoning question. We developed some methods for doing that, and very simply, a machine should think about whatever thoughts are going to improve its decision quality. We were able to show that both for Othello, which is a standard two-player game, and, uh, for backgammon, which includes, uh, dice rolls, so it's a two-player game with uncertainty. For both of those cases, we could come up with algorithms that were actually much more efficient than the standard alpha-beta search, uh, which chess programs at the time were using, and that, those programs could beat me. And I think you can see the same basic ideas in AlphaGo and AlphaZero today. The way they explore the tree is using a form of meta-reasoning to select what to think about based on how useful it is to think about it.
3:57 – 6:01
AlphaGo’s two superpowers: position intuition + selective deep lookahead
1. LFLex Fridman
  Is there any insights you can describe without Greek symbols of how do we select which paths to go down?
2. SRStuart Russell
  There, there's really two kinds of learning going on. So, uh, uh, as you say, AlphaGo learns to evaluate board positions. So it can, it can look at a Go board, and it actually has probably a superhuman ability to instantly tell how promising that situation is.
3. LFLex Fridman
  Mm-hmm.
4. SRStuart Russell
  To me, the amazing thing about AlphaGo is not that it can beat the world champion with its hands tied behind its back, but, uh, the fact that if you stop it from searching altogether, so you say, "Okay, you're not allowed to do any thinking ahead, right? You can just consider each of your legal moves, and then look at the resulting situation and evaluate it." So what we call a depth-one search.
5. LFLex Fridman
  Mm-hmm.
6. SRStuart Russell
  So just the immediate outcome of your moves and decide if that's good or bad. That version of AlphaGo can still play at a professional level, right? And even professionals are sitting there for five, 10 minutes deciding what to do, and AlphaGo, in less than a second, can instantly intuit what is the right move to make based on its ability to evaluate positions. Um, and that is remarkable, um, because, you know, we don't have that level of intuition about Go. We actually have to think about the situation. So anyway, that capability that, um, AlphaGo has is one big part of why it beats humans.
7. LFLex Fridman
  Right.
8. SRStuart Russell
  The other big part is that it's able to look ahead 40, 50, 60 moves into the future.
9. LFLex Fridman
  Mm-hmm.
10. SRStuart Russell
  And, you know, if it was considering all possibilities 40 or 50 or 60 moves into the future, that would be, you know, 10 to the 200 possibilities, so way, way more than, you know, atoms in the universe.
11. LFLex Fridman
  Mm-hmm.
6:01 – 9:13
A practical intuition for search: promise, uncertainty, and changing your mind
1. SRStuart Russell
  And, and so on. So it's very, very selective about what it looks at.So, let me try to give you an intuition about how you decide what to think about. It's a combination of two things. One is, um, how promising it is.
2. LFLex Fridman
  Mm-hmm.
3. SRStuart Russell
  Right? So, if you're already convinced that a move is terrible, there's no point spending a lot more time convincing yourself that it's terrible.
4. LFLex Fridman
  Mm-hmm.
5. SRStuart Russell
  Uh, because it's probably not gonna change your mind. So, the- the real reason you think is because there's some possibility of changing your mind about what to do.
6. LFLex Fridman
  Mm-hmm.
7. SRStuart Russell
  Right? And it's that changing of mind that would result then in- in a better final action in the real world. So, that's the purpose of thinking, is to improve the final action in the real world. And so, if you think about a move that is guaranteed to be terrible, you can convince yourself it's terrible, you're still not gonna change your mind.
8. LFLex Fridman
  Mm-hmm.
9. SRStuart Russell
  Right? But on the other hand here, suppose you had a choice between two moves. One of them, you've already figured out is guaranteed to be a draw, let's say.
10. LFLex Fridman
  Mm-hmm.
11. SRStuart Russell
  And then the other one looks a little bit worse. Like it looks fairly likely that if you make that move, you're gonna lose. But there's still some uncertainty-
12. LFLex Fridman
  Mm-hmm.
13. SRStuart Russell
  ... about the value of that move. There's still some possibility that it will turn out to be a win.
14. LFLex Fridman
  Mm-hmm.
15. SRStuart Russell
  Right? Then it's worth thinking about that. So even though it's less promising on average than the other move, which is guaranteed to be a draw, there's still some purpose in thinking about it, because there's a chance that you'll change your mind and- and discover that in fact it's a better move. So, it's a combination of how good the move appears to be and how much uncertainty there is about its value. The more uncertainty, the more it's worth thinking about, because there's a higher upside, if you wanna think of it that way.
16. LFLex Fridman
  And of course, in the beginning, especially in the AlphaGo Zero formulation, it's, everything is shrouded in- in uncertainty. So, you're really swimming in a sea of, uh, uncertainty, so it- it benefits you to... I mean, actually following the same process as you described, but because you're so uncertain about everything, uh, y- you basically have to try a lot of different directions.
17. SRStuart Russell
  Yeah. So- so the- the early parts of the search tree, uh, are fairly bushy-
18. LFLex Fridman
  Mm-hmm.
19. SRStuart Russell
  ... um, that it w- it will look at a lot of different possibilities. But fairly quickly, the degree of certainty about some of the moves... I mean, if a move is really terrible, you'll pretty quickly find out, right? You'll lose half your pieces or half your territory, and, um, and then you'll say, "Okay, this- this is not worth thinking about anymore." And then, so further down, the tree becomes very long and narrow.
20. LFLex Fridman
  Mm-hmm.
21. SRStuart Russell
  And, uh, you're- you're following various lines of play, you know, 10- 10, 20, 30, 40, 50 moves into the future. And, um, you know, that again is something that- that human beings have a very hard time doing, uh, mainly because they just lack the short-term memory. You just can't remember a sequence of moves that's 50 moves long, uh, and you can't- you can't, uh, imagine the board correctly, uh, for that many moves into the future.
9:13 – 14:05
Human vs machine intuition in chess: patterns, forcing lines, and mistakes
1. LFLex Fridman
  Of course, the top players, I- I'm much more familiar with chess, but the top players probably have... they have echoes of the same kind of intuition and instinct that, in a moment's time, AlphaGo applies when they see a board. I mean, they've seen those patterns. Human beings have seen those patterns before at the top, at the grandmaster level. It seems that there's some, uh, similarities, or maybe it's- it's our imagination creates a- a vision of those similarities, but it feels like this kind of pattern recognition that the AlphaGo approaches are- are using is similar to what human beings at the top level are using.
2. SRStuart Russell
  I think there's, uh, there's some truth to that.
3. LFLex Fridman
  Oh, but not entirely?
4. SRStuart Russell
  Yeah. I- I mean, I think the- the extent to which a human grandmaster can reliably rec- instantly recognize the right move and instantly recognize the value of a position, uh, I think that's a little bit overrated.
5. LFLex Fridman
  But if you sacrifice a queen, for example, I mean, there's these- there's these beautiful games of chess with Bobby Fischer or somebody, where it's seeming to make a bad move, and I'm not sure there's a- a- a perfect degree of calculation involved, where they've calculated all the possible things that happen-
6. SRStuart Russell
  Mm-hmm.
7. LFLex Fridman
  ... but there's an instinct there, right? That somehow adds up to go down that path.
8. SRStuart Russell
  Th- yeah. So, I think what happens is you- you- you get a sense that there's some possibility in the position, uh, even if you make a weird-looking move, um, that it opens up some- some lines of- of calculation that otherwise would be, uh, definitely bad. And, um, and it's that intuition that there's something here in this position that might, uh, might yield a win.
9. LFLex Fridman
  Down the set of possibilities, yeah.
10. SRStuart Russell
  And then- and then you follow that, right? And- and in some sense, when a- when a chess player is following a line in- in his or her mind, they're- they're mentally simulating what the other person is gonna do, what the opponent is gonna do. And they can do that as long as the moves are kind of forced.
11. LFLex Fridman
  Mm-hmm.
12. SRStuart Russell
  Right? As long as there's a, you know, there's a- a for- what we call a forcing variation, where the opponent doesn't really have much choice how to respond, and then you see if you can force them into a situation where you win. You know, we see plenty of mistakes, uh, even- even in grandmaster games, where they just miss some simple three, four, five-move, uh, combination that, you know, wasn't particularly apparent in- in the position, but was still there.
13. LFLex Fridman
  That's the thing that makes us human.
14. SRStuart Russell
  Yep.
15. LFLex Fridman
  So, when... you mentioned that in Othello, those games were, after some, uh, meta-reasoning improvements and research, was able to beat you.... how did that make you feel?
16. SRStuart Russell
  Part of the meta-reasoning capability that it had, um, (clears throat) was based on learning.
17. LFLex Fridman
  Mm-hmm.
18. SRStuart Russell
  And, um, and you could sit down the next day and, and you could just feel that it had got a lot smarter. (clears throat)
19. LFLex Fridman
  Yeah.
20. SRStuart Russell
  You know, and all- and all of a sudden, you really felt like you were sort of pressed against the wall because-
21. LFLex Fridman
  Yeah.
22. SRStuart Russell
  ... uh, it was, it was much more aggressive and, and was totally unforgiving of, of any minor mistake that you might make. And, uh, and actually s- it seemed understood the game better, uh, than I did. And, you know, Garry Kasparov has this quote where, um, during his match against Deep Blue, he said he suddenly felt that there was a new kind of intelligence across the board.
23. LFLex Fridman
  Hmm.
24. SRStuart Russell
  So-
25. LFLex Fridman
  Do you think that's a scary or an exciting possibility-
26. SRStuart Russell
  Um.
27. LFLex Fridman
  ... for Garry Kasparov and for yourself? In, in the context of chess, purely sort of in this, like, that feeling, whatever that is.
28. SRStuart Russell
  I, I think it's definitely, um, an exciting feeling. You know, this is what made me work on AI in the first place was as soon as I really understood what a computer was, I wanted to make it-
29. LFLex Fridman
  Mm-hmm.
30. SRStuart Russell
  ... smart. You know, I started out with, uh, the first program I wrote was for the Sinclair Programmable Calculator.
14:05 – 16:55
From game AI to real-world AI: partial observability, uncertainty, and long time horizons
1. SRStuart Russell
  And I think the, the, the thing that's scary is not, is not the chess program because, you know, chess programs, they're not in the taking-over-the-world business.
2. LFLex Fridman
  (laughs)
3. SRStuart Russell
  Uh, but if you extrapolate, you know, there are things about chess that don't resemble the real world, right? We know, we know the rules of chess. The chess board is completely visible to the program, where of course the real world is not. Most y- most of the real world is, is not visible from wherever you're sitting, so to speak. And, uh, to overcome those kinds of problems, you need qualitatively different algorithms. Another thing about the real world is that, you know, we, we regularly plan ahead on the timescales involving billions or trillions of steps.
4. LFLex Fridman
  Mm-hmm.
5. SRStuart Russell
  Um, now we don't plan those in detail, but, you know, when you choose to do a PhD at Berkeley, that's a five-year commitment and that amounts to about a trillion motor control steps that you will eventually, uh, be committed to.
6. LFLex Fridman
  Including going up the stairs, opening doors, drinking water-
7. SRStuart Russell
  Type, yeah, I mean every-
8. LFLex Fridman
  ... typing. (laughs)
9. SRStuart Russell
  ... every, every finger movement while you're typing, every character of every paper and, you know-
10. LFLex Fridman
  Yeah.
11. SRStuart Russell
  ... thesis and everything else. So you're not committing in advance to the specific motor control steps, but you're still reasoning on a timescale that will eventually reduce to, uh, trillions of motor control actions. And, uh, so for all of these reasons, you know, AlphaGo and, and Deep Blue and so on don't represent any kind of threat to humanity, but they are a step towards it, right?
12. LFLex Fridman
  Yes.
13. SRStuart Russell
  That... And progress in AI occurs by essentially removing one by one-
14. LFLex Fridman
  Mm-hmm.
15. SRStuart Russell
  ... these assumptions that make problems easy, like the assumption of complete observability of the situation, right? If we remove that assumption, you need a much more complicated kind of computing design and you n- you need something that actually keeps track of all the things you can't see and tries to estimate what's going on, uh, and there's inevitable uncertainty, uh, in that. So it becomes a much more complicated problem. But, you know, we are removing those assumptions. We are starting to have algorithms that can cope with much longer timescales, that can cope with uncertainty, that can cope with partial observability. And so each of those steps sort of magnifies by 1,000 the range of things that we can do with AI systems.
16:55 – 20:40
Why Go felt different—and why AlphaGo was (slightly) disappointing as a roadmap
1. LFLex Fridman
  So the way I started in AI, I wanted to be a psychiatrist for a long time. I wanted to understand the mind in high school, and of course program and so on. And then I showed up, uh, University of Illinois to an AI lab and they said, "Okay, I don't have time for you, but here's a book, AI: A Modern Approach." I think it was the first edition at the time.
2. SRStuart Russell
  Mm-hmm.
3. LFLex Fridman
  (laughs) And, "Here, go, go, go learn this." And I remember the lay of the land was, well, it's incredible that we solved chess, but we'll never solve Go. I mean, it was pretty certain that Go in the way we thought about systems that reason was impossible to solve, and now we've solved it, so it's a very-
4. SRStuart Russell
  Well, I think I, I would have said that it's unlikely we could take the kind of algorithm that was used for chess and just get it to scale up, uh, and work well for Go. And at the time what we thought was that in order to solve Go, we would have to do something similar to the way humans manage the complexity of Go, which is to break it down into kind of sub-games. So-
5. LFLex Fridman
  Mm-hmm.
6. SRStuart Russell
  ... when a human thinks about a Go board, they think about different parts of the board as sort of weakly connected to each other, and they think about, "Okay, within this part of the board, here's how things could go. In that part of the board, here's how things could go."
7. LFLex Fridman
  Mm-hmm.
8. SRStuart Russell
  And then you try to sort of couple those two analyses together.... and deal with the interactions and maybe revise your views of how things are gonna go in each part, and then you've got maybe five, six, seven, 10 parts of the board.
9. LFLex Fridman
  Mm-hmm.
10. SRStuart Russell
  And, um, that actually resembles the real world much more than chess does-
11. LFLex Fridman
  Mm-hmm.
12. SRStuart Russell
  ... because in the real world, you know, we have work, we have home life, we have sport, you know, whatever... different kinds of activities, you know, shopping. These all are connected to each other but they're weakly connected. So, when I'm typing a paper, you know, I don't simultaneously have to decide which order I'm gonna get the, you know, the milk and the butter. You know, that doesn't affect the typing. But I do need to realize, "Okay, I better finish this before the shops close because I don't have anything... I don't have any food at home."
13. LFLex Fridman
  Right.
14. SRStuart Russell
  Right? So there's some weak connection, but not in the way that chess works where everything is tied into a single stream of thought. So the, the thought was that Go... to, to solve Go, we'd have to make progress on stuff that would be useful for the real world. And in a way, AlphaGo is a little bit disappointing-
15. LFLex Fridman
  (laughs) Right.
16. SRStuart Russell
  ... because the p- the program design for AlphaGo is actually not that different from, from Deep Blue or f- or even from Arthur Samuel's checker playing program from the 1950s. And in fact, the... so the two things that make AlphaGo work is one, one is its amazing ability, ability to evaluate the positions, and the other is the meta-reasoning capability which, which allows it to, to explore some paths in the tree very deeply and to abandon other paths very quickly.
17. LFLex Fridman
  So th- this word meta-reasoning, uh, while technically correct, inspires perhaps the, the wrong degree of power that AlphaGo has. For example, the word reasoning is a, is a powerful word. Let me ask you sort of... So you were part of the symbolic AI world for a while, like, where AI was, uh-
18. SRStuart Russell
  Mm-hmm.
19. LFLex Fridman
  There was a lot of excellent interesting ideas there that unfortunately met a winter. And (laughs) so, i- do you think it reemerges as if-
20:40 – 23:25
AI winters and hype cycles: expert systems’ failure modes and today’s risk of overpromising
1. SRStuart Russell
  Oh, so I, I would say... Yeah, it's not quite as simple as that. So the, the AI winter... The fir- the first winter that was actually named as such was the one in the late '80s, and that came about because, uh, in the mid-'80s, there was a... really a concerted attempt to push AI out into the real world, uh, using what, uh, was called expert system technology. And for the most part, that technology was just not ready for prime time. They were trying, uh, in many cases to do a form of uncertain reasoning-
2. LFLex Fridman
  Mm-hmm.
3. SRStuart Russell
  ... uh, judge, you know, judgment, combinations of evidence, diagnosis, those kinds of things, which was simply invalid. Uh, and when you try to apply invalid reasoning methods to real problems, you can fudge it for small versions of the problem, but when it starts to get larger, the thing just falls apart. So, many companies found that, uh, the stuff just didn't work and they were spending tons of money on consultants to try to make it work and there were, you know, other practical reasons like, you know, they, they were asking the companies to buy incredibly expensive Lisp machine, uh, workstations which were literally between $50,000 and $100,000 in, uh, you know, in 1980s money, which was... would be, like, between $150,000 and $300,000 per workstation in current prices, so-
4. LFLex Fridman
  And then the bottom line, they weren't, uh, seeing a profit from it?
5. SRStuart Russell
  Yeah, um, they, they... In many cases. I think there were some successes, there's no doubt about that. But people, I would say, overinvested. Every major company was starting an AI department, just like now, and I worry a bit that we might see similar disappointments, not because the technol- the current technology is invalid, but it's limited in its scope. And, uh, it's almost the, the dual of the, you know, the scope problems that expert systems had, so...
6. LFLex Fridman
  What have you learned from that hype cycle and what can we do to prevent another winter, for example?
7. SRStuart Russell
  Uh, yeah. So when I'm, I'm giving talks these days, that's one of the warnings that I give. Uh, so there's this two, two-part warning slide. One is that, uh, you know, rather than data being the new oil, data is the new snake oil.
8. LFLex Fridman
  (laughs)
9. SRStuart Russell
  Uh...
10. LFLex Fridman
  That's a good line.
11. SRStuart Russell
  And then, um...
12. LFLex Fridman
  (laughs)
23:25 – 26:22
Self-driving cars: perception progress, reliability math, and the edge-case wall
1. SRStuart Russell
  And then the other, uh, is that we might see a kind of very visible failure in some of the major application areas, and I think self-driving cars would be the flagship. And, uh, I think when you look at the history... So the first self-driving car was on the freeway, uh, driving itself, changing lanes, overtaking, in 1987. And, uh, so it's more than 30 years.
2. LFLex Fridman
  Yep.
3. SRStuart Russell
  And God, that kind of looks like where we are today, right? You know, prototypes on the freeway, changing lanes and overtaking.
4. LFLex Fridman
  Yep.
5. SRStuart Russell
  Um, now, I think significant progress has been made, particularly on the perception side. So, we worked a lot on autonomous vehicles in the early mid-'90s at Berkeley, you know, and we had our own big demonstrations, you know. We, we put congressmen into-
6. LFLex Fridman
  Yep.
7. SRStuart Russell
  ... self-driving cars and, and had them zooming along the freeway.And, uh, the problem was clearly perception, right?
8. LFLex Fridman
  At the time, the problem was the perception?
9. SRStuart Russell
  At the- Yeah. So w- you know, in simulation with perfect perception, you could actually show that you can drive safely for a long time even if the other car's misbehaving, and- and so on. But simultaneously, we worked on mach- machine vision for detecting cars and tracking pedestrians, and so on, and we couldn't get the reliability of detection and tracking up to a high enough, uh, particu- level, particularly in bad weather conditions, uh, nighttime, rain, fog.
10. LFLex Fridman
  Good enough for demos, but perhaps not good enough to cover the general ca- uh, the- the general operation (inaudible)
11. SRStuart Russell
  Yeah. So the thing about driving is, you know, so suppose you're a taxi driver, you know, and you drive every day, eight hours a day for 10 years, right? That's 100 million seconds of driving, you know? And any one of those seconds, you could make a fatal mistake.
12. LFLex Fridman
  Yeah.
13. SRStuart Russell
  So you're talking about eight-nines of reliability, right? Now, if your vision system only detects 98.3% of the vehicles, right? (laughs)
14. LFLex Fridman
  Hm.
15. SRStuart Russell
  Then that's sort of, you know, one and a bit nines of reliability.
16. LFLex Fridman
  Yeah.
17. SRStuart Russell
  So you have another seven orders of magnitude to go. And, um, a- and this is what people don't understand. They- they think, "Oh, because I had a successful demo, I'm pretty much done." But you're n- you're not even within seven orders of magnitude of being done, and that's the difficulty, and it's- it's not the, "Can I follow a white line?" That's not the problem, right? We can follow a white line all the way across the country. But it's the- it's the weird stuff that happens, it's, you know...
18. LFLex Fridman
  It's all the edge cases, yeah.
26:22 – 36:12
Driving is interactive: intent inference, game theory, and emergent communication behaviors
1. SRStuart Russell
  The edge case, other drivers doing weird things. Um, you know, so if you talk to Google, right? So they had, um, actually very classical architecture where, you know, you had machine vision which would detect all the other cars, and pedestrians, and the white lines, and the road signs, and then basically that was fed into a logical database.
2. LFLex Fridman
  Mm-hmm.
3. SRStuart Russell
  And then you had a classical 1970s rule-based expert system, um, telling you, "Okay, if you're in the middle lane, and there's a bicyclist in the right lane who is signaling this, then- then- then- then you do that."
4. LFLex Fridman
  Yeah.
5. SRStuart Russell
  Right? And what they found was that every day, they'd go out, and there'd be another situation that the rules didn't cover, you know? So they- they'd come to a traffic circle, and there's a little girl riding her bicycle the wrong way around the traffic circle. Okay, what do you do? "We don't have a rule." "Oh my god, okay, stop." And then you- you know, they'd come back and add more rules, and they just found that this was not really converging. And, um... And if you think about it, right, how d- how do you deal with an unexpected situation? Meaning one that you've never previously encountered and the sort of the- the reasoning required to figure out the solution for that situation has never been done, and so it doesn't match any, uh, previous situation in terms of the kind of reasoning you have to do? Well, you know, in chess programs this happens all the time, right? You're constantly coming up with situations you haven't seen before-
6. LFLex Fridman
  Hmm. That's right.
7. SRStuart Russell
  ... and you have to reason about them, and you have to think about, "Okay, here are the possible things I could do, here are the outcomes, here's how desirable the outcomes are," and then pick the right one. You know, in the '90s we were saying, "Okay, this is how you're gonna have to do automated vehicles. They're gonna have to have lookahead capability." But the lookahead for driving is more difficult than it is for chess because-
8. LFLex Fridman
  'Cause of humans.
9. SRStuart Russell
  ... the other... Right, there's humans, and they are less predictable than a op-
10. LFLex Fridman
  Than chess pieces.
11. SRStuart Russell
  ... than a... Well, than w- You have an opponent in chess who's also somewhat unpredictable.
12. LFLex Fridman
  Hm.
13. SRStuart Russell
  But for example, in chess, you always know the opponent's intention. They're trying to beat you, right? Whereas in driving, you don't know, is this guy trying to turn left? Or has he just forgotten to turn off his turn signal?
14. LFLex Fridman
  Mm-hmm.
15. SRStuart Russell
  Or is he drunk? Or is he, you know, changing the channel on his radio? Or whatever it might be, you gotta try and figure out the mental state, the intent of the other-
16. LFLex Fridman
  Mm-hmm.
17. SRStuart Russell
  ... drivers to forecast the possible evolutions of their trajectories, and then you gotta figure out, "Okay, which is the trajectory for me that's going to be safest?" And those all interact with each other-
18. LFLex Fridman
  Mm-hmm.
19. SRStuart Russell
  ... because the other drivers are gonna react to your trajectory, um, and so on. So, you know, they've got the classic merging onto the freeway problem where you're kind of racing a vehicle that's already on the freeway, and you're... are you gonna pull ahead of them? Or are you gonna let them go first and pull in behind? And- and you get this sort of uncertainty about who's going first. So all those kinds of things mean that you need a decision-making architecture that's very different from either a rule-based system or, it seems to me, a- kind of an end-to-end neural network system. You know, so just as AlphaGo is pretty good when it doesn't do any lookahead, but it's way, way, way, way better when it does, I think the same is gonna be true for driving. You can have a driving system that's pretty good when it doesn't do any lookahead, but that's not good enough. Um, you know, and we've already seen multiple deaths caused by poorly designed, uh, machine learning algorithms that don't really understand what they're doing.
20. LFLex Fridman
  Yeah, on- on several levels. I think, uh, so on the perception side, uh, there's mistakes being made by those algorithms where the perception is very shallow. On the planning side, the lookahead like you said, and the thing that we come- come up against that's really interesting when you try to deploy systems in the real world is...... you can't think of an artificial intelligence system as a thing that responds to the world always. You have to realize that it's an agent that others will respond to as well. So, in order to drive successfully, you can't just try to do obstacle avoidance.
21. SRStuart Russell
  Right.
22. LFLex Fridman
  You-
23. SRStuart Russell
  You can't pretend that you're invisible. (laughs) Right?
24. LFLex Fridman
  (laughs)
25. SRStuart Russell
  You're the invisible car.
26. LFLex Fridman
  Right. So-
27. SRStuart Russell
  Uh, it doesn't work that way.
28. LFLex Fridman
  I mean, but y- you have to assert... y- others have to be scared of you. Just, we're all s- there's this tension. There's this game... So if w- we study a lot of work with pedestrians.
29. SRStuart Russell
  Mm-hmm.
30. LFLex Fridman
  If you approach pedestrians as purely an obstacle avoidance, so you're, you're doing look ahead as in modeling the intent, then you're, you're- they're not going to... they're going to take advantage of you. They're not going to respect you at all. There has to be a tension, a fear, a some amount of uncertainty. That's how we have creat- we keep on-
36:12 – 52:19
The control problem: why fixed objectives fail and ‘humble’ AI is necessary
1. LFLex Fridman
  But you've often talk about, uh, concerns you have of how things may go, uh, wrong. So, uh, I've talked to, uh, Max Tegmark, uh, there's a lot of interesting ways to think about AI safety. You're one of the m- seminal people thinking about this problem among sort of being in the weeds of actually, uh, solving specific AI problems. You're also thinking about the big picture of where we're going.
2. SRStuart Russell
  Mm-hmm.
3. LFLex Fridman
  So-Can you talk about several elements of it? Let's just talk about maybe the control problem, so this idea of losing ability to control the behavior in a- of a AI system. So w- how do you see that? How do you see that coming about? What do you think we can do, uh, to manage it?
4. SRStuart Russell
  Well, so it- it doesn't take a genius to realize that if you make something that's smarter than you, you might have a problem (laughs) . You know, and Turing, uh, Alan Turing, you know, wrote about this and gave lectures about this, you know, in, I think, 1951. He did a lecture on the radio, and, uh, he basically says, you know, once the machine thinking method starts, uh, you know, very quickly, they'll outstrip humanity. And, uh, you know, if we're lucky, we might be able to... I think he says i- if... "We may be able to turn off the power at strategic moments, but even so, our species would be humbled."
5. LFLex Fridman
  Yeah, you-
6. SRStuart Russell
  And actually, I think he was wrong about that, right? (laughs) 'cause you- 'cause you got- you know, if it's a sufficiently intelligent machine, it's not gonna let you switch it off. Uh-
7. LFLex Fridman
  So, uh-
8. SRStuart Russell
  ... it's actually in competition with you.
9. LFLex Fridman
  So, what do you think is meant, just for a quick tangent, if we shut off this super intelligent machine, that our species would be humbled?
10. SRStuart Russell
  I think he means that we would realize that we are inferior, right? That we- we only survive by the skin of our teeth because we happened to get to the off switch (laughs) , you know, just in-
11. LFLex Fridman
  That was a close call.
12. SRStuart Russell
  ... just in time, uh, you know?
13. LFLex Fridman
  Yeah.
14. SRStuart Russell
  And if- and if we hadn't, then, uh, we would have lost control over the Earth.
15. LFLex Fridman
  So, do you... Are you more worried when you think about this stuff about super intelligent AI, or are you more worried about super powerful AI that's not aligned with our values? So, the paperclip, uh, scenario is kind of-
16. SRStuart Russell
  Mm-hmm.
17. LFLex Fridman
  ... uh-
18. SRStuart Russell
  I think, um... So, the main problem I'm working on is- is the control problem, the- the problem of machines pursuing objectives that are, as you say, not aligned with human objectives. And- and this has been- this has been the way we've thought about AI since the beginning. You- you build a machine for optimizing, and then you put in some objective, and it optimizes, right? And- and, um, you know, we- we can think of this as the- the King Midas problem, right? 'Cause if, you know, so King Midas put in this objective, right? "Everything I touch should turn to gold." And the gods, you know, that's like the machine, they said, "Okay, done." You know, "You now have this power." And of course, his food and his drink and his family all turn to gold, and then he dies of misery and starvation. And, um, this is... You know, it's- it's a warning. It's- it's a failure mode that pretty much every culture, uh, in history has had some story along the same lines. You know, there's the- the genie that gives you three wishes and-
19. LFLex Fridman
  Yeah.
20. SRStuart Russell
  ... you know, the third wish is always, you know, "Please undo the first two wishes 'cause I messed up." And, uh, you know, and when Arthur Samuel wrote his chess, uh, his checker-playing program, which learned to play checkers considerably better than Arthur Samuel could play and actually reached a pretty decent standard, uh, Norbert Wiener, who was a- one of the major mathematicians of the 20th century, he's sort of the father of modern automation control systems, you know, he saw this and he basically extrapolated, uh, you know, as Turing did and said, "Okay, uh, this is how we could lose control." And, uh, specifically that we have to be certain that the purpose we put into the machine is the purpose which we really desire. And the problem is we can't do that, right?
21. LFLex Fridman
  You- you mean we're not... it's, uh, very difficult, uh, to encode, to- to- to put our values on paper is really difficult? Or are you just saying it's impossible?
22. SRStuart Russell
  Uh, theor- (laughs)
23. LFLex Fridman
  (laughs) The line is gray between the two.
24. SRStuart Russell
  So- so, th- theoretically, it's possible but, uh, i- in practice, it's extremely unlikely that we could specify correctly in advance the full range of concerns of humanity. Uh, yeah.
25. LFLex Fridman
  The... You talked about, uh, cultural transmission of values I think is how humans-to-human transmission of values happens, right? Uh...
26. SRStuart Russell
  Well, we learn... You know, I mean, w- as we grow up, we learn about the values that matter, how things- how things should go, what is reasonable to pursue and what isn't reasonable to pursue. Uh, and it's-
27. LFLex Fridman
  Do you think machines can learn in the same kind of way?
28. SRStuart Russell
  Yeah, so I- I think that, um, what we need to do is to get away from this idea that you build an optimizing machine and then you put the objective into it because if it's possible that you might put in a wrong objective, and we already know this is possible 'cause it's happened lots of times-
29. LFLex Fridman
  Mm-hmm.
30. SRStuart Russell
  ... right? That means that the machine should never take an objective that's given as gospel truth because once it takes the mach- the- the objective as gospel truth, all right, then it believes that whatever actions it's taking in pursuit of that objective are the correct things to do. So, you could be jumping up and down and saying, "N- you know, no, no, no, you're gonna destroy the world!" But the machine knows what the true objective is and it's pursuing it.
52:19 – 1:07:27
Scalability and oversight: from pharma regulation to social media optimization and deepfakes
1. SRStuart Russell
  But when we look at AI systems or even just computers in general, right? They have this different characteristic from ordinary human activity in the past. So let's say you were a surgeon, you had some idea about how to do some operation-
2. LFLex Fridman
  Mm-hmm.
3. SRStuart Russell
  ... right? Well, and- and let's say you were wrong, right?That that way of doing the operation would mostly kill the patient.
4. LFLex Fridman
  Mm-hmm.
5. SRStuart Russell
  Well, you'd find out pretty quickly, like after three, maybe three or four tries, right? But that isn't true for pharmaceutical companies because they don't do three or four operations, they- they manufacture three or four billion pills and they sell them and then they find out maybe six months or a year later that, oh, people are dying of heart attacks or getting cancer from this drug. And so that's why we have the FDA, right? Because of the scalability of pharmaceutical production. And, you know, and there have been some unbelievably bad episodes in the history of pharmaceuticals and- and adulteration of- of products and so on that- that have killed tens of thousands or paralyzed hundreds of thousands of people. Now with computers, we have that same scalability problem that you can sit there and type for I equals one to five billion do, right?
6. LFLex Fridman
  Yeah.
7. SRStuart Russell
  And all of a sudden you're having an impact on a global scale. And yet we have no FDA, right? There's absolutely no controls at all over what a bunch of undergraduates with too much caffeine can do to the world. And, uh, you know, we look at what happened with Facebook, well social media in general, and click-through optimization. So you have a simple feedback algorithm that's trying to just optimize click-through, right?
8. LFLex Fridman
  Mm-hmm.
9. SRStuart Russell
  That sounds reasonable, right? Because you don't want to be feeding people ads that they don't care about and not interested in. And you might even think of that process as simply adjusting the- the feeding of ads or news articles or whatever it might be to match people's preferences, right? Which sounds like a good idea. But in fact, that isn't how the algorithm works, right? You make more money... the algorithm makes more money if...... it can better predict what people are going to click on, 'cause then it can feed them exactly that, right? So the way to maximize click-through is actually to modify the people-
10. LFLex Fridman
  Right.
11. SRStuart Russell
  ... to make them more predictable. Uh, and one way to do that is to feed them information which will change their behavior and preferences towards extremes that make them predictable. You know, whatever is the nearest extreme or the nearest predictable point, uh, that's where you're going to end up. And the machines will force you there. Now, and- and I think there's a reasonable argument to say that this, among other things, is contributing to the destruction of democracy in the world. And where was the oversight of this process? Where were the people saying, "Okay, you would like to apply this algorithm to five billion people on the- on the face of the Earth. Can you show me that it's safe? Can you show me that it won't have various kinds of negative effects?" No, there was no one asking that question. There was no one placed between, you know, the- the undergrads with too much caffeine and the human race.
12. LFLex Fridman
  Well, it's-
13. SRStuart Russell
  It's just they just did it.
14. LFLex Fridman
  And s- but some, uh, w- way outside the scope of my knowledge, so economists would argue that the, what- what is it? The invisible hand, so the- the capitalist system, it was the oversight. So if you're going to corrupt society with whatever decision you make as a company, then that's going to be reflected in people not using your product. S- sort of one-
15. SRStuart Russell
  (laughs)
16. LFLex Fridman
  That's one model of oversight. So...
17. SRStuart Russell
  We- we shall see, but, you know-
18. LFLex Fridman
  We shall see, yeah.
19. SRStuart Russell
  ... in the meantime, you know, that, but you- you might even have broken the, uh, the political system that enables capitalism to function.
20. LFLex Fridman
  Well, you've changed it. (laughs) So-
21. SRStuart Russell
  We shall see, yeah.
22. LFLex Fridman
  Cha- change is, uh, often painful. So my- my question is, uh, absolutely, it's f- it's fascinating. You're absolutely right that there was zero oversight on algorithms that could have a profound, civilization-changing, uh, effect. So do you think it's possible? I mean, I'm- I haven't... Have you seen government? So do you think it's possible to create regulatory bodies, uh, oversight over AI algorithms, which are inherently such cutting edge set of, uh, ideas and technologies?
23. SRStuart Russell
  Yeah, but I think it takes time to figure out what kind of oversight, what kinds of controls. I mean, it took time to design the FDA regime. Uh, you know, and some people still don't like it and they wanna fix it. Um, and I think there are clear ways that it could be improved. But the whole notion that you have stage one, stage two, stage three, and here are the criteria for what you have to do to pass a stage one trial, right?
24. LFLex Fridman
  Yes.
25. SRStuart Russell
  We haven't even thought about what those would be for algorithms. So, I mean, I- I think there are, there are things we could do right now with regard to bias, for example. We- we have a pretty good technical handle on, uh, how to detect algorithms that are propagating bias that exists in datasets, um, how to de-bias those algorithms.
26. LFLex Fridman
  Mm-hmm.
27. SRStuart Russell
  Uh, and- and even what it's gonna cost you to do that. So I think we could start having some standards, um, on that. I think there are, there are things to do with impersonation and falsification that we could, we could work on. So, uh...
28. LFLex Fridman
  Deepfakes, yep.
29. SRStuart Russell
  Uh...
30. LFLex Fridman
  Or, uh, uh, the in-
1:07:27 – 1:14:30
Why researchers downplay existential risk: motivated cognition, nuclear history echoes, and timelines
1. SRStuart Russell
  Mm-hmm. I, I think there is a, you know, a kind emot- motivated cognition, which is a... A term in psychology means that you believe what you would like to be true, uh, rather than what is true. And, uh, you know, it's- it's unsettling to think that what you're working on might be the end of the human race, obviously. So you would rather instantly deny it and come up with some reason why it couldn't be true. And, uh, you know, I have... I collected a, a long list of reasons that extremely intelligent, competent AI scientists have come up with for why we shouldn't worry about this. You know, for example, calculators are superhuman at arithmetic and they haven't taken over the world, so there's nothing to worry about.
2. LFLex Fridman
  (laughs)
3. SRStuart Russell
  Well, okay, my five-year-old, you know, could have figured out why that was, uh, uh, an unreasonable (laughs) and, and really quite weak argument. Um, you know, another one was, uh, you know... W- while it's theoretically possible that you could have a superhuman AI destroy the world, you know, it's also theoretically possible that a black hole could materialize-
4. LFLex Fridman
  Mm-hmm.
5. SRStuart Russell
  ... right next to the earth and destroy h- uh, humanity. I mean, yes, it's theoretically possible, quantum theoretically, extremely unlikely that it would just materialize right there. Um, but that's a completely bogus analogy because, you know, if the whole physics community on Earth was working to materialize a black hole in near-Earth orbit, right? Wouldn't you ask them, "Is that a good idea?"
6. LFLex Fridman
  (laughs)
7. SRStuart Russell
  "Is that gonna be safe?" You know, "What if you succeed?"
8. LFLex Fridman
  Right.
9. SRStuart Russell
  Right? And that's the thing, right? The AI community has sort of refused to ask itself, what if you succeed?
10. LFLex Fridman
  Yeah.
11. SRStuart Russell
  And initially, I think that was because it was too hard, but, you know, Alan Turing asked himself that and he said we'd be toast.
12. LFLex Fridman
  (laughs)
13. SRStuart Russell
  Right? If we were lucky, we might be able to switch off the power, but probably we'd be toast.
14. LFLex Fridman
  But there's also an aspect that because we're not exactly sure what the future holds, it's not clear exactly, so technically, what to worry about, sort of how-
15. SRStuart Russell
  Mm-hmm.
16. LFLex Fridman
  ... things go wrong. And so, uh, there is something, it feels like, maybe you can correct me if I'm wrong, but th- there's something paralyzing about worrying about something that logically is inevitable, but you don't really know what that will b- look like.
17. SRStuart Russell
  Yeah, I think that's, that's... Uh, it's a reasonable point and, you know, the... You know, it's certainly, in terms of existential risks, it's different from, you know, asteroid collides with the Earth.
18. LFLex Fridman
  Right.
19. SRStuart Russell
  Right? Which, again, is quite possible. Uh, you know, it's happened in the past. Uh, it'll probably happen again. We don't right- we don't know right now, but if we did detect an asteroid that was going to hit the Earth in 75 years time, we'd certainly be doing something about it.
20. LFLex Fridman
  Well, it's clear there's a big rock and there's-
21. SRStuart Russell
  (laughs)
22. LFLex Fridman
  We'll probably have a meeting and see what do we do about the big rock. With AI...
23. SRStuart Russell
  Right. With AI, I mean, there are very few people who think it's not gonna happen within the next 75 years. I know Rod Brooks doesn't think it's gonna happen. Uh, maybe Andrew Ng doesn't think it's happening. But, you know, a l- a lot of the people who work day to day, uh, you know, as you say, at the rock face, they think it's gonna happen. I think the median estimate from AI researchers is somewhere in 40 to 50 years from, from now or maybe even a little... You know, I think in Asia, they think it's gonna be even faster than that. I am, I am a little bit more conservative. I think it probably take longer than that. But I think it's, you know, as happened with nuclear weapons, um-
24. LFLex Fridman
  It could happen overnight.
25. SRStuart Russell
  ... it can happen overnight, that you have these breakthroughs. And we need more than one breakthrough but, you know, the... It's on the order of half a dozen. I mean, this is a very rough scale, but sort of half a dozen breakthroughs of that nature would have to happen for us to reach the- superhuman AI. But the, you know, the AI research community is vast now. The massive investments from governments, from corporations, uh, tons of really, really smart people. You know, you just have to look at the rate of progress in different areas of AI to see that things are moving pretty fast. So, so to say, "Oh, it's just gonna be thousands of years," I mean, I- I don't see any basis for that. You know, I see, you know, for example, the, the, the Stanford hundred year AI project, right? Which, um, is supposed to be sort of, you know, the serious establishment view. Uh, their most recent report actually said it's probably not even possible.
26. LFLex Fridman
  Oh, wow.
27. SRStuart Russell
  Right? Which if, if you want a perfect example of people in denial, that's it. Because, you know, for the whole history of AI, we've been saying to philosophers who said it wasn't possible, "Well, you have no idea what you're talking about. Of course it's possible," right? "Give me an, give me an argument for why it couldn't happen." And there isn't one, right? And now because people are worried that maybe AI might get a bad name or, or I just don't want to think about this, they're saying, "Okay, well, of course it's not really possible," you know? And we imagine, right? Imagine if, you know, the, y- the leaders of the cancer biology community...... uh, got up and said, "Well, you know, of course, curing cancer, it's not really possible." (laughs)
28. LFLex Fridman
  (laughs)
29. SRStuart Russell
  (laughs) It would be a complete outrage-
30. LFLex Fridman
  (laughs)
1:14:30 – 1:21:40
Three failure modes and ‘provably beneficial’ machines: control, misuse, and overuse (WALL-E)
1. SRStuart Russell
  ...not make the AI." And I think that's really hard to do, to, uh... And I'm not actually proposing that that's a feasible course of action. Um, and I w- I also think that, you know, if properly controlled, AI could be incredibly beneficial. So the... But it seems to me that there's a, there's a consensus that one of the major failure modes is this loss of control, that we create AI systems that are pursuing incorrect objectives, and because the AI system believes it knows what the objective is, it has no incentive to listen to us anymore, so to speak, right? It, it's just carrying out the, the strategy that it, it has computed as being the optimal solution. And, uh, you know, it may be that, in the process, it needs to acquire more resources to increase the possibility of success or, you know, prevent various failure modes by defending itself against interference. And so that collection of problems, I think, is something we can address.
2. LFLex Fridman
  Yes.
3. SRStuart Russell
  Uh, the other problems, uh, are, roughly speaking, you know, misuse, right? So even if we solve the control problem, we make perfectly safe, controllable AI systems, well, why? You know, why does Dr. Evil going to use those, right? He wants to just take over the world, and he'll make unsafe AI systems that, that then get out of control. So that's one problem which is sort of a, you know, uh, partly a policing problem, partly a, a sort of a cultural problem for the profession of how we teach people, uh, what kinds of AI systems are safe.
4. LFLex Fridman
  You talk about autonomous weapons system and how pretty much everybody agrees that there's too many ways that that can go horribly wrong, and this great, uh, Slaughterbots movie that kinda illustrates that beautifully.
5. SRStuart Russell
  (laughs)
6. LFLex Fridman
  And it-
7. SRStuart Russell
  Well, I want to talk a- that, that's another, there's another topic I, I'm happy to talk about. The... Just wanna mention the th- what I see as the third major failure mode, which is overuse. Not so much misuse, but overuse of AI, that we become overly dependent. So I call this the WALL-E problem. So if you've seen WALL-E-
8. LFLex Fridman
  Yeah.
9. SRStuart Russell
  ... the movie, all right, all the humans are on the spaceship, and the machines look after everything for them, and they just watch TV and drink Big Gulps. And, uh, they're all sort of obese and stupid and, and they've sort of totally lost any notion of human autonomy. And, um, you know, so i- i- in effect, right, this would happen like the slow-boiling frog, right? We would gradually turn over more and more of the management of our civilization to machines, as we are already doing.
10. LFLex Fridman
  Mm-hmm.
11. SRStuart Russell
  And this, you know, this, if this process continues, you know, we, we sort of gradually switch from sort of being the masters of technology to just being the guests. Right? So, so we become guests on a cruise ship, you know, which is fine for a week, but not f- not for the rest of eternity.
12. LFLex Fridman
  Right.
13. SRStuart Russell
  You know, and it's almost irreversible, right? Once you, once you lose the incentive to, for example, you know, learn to be an engineer or a doctor or a sanitation, uh, operative or, or any other of the, uh, uh, the infinitely many ways that we maintain and propagate our civilization. You know, if you, if you don't have the incentive to do any of that, you won't, and then it's really hard to recover.
14. LFLex Fridman
  And of course, AI is just one of the technologies that could... That third failure mode result in that. There's probably other... Technology in general detaches us from, um...
15. SRStuart Russell
  It does a bit, but the, the, the-
16. LFLex Fridman
  ...
17. SRStuart Russell
  (...) independence. ...difference is that in terms of the knowledge to, to run our civilization, you know-
18. LFLex Fridman
  Ah.
19. SRStuart Russell
  ...up to now, we've had no alternative but to put it into people's heads.
20. LFLex Fridman
  Right.
21. SRStuart Russell
  Right? And if you, if you-
22. LFLex Fridman
  But with software, with Google, I mean, so software in general, so AI broadly defined
23. SRStuart Russell
  So, so compute- computers in general-
24. LFLex Fridman
  Yeah.
25. SRStuart Russell
  ...but, but the, you know, the knowledge o- of how, you know, how a sanitation system works, you know, that's an... The AI has to understand that. It's no good putting it into Google. So, I mean, we've, we've always put knowledge in- on paper, but paper doesn't run our civilization. It only runs when it goes from the paper into people's heads again. Right? So we've always propagated civilization-... through human minds.
26. LFLex Fridman
  Mm-hmm.
27. SRStuart Russell
  And we've spent about a trillion person years doing that.
28. LFLex Fridman
  (laughs)
29. SRStuart Russell
  I, I... Literally, right?
30. LFLex Fridman
  Trillion.
1:21:40 – 1:26:05
Closing reflections: intellectual burden, self-skepticism, and favorite AI sci-fi robots
1. LFLex Fridman
  So there's very few people that represent artificial intelligence more than you, Stuart Russell. (laughs)
2. SRStuart Russell
  If you say so, okay. That's very kind. (laughs)
3. LFLex Fridman
  So-
4. SRStuart Russell
  So it's all my fault. Is that what you're saying?
5. LFLex Fridman
  It's all your fault. (laughs)
6. SRStuart Russell
  Okay, good.
7. LFLex Fridman
  No, right. Um, i- y- you're often brought up as the person, "Well, Stuart Russell, like, the AI person is worried about this, that's why you should be worried about it." Do you feel the burden of that? I don't know if you feel that at all. But when I talk to people, like from... Uh, y- you talk about s- people outside of computer science-
8. SRStuart Russell
  Mm-hmm.
9. LFLex Fridman
  ... when they think about this, "Stuart Russell, uh, is worried about AI safety, you should be worried too," do you feel the burden of that?
10. SRStuart Russell
  I mean, in a practical sense, uh, yeah, because, uh, I get, uh, you know, a dozen sometimes 25 invitations a day to talk about it, to give interviews, to write press articles, and so on. So, um, in that very practical sense, I'm seeing that people are concerned and really interested about this, um.
11. LFLex Fridman
  Are you worried that you could be wrong, as all good scientists are?
12. SRStuart Russell
  Of course. I worry about that all the time. I mean, that's, that's always been the way that I've, I've worked, you know, is like I, I have an argument in my head with myself, right?
13. LFLex Fridman
  (laughs)
14. SRStuart Russell
  So I have, I have some idea and then I think, "Okay, how could that be wrong? Or did someone else already have that idea?" So I'll go and, you know, search in as much literature as I can to f- to see whether someone else already thought of that or, or even refuted it. So, you know, I... Right now, I'm, I'm reading a lot of philosophy because, you know, in, in the form of the debates over, over utilitarianism and, and-
15. LFLex Fridman
  Mm-hmm.
16. SRStuart Russell
  ... other kinds of moral, uh, moral formulas, shall, shall we say, people have already thought through some of these issues. But, you know, one, one of the things I'm, I'm not seeing in a, in a lot of these debates is, is this specific idea about, uh, the importance of uncertainty in the objective.
17. LFLex Fridman
  Mm-hmm.
18. SRStuart Russell
  Um, that this is the way we should think about machines that are beneficial to humans, so this idea of, of provably beneficial machines based on, uh, explicit uncertainty in the objective. Um, you know, it seems to be... Y- you know, my, my gut feeling is this is the core of it. It's gonna have to be elaborated in a lot of different directions and there are a lot of-
19. LFLex Fridman
  Provably beneficial.
20. SRStuart Russell
  Yeah. But there, there... I, I mean, it has to be, right?
21. LFLex Fridman
  Right.
22. SRStuart Russell
  We, we can't afford, you know, hand-wavy beneficial-
23. LFLex Fridman
  Yeah.
24. SRStuart Russell
  ... uh, because there are-
25. LFLex Fridman
  Booms.
26. SRStuart Russell
  You know, whenever we do hand-wavy stuff, there are loopholes. And the thing about super intelligent machines is they find the loopholes, you know, just like, you know, tax evaders. Uh, if you don't write your tax law properly, (laughs) the people will find the loopholes and, and end up paying no tax, and, and, uh, so you should think of it this way and, and getting those definitions right, you know, it is really a long process, you know. So you can, you can define mathematical frameworks and within that framework you can prove mathematical theorems that, yes, this will... You know, this, this theoretical entity will be provably beneficial to that theoretical entity-
27. LFLex Fridman
  Mm-hmm.
28. SRStuart Russell
  ... but that framework may not match the real world in some crucial way. Um-
29. LFLex Fridman
  So it's a long process, thinking through it to iterating and so on.
30. SRStuart Russell
  Yeah.

Episode duration: 1:26:20

Install uListen for AI-powered chat & search across the full episode — Get Full Transcript

Transcript of episode KsZI5oXBC0k

Get more out of YouTube videos.

High quality summaries for YouTube videos. Accurate transcripts to search & find moments. Powered by ChatGPT & Claude AI.

iOS

Android

Claude

Chrome

Early chess programming: punch cards, tight compute budgets, and alpha-beta search

Meta-reasoning: deciding what to think about in enormous search trees

AlphaGo’s two superpowers: position intuition + selective deep lookahead

A practical intuition for search: promise, uncertainty, and changing your mind

Human vs machine intuition in chess: patterns, forcing lines, and mistakes

From game AI to real-world AI: partial observability, uncertainty, and long time horizons

Why Go felt different—and why AlphaGo was (slightly) disappointing as a roadmap

AI winters and hype cycles: expert systems’ failure modes and today’s risk of overpromising

Self-driving cars: perception progress, reliability math, and the edge-case wall

Driving is interactive: intent inference, game theory, and emergent communication behaviors

The control problem: why fixed objectives fail and ‘humble’ AI is necessary

Scalability and oversight: from pharma regulation to social media optimization and deepfakes

Why researchers downplay existential risk: motivated cognition, nuclear history echoes, and timelines

Three failure modes and ‘provably beneficial’ machines: control, misuse, and overuse (WALL-E)

Closing reflections: intellectual burden, self-skepticism, and favorite AI sci-fi robots

Get more out of YouTube videos.