The Alignment Problem - Brian Christian | Modern Wisdom Podcast 297

You have a system, you want it to do X, you give it a set of examples and you say, you know, "Do that, do this kind of thing." What could go wrong? Um, well, there's this laundry list of things that could go wrong.

(wind blowing) What does the quote, "Premature optimization is the root of all evil" mean?

Mm. So this line comes from Donald Knuth, who is one of the, uh, I think of him as kind of like the Yoda of computer science, um, just dispensing these, uh, these gems of wisdom. Um, and there are many, I think, I think many, like many aphorisms, you can take it in a number of different directions. Um, one of the ways that I think about it is, um, you know, a lot of, a lot of the way that we make progress in math and computer science is through models. You make a model that sort of approximates the, um, the phenomenon that you're trying to deal with. You know, there's a, a great quote from Peter Norvig, another, uh, one of these, uh, luminaries in, in computer science. He's quoting someone from NASA saying, "Our job was not to land on Mars. It was to land on the mathematical model of Mars provided to us by the geologists." Um, and so this idea that premature optimization is the root of all evil, um, I think if you, if you kind of mistake the map for the territory, so to speak, um, if you forget that there is a gap between your model and what the reality actually is, um, then you can commit yourself, um, to a set of assumptions that are later going to bite you. Um, and so this is the sort of thing that people who are worried about AI safety, uh, you know, this is what keeps them up at night.

What is the alignment problem? That's what we're going to be talking about today. We might as well define our terms.

Yeah, so the alignment problem, um, is this idea in AI and machine learning of the potential gap between what your intention is when you build an AI system or a machine learning system-

Mm-hmm.

... and the objective that the system has. Um, so it's the potential misalignment, so to speak, between, uh, your intention, your expectation, how you want the system to behave, and what that system ultimately ends up doing.

Why does it matter?

I mean, this is, this is a fear that has existed in computer science going back to at least 1960. So Norbert Wiener, the MIT cyberneticist was writing about this. Um, and, you know, he says, "If we, if we use to achieve some purpose a mechanical agency that we can't interfere with once we've started it, uh, then we had better be quite sure that the purpose that we put into the machine is the thing that we really want." Um, and I think a lot of people increasingly since like 2014, it has become more and more mainstream within the computer science community itself as one of the most significant challenges facing, uh, the field as we sort of enter this era of AI, um, that we may develop systems which, you know, we, with the best of intentions, try to encode some objective into the system. The system with, you know, sort of the best of intentions attempts to do what it thinks we want, um, but there's some fundamental misalignment and that results in, you know, whatev- whatever the harm may be, whether it's, um, you know, dark-skinned people not getting recognized by, um, you know, a facial recognition system, or disparities in the way that parole is being, um, dealt with, you know, at a societal level. Um, it could be self-driving cars that fail to recognize jaywalkers and so they kill anyone who's crossing in the middle of the street because there were no jaywalkers in their training data. We may actually throw society as a whole off the rails, um, by some system with, you know, enough power to shape the course of human civilization, but, uh, without the appropriate wisdom to, to know exactly what to be doing.