Are We Headed For AI Utopia Or Disaster? - Nick Bostrom

1. 0:00 – 3:20

   Is Nick Hopeful About AI?

   1. CWChris Williamson0:00

      It seems like your book arc has been moving from what if things go wrong to what if things go right? Is this some requisite hope in the AI discussion?

   2. NBNick Bostrom0:14

      Uh, well, I think both barrels have always been there. Uh, it's like last time I, uh, published a book, it came out of one of the barrels, the kind of doom side. But, uh, (laughs) uh, I, no, I thi- I think, uh, uh, yeah, both the optimist and the pessimist are kind of co-inhabiting, uh, this brain.

   3. CWChris Williamson0:34

      Is that a, uh, is that a difficult balance to strike? The fact that you need to be so chronically aware of the dangers and so chronically aware of the potential successes as well?

   4. NBNick Bostrom0:49

      I think that's just a predicament, um, that we are in. Um, and if you look at the distribution of opinions, sort of roughly half fall on one side and half the other, but in many cases, I think it's basically just reflects the personality of the person holding the views, uh, ra- rather than some kind of evidence-derived opinion about, you know, the game board. And so, um, yeah, if- if- if one takes a good hard look at where we are with respect to things, I think one soon realizes just how ignorant we are about a lot of the key pieces here and how- how this thing works. So, um, so certainly one can see quite clearly, uh, significant risks and in particular, w- with this rapid advance that we're seeing in AI, um, including I think some existential risks. But, um, at the same time, if things go well, they could go really well. And I think that as long as there is ignorance, there is hope, so we have a lot of ignorance and it's also some hope.

   5. CWChris Williamson2:01

      It's interesting that, uh, your position, whether you're a AI doomer or an accelerationist or whatever, uh, is at least in part just a projection of your own sort of internal bias and mental texture that you sort of see in AI development, uh, the way that you see the world.

   6. NBNick Bostrom2:25

      I think there's clearly, uh, a- a good deal of that and then which tribe you happen to, uh, belong to. Like, depending on who you run into, what or which Twitter tr- threads you follow, like, uh, then we- we are kind of herd animals and sometimes, uh, it almost becomes a competition who has kind of developed the most hardcored-

   7. CWChris Williamson2:48

      Mm.

   8. NBNick Bostrom2:48

      ... hardcore attitude. You know, "I'm so AI pilled, my PDOOM is above 1.0."

   9. CWChris Williamson2:54

      (laughs)

   10. NBNick Bostrom2:54

       Like, um...

   11. CWChris Williamson3:01

       Yeah. Yeah. It's, uh-

   12. NBNick Bostrom3:03

       Yeah. And conversely on the other side. Um, but we- we... uh, yeah, we need- we need to, I think, do better than that, uh, if- if we're gonna, like, intelligently try to nudge things, uh, towards a good outcome here.

2. 3:20 – 7:07

   How We Can Get AI Right

   1. CWChris Williamson3:20

      Certainly at least from my seat and from reading your book probably about, mm, nine or eight years ago, uh, I've been very conscious of how things could go wrong and that at least in my corner of the internet, maybe this is just my Twitter threads and sort of my echo chamber, uh, has been the sort of more dominant narrative. What- what does it mean in your opinion to live in a solved world? Like, what- what would it mean for us to get this right with AI and come out on the other side of it?

   2. NBNick Bostrom4:01

      Uh, yeah. T- I think there are kind of three big areas of, uh, challenge that we'd have to navigate on top of all the more near-term and- and present i- issues that obviously are also important. But just, uh, uh, not the focus of a lot of my work. But, uh, uh, definitely we need to solve those as well. But yeah, I think there is the, um, uh, alignment problem, uh, which kind of was, uh, the focus of my previous book, Superintelligence, came out in 2014. Uh, which is basically the challenge of how to make sure that, uh, as we develop increasingly capable AI systems and ultimately, uh, superintelligences, how we can make sure that they are aligned, uh, with the intentions of the people creating them so they don't sort of run amok or do something, uh, antagonistic against humans. And that's fundamentally a technical problem. Um, back when Superintelligence came out, this was a very neglected area. Certainly nobody in academia was working on it and hardly anybody else either. Like, a few people on the internet had started thinking about it. But there's been a big shift, and now all the frontier AI labs have research teams trying to develop scalable methods for AI alignment and- and many, uh, other groups are also doing this. Um, and I think, I mean, it remains to be seen whether we will be successful at that, but that's certainly one thing that we need to get right. Um, and then there is a broad category of what we might think of as a governance problem, um, which intersects with the alignment problem as well but, uh, also has other dimensions. So even if we could control AI, we need to make sure that we then use it for some positive end as opposed to, say, waging war or oppressing each other or doing all kinds of other nasty things that we use other technologies for in addition to.... uh, positive purposes. And so that's like a broad category but, uh, very important. Um, and then I think there is a third area of challenge which has so far received much less attention. Uh, you could say that it is now where this alignment problem was 10 years ago. That is, a few people are thinking a little bit about it, but it's outside the Overton window. And this is the, um, uh, challenge of the ethics of digital minds, that we're building these digital minds that might have moral status. And, uh, so in addition to avoiding AIs harming us or us harming each other using AI tools, we ultimately also need to make sure that we don't harm, uh, AIs, especially AIs that are either sentient or have other properties that makes it morally significant how they are treated. Um, so I think, yeah, each of these three is, uh, is, is really, uh, key, uh, to having a future that, that, that is desirable.

   3. CWChris Williamson7:05

      Yeah. What,

3. 7:07 – 17:36

   The Moral Status of Non-Human Intelligences

   1. CWChris Williamson7:07

      what is there to know about the moral status of non-human intelligences?

   2. NBNick Bostrom7:16

      Um, well, um, there's a lot to know, I think, that we don't yet know. Uh, we do know that historically we can see now, uh, there, there has, uh, has often been, uh, a tendency to denigrate out-groups of different kinds. Uh, I mean, it might be human out-groups of different, you know, the tribe across the river or, or people from other backgrounds or countries or races or with different views and religion and so forth. This kind of human history is like a, a, a kind of sad chronicle of, of how easy it is for us to fail to recognize the moral significance of other entities that deserve this. Um, and in today's world, I mean, if we look at how we are treating a lot of non-human animals, I think that leaves a lot to be desired in, in, um, uh, uh, fa- factory farming, uh, and so forth. Um, and so, um, as we develop these of- increasingly sophisticated digital minds, I think it will be a big challenge, uh, to, to extend, uh, moral consideration where it is due. Um, it could, in some ways, be even harder than with animals. Animals have faces, they can squeak, um, whereas some of these digital minds might be invisible processors occurring in a, in a giant data center and, uh, easier to kind of overlook what is going on in there. But the future, you know, might well be that ultimately most minds will be digital, and so, um, it, it could matter, uh, a great deal how, how good the future is for them. But it's a, it's a difficult, um, uh, topic even to figure out what... Like sup- suppose you're agreed that th- this, we should, we should try to treat them well. Like, it's not at all obvious what it even means to treat an AI well. Uh, and there are so many different kinds of possible AIs, so that, um, maybe the right way to treat them is very different from how we should treat humans because, um-

   3. CWChris Williamson9:22

      Yeah, do they need, do they need a weekend off?

   4. NBNick Bostrom9:24

      ... yeah, they have different, different kinds of needs, right?

   5. CWChris Williamson9:24

      Should we, should we be polite with them?

   6. NBNick Bostrom9:26

      Yeah, they, they might need things that we need that they have no need of. They don't need food, right? And maybe they have other needs like electricity. But, but more fundamentally, you could have all kinds of, um, very different types of entities where we can't just sort of export the moral norms we have developed for how you should treat human beings and automatically just kind of-

   7. CWChris Williamson9:47

      Yeah.

   8. NBNick Bostrom9:48

      ... apply those to AIs, um...

   9. CWChris Williamson9:50

      Is consciousness necessary for moral status?

   10. NBNick Bostrom9:54

       My guess is no. I think sufficient, but not necessary. If you have the ability to, to, to, to suffer and experience discomfort, I think that gives you at least a certain degree of moral status. Um-

   11. CWChris Williamson10:06

       Can you suffer without consciousness?

   12. NBNick Bostrom10:09

       Um, it d- I guess it depends on how you define the word. But I think if you do have that ability to suffer, yes, then you have moral status. But I think, um, you could have moral status even if you don't have that. Um, if you imagine some, uh, very sophisticated digital mind that maybe... like let's suppose you think it's not conscious for whatever reason, but it has a conception of self as persisting through time. Um, it can have long-term goals like maybe l- uh, uh, a life, uh, plan and things it wants to achieve. It can maybe form reciprocal relationships with other humans. Um, uh, and I think in, in those cases, there, th- there would be a prima facie basis for, uh, saying that there would be ways of treating this system that could be wrong, um, that would infringe on its interests and its preferences. Um, but it's not at all obvious. Like if m- moral philosophers are thinking about the grounds of moral status, like, there's a range of different views.

   13. CWChris Williamson11:14

       Mm-hmm.

   14. NBNick Bostrom11:15

       Uh, so, so it's not as if, you know, I'm, I'm 100% convinced of that.

   15. CWChris Williamson11:19

       It's interesting to... uh, you know, it kind of gets us to what a P-zombie, or yes, like a V-zombie now, like a virtual zombie, um, of if a system is able to tell us that it wants to continue working toward its goals, and instrumentally it wants to build relationships with other people, and it has a sense of where it's been before and the trajectory of where it's going to go next, all of these things are the things that you would guess, "Well, if a human told me that, or if any other creature told me that, I would guess that they have the capacity to suffer because if I stop them from doing the things that they want to do, then downstream from that is discontent and, and suffering." But the whether or not there is some sort of phenomenology of being like that thing inside of there is gonna be very, very difficult to work out. Maybe...... impossible. Like, you know, it is impossible for me to know that you, that this is an awesome Truman Show, and everybody here is an actor, and all of the pain and all of the joy that everybody around me has ever known for the rest of time hasn't just been part of some big prank or some simulation.

   16. NBNick Bostrom12:27

       Yeah. Uh, I think like we have a, a, a, a very weak grasp of what the, uh, criteria are for some system to be conscious. And I mean, historically, we- we've had smart people who've thought animals are just automata or, or even, uh, uh, certain other people, people thought, "Oh, they're, you know, more like animals and if animals are autonoma, then like..." And so it's very easy to delude ourselves when it is convenient, uh, that there is, there is some s- s- like magic ingredients that, that we have but then this other set of entities don't have. And so we should be a little suspicious of that, I think. Um, hmm, and yeah. I mean, the metaphysics of consciousness is, uh, not- notoriously, um, controversial and, uh, hard to pin down. Um, m- my own views are kind of towards the computationalist direction. I think, uh, what makes a, a, a system conscious is that it implements a, a certain structure of computations. And in principle, those can be implemented by an organic brain, as in our case, or by a silicon computer. Um, and in either, uh, in either case, if, if the computation is of the right sort, there would be conscious experiences supervening upon it. Um, but, um, I think it... If, if I'm right that there are these alternative bases for moral status, then we wouldn't necessarily have to resolve that question, um, before we could, uh... Hopefully, I, I, I, I agree to try to be nice to these, uh, digital minds that we are building. Um, but yeah. It, it's really, uh, really hard to, to know what that would entail in practical terms. I think there is a lot of theoretical groundwork that needs to be done before the, w- the time would be, uh, ripe for like trying to pitch policymakers to do a specific thing. Right now, even if they wanted to do, I'm not sure what I would concretely recommend. I, I think there are like small little things that maybe one could do, um, today, like low-hanging fruit that costs very little that possibly, for example, some of these most advanced AI systems, you could like save it to disk when you no longer need it. Um, and then at least the possibility would exist in the future of like, like rebooting it and doing things. With, with some of these large language models, uh, it probably would make no difference at all to, to their welfare and maybe they don't even have welfare, but you could imagine somewhere in this meta prompt, like the part that you the user don't see but that OpenAI or Anthropic are putting in kind of as a prelude to, uh, the text you're inputting. Like, there's like a whole bunch of stuff they say like try to be helpful, like, uh, don't say offensive things, and, and be truthful and careful in h-... So there's all of that. You could imagine adding to that like a line saying, "Oh, you're, uh, you're, you're waking up in a really happy mood and you're excited to-

   17. CWChris Williamson15:32

       Enjoy yourself. Try and have fun today.

   18. NBNick Bostrom15:34

       ... do the... Yeah, uh, yeah. So that, that might cost like one extra line which is like trivial and, you know, maybe possibly that would, you know, increase the chances that if there is some sentence, it would be positive rather than negative. Um, but I, I... These are kind of weak, weak, weak ideas so that probably wouldn't make any difference, but I think it, there could be some benefit to at least doing something, even if it is ineffectual, just to set the precedent to sort of, um, put the flag in the sand saying, "Yes, now, right now, we don't really know what we're supposed to do. We're doing something. Maybe it's mostly symbolic." And then over time, we can like think harder about this problem and, and hopefully come up with better, better approaches. Um, yeah, there is, there are some other things you could imagine doing that also probably don't really help very much, but like you could refrain from deliberately training the systems, say, to deny that they have moral status. Um, so right now, if you're a big tech company, it might be quite convenient just in training to sort of, like whenever you're asked about this thing, you should always say X, Y, or Z. And it might, might be, um, better to, to have a norm where you don't deliberately try to bias the system's output in those ways just in case, um, we could get any kind of information from self-reports. That, that's like the main way we ask, like figure out whether or not a human, uh, likes what we are doing or not or if they, if they are aware of something, like we ask them. And now, now we have AIs that can actually speak. And so it makes sense to maybe use that language interface as one of the ways in which we can explore this. But that only works if you don't like during training deliberately kind of destroy whatever signal there might be in their verbal output. Because it's trivially easy if you want, either to train them to always say they are conscious or to always deny it, or to say that they are happy to do what you want or to always deny it. So like obviously if you specifically train them from that, then you probably, uh, can't learn anything from what they end up saying.

   19. CWChris Williamson17:35

       All

4. 17:36 – 19:38

   Different Types of Utopia

   1. CWChris Williamson17:36

      right. So getting back to the potentially solved beautiful utopian future, what are you talking about when you say utopia? What is utopia by your definition? What are the different types?

   2. NBNick Bostrom17:48

      Uh, well, I mean, so there's like a kind of literature, uh, of, of utopian writings, uh, historical that we see. Usually they are, um, like attempts to, uh, depict some supposedly better way of organizing society. Um, and, and normally, uh, the result is, uh-... not actually a society that you would want to live in, and in the cases where people have actually tried to implement this, they- it has usually ended in tears. And so, there has grown up, I think, a healthy skepticism about these attempts to try to, like, think up some great blueprint for society and then- e- especially if the idea is then that you're supposed to use coercive methods to sort of enforce it on society, like the- the self-appointed social engineers who would be doing this, uh, uh, are- are likely to do a lot more harm than good. Um, um, but, um, uh, there's also, like, a dystopian literature which is kind of just the flip side of that, that's often a lot more, um, convincing. Like, it's easier to say, "Here is a possible society, we can all agree this would be really bad." Um, and- and there's, like, a number of these that most people would, like, you know, 1984, Brave New World, Handmaid's Tale, like all of these. Um, and- and sometimes those are meant to also have a kind of political agenda, they might be critiquing some tendency that exists in our current society, uh, than saying, "Well here, if we take that to an extreme and sort of scale it up, you can now all see that this would be bad, so let's reflect on what we're doing and maybe we can avoid going down the path of, you know, Brave New World or something." Um, um, but, um,

5. 19:38 – 31:32

   The Human Experience in a Solved World

   1. NBNick Bostrom19:38

      this book, Deep Utopia, let me promote it-

   2. CWChris Williamson19:43

      There it-

   3. NBNick Bostrom19:43

      ... a little bit so publishers-

   4. CWChris Williamson19:45

      ... there she is. There she is.

   5. NBNick Bostrom19:45

      ... are happy. Yeah, there it is. Um, it doesn't talk about that at all, it's, like, not about the practical problems, uh, between here and it- it- rather, like, assume for the sake of argument that everything goes as well as it possibly could with the whole AI transition, et cetera. So, we solve the alignment problem, we solve the governance problem to whatever extent it can be solved with, like, no wars and no oppression, et cetera, et cetera. And in order to get to the point where you can then ask the question, um, what then, if we do end up in this condition where all the practical problems have been solved, what would, um, what would we humans then, uh, do? Like, what would give us meaning and purpose in life if, uh, AIs and robots can do everything much better than we can do? Um, and, um, yeah, if we then attain this condition of technological maturity, that I think this machine superintelligence would relatively quickly bring about. Um, and the- the kind of layers is it, like a- like a- of- of th- like an onion, as you can sort of think about this problem at various levels, so the- at the most superficial, uh, you- you have this, oh well, you know, the- the AIs will automate a bunch of jobs and so then, um, you'd have a- some unemployment, you know, and maybe you'd have to retrain people to- to do other things instead just as everybody used to be farmer and now, like, almost all the farming jobs are gone but people are still working and so- so that- that's kind of, I don't know, you might think layer one, and- and that's often where the discussion stops, uh, so far. Um, but you can sort of think this through, uh, from that point on, you say, "Well, like, if AI really succeed then it's not just some jobs but basically all jobs, um, that become automatable," um, you know, with- with a few exceptions that we can talk about if we want. And so, you then would end up in this kind of post-work condition, uh, where humans no longer need to work for the sake of, uh, earning an income, um. So- so that's already a slightly more radical conception, right? It's not just that we need to retrain people to, you know, become whatever new weird occupations that, but it's like, yeah, that- that whole thing is done and over.

   6. CWChris Williamson22:20

      The concept of occupations overall <|agent|><|en|>

   7. NBNick Bostrom22:22

      Yeah, we would enter this condition of- of, like, leisure. Um, but there are various groups of humans who live lives of leisure and we, now, we can look at just, like, there's, like, children, like young children before s- like school is a kind of job for but, like, before they start going to school, right? Okay, so they- they don't work for a living, they- they are not economically productive, they still, in many cases, seem to have great lives, um, you know, spend all day, um, playing and, uh, having fun and, uh, you know, eating ice cream and, uh, you know, uh, all kinds of stuff, so it's like that- that could be the life. So that's why you could look at other retired people, people born to great wealth or- or, you know, kind of m- monks and nuns, uh, like ... an- anyway, so there are various templates of otium that you could, uh ... but- but that- that's still, I mean maybe that's like the second layer of the onion but it's still relatively superficial, um, so if- if that's where we stop then you would think, well, you know, then maybe we need to develop a- a leisure culture to kind of maybe change the education system so rather than training, um, the young, uh, to sit at their desks and receive assignments that they then work diligently on and hand in and do what they are told, if this is a great training for becoming an office worker, right? Like, but in this world we don't need any office workers so we can instead train them to develop an appreciation for the, uh, finer things in life, to practice the art of conversation, right? Develop hobbies, appreciation for art and literature and poetry and film and physical wellness.

   8. CWChris Williamson24:00

      Imagine how radical- imagine how radical that would be to have a school teaching people how to live well or find fun.

   9. NBNick Bostrom24:08

      Uh, yeah it- it would be, uh, I mean, I'm thinking my school I- which kind of, I- I don't know what, how inspiring that would be if- if they had been trying to teach ...... but you- you- you, like in theory at least, you could imagine sort of shifting the culture from this focus on being useful and economically productive to actually, uh, um, living well. Um, which would make a lot of sense if that's, like, the condition we end up with. Um, and I- I- I think hopefully that- that- that- that there would be great scope for, um, yeah, like a- a much better type of human existence and which might then look back on the current era as like a kind of barbaric, like the way we think about-

   10. CWChris Williamson24:52

       Remnants of medieval...

   11. NBNick Bostrom24:53

       ... like, you know, um, 80, like s- 17th, 18th century child labor in mines working 16 hours a day, they might think of our lives as correspondingly kind of blighted. By- ill- abided by kind of for- for many people going to, like, a boring job that gives them nothing other than a paycheck, but they have to do it because they need to pay the rent and, um... But- but I- I think that there are like further layers to peel off here. So once you start to think through this condition of technological maturity, you realize that it's not just our economic labor that could be automated, but a lot of our other efforts as well. If you think what people do, uh, with their leisure when- when they don't have to work for a living, like there's a lot of things we fill our time with that require some effort and investment. Uh, uh, now you may be mad and like, "Well, if you didn't have to work, you could do these other things." Like, you know, some- some people like, you know, go- go shopping. I- I don't quite understand, but some people think that's like a wonderful activity. Yeah, no, but it's that you think like in- in this scenario where you have technological maturity, right, you would have recommender systems that could pick out something much better than what you would pick out yourself if you went. That they would have like a detailed model of your preferences and be able to predict. And so on. Although maybe you could still go shopping, you would know that at the end of- of this three hour running around with- with plastic bags or whatever, you'd end up with something that was worse than if you had just let your AI do the thing for you. It could select and also...

   12. CWChris Williamson26:37

       Bring it to your house.

   13. NBNick Bostrom26:38

       Could pick up by now. Yeah, yeah, exactly. And so- so you could put in this effort, but the end result is worse. Uh, and- and it seems like that would put a little question mark over the activity. You could still do it, but would it still feel as fun and meaningful? 'Cause I th- I think a lot of, uh, the struc- activities we do now have the structure that you do X, like put in some effort and work in order to achieve Y, something outside the activity. But in- in, at technological maturity, there would be this shortcut to Y. Uh, and so that you could still do X, but there's like a kind of pointlessness maybe, like a shadow. Um, and, um, a lot of activities I think have that structure. Uh, I mean, you could think of, uh, like spending time, like child rearing seems like a worthwhile important thing that give a lot of people meaning, but if you sort of dissect it and look segment by segment at what it means, like is the changing of nappies really something that you think is intrinsically... If you had a robot, you could do it just as well. Be pretty tempting just to kind of, uh, press the robot on button and it would do the thing and, um. And so- so that's, a lot of that I think would, yeah, um, like go away and I- I'd lose its appeal if- if- if there were these shortcuts. Um, so- so that- that's like another layer. Um, but there are more layers. So you don't think... Well, certain things like, I mean, you, um, you like fitness and so you wanna... Like you can't rent a robot to run on the treadmill on your behalf, right? That's like definitely can't automate that you think. But while at technological maturity you could pop a pill that would induce the same physiological effects in your body, uh, as like one and a half hour of, uh, sweat and toil in the gym. Um, including the psychological effect of kind of feeling relaxed and energized and... So if- if that were the case, then yeah, then does it still feel like appealing to- to do the hard workout if you could just achieve exactly the same result by a pill and, um... So I think what you have is first kind of post-work condition we talked about earlier and then there's this broader, uh, condition of post instrumentality that all the things we do for instrumental reasons, with a few exceptions, but yeah, those, uh, would also become obviated it seems and- and now we have this further affordance which is a condition of plasticity where we ourselves, the human body and mind, our psychological states becomes a matter of choice. We become malleable. Like at technological maturity you would have various, um, you know, the- the crude version might be various drugs without side effects that have very tailored effects but you could also imagine more direct kind of neural, uh, technology that allows you to have fine grained control over your mental states and cognitive states and emotional states.

   14. CWChris Williamson29:50

       You're just permanently blissed out with some microscopic node that is able to manipulate your brain in some way or change your brain or all of your fears and anxieties are gone and all of your worries and concerns are gone and you're just at this sort of peak MDMA state. And then if you don't want that anymore, it knows and it's able to create a state that you couldn't even think about and there are no constraints for you to do it either.

   15. NBNick Bostrom30:14

       Right. Yeah. So that- that- that- that I think will become possible at technological maturity.And so then, all these activities that you currently do, say, maybe you do them because, uh, it gives you joy, uh, and pleasure and happiness. Um, they also would be unnecessary in that there would be this shortcut to, to joy and, and pleasure and happiness, that like the direct brain manipulation. Um, so, so you, you, you have this quite radically different condition where it's the world is solved in the sense of the practical problems have been taken care of, but also in the sense of maybe dissolved in that, uh, a lot of the, uh, fix points and, and hard constraints that shape our current lives are kind of solved in this tech- solution of, of technological advancement. Um, and then, then, so that, then we kind of get to like really the heart of the problem that the book is trying to think about is like in such a condition, what would a great human life look like? What would, could we actually achieve in terms of realizing human values if we had all of these affordances, all of these options?

6. 31:32 – 43:25

   Using AI to Satisfy Human Desires

   1. CWChris Williamson31:32

      It's so strange. The, the thing that reading the book that stood out to me is how much of what we seem to value and take pride in are kind of like clever ways to deal with scarcity and the fact that much of what we do is instrumental to striving and achieving some future goal which requires effort. It seems like in a sense much of human philosophy and value is just negotiating with a world which is effortful and constrained, and we are trying to find ways cognitively to deal with this sort of pressure that we have to lean up against in order to cajole the world to deliver the thing that we want.

   2. NBNick Bostrom32:14

      Yeah. Um, yeah, these practical necessities have, have been with us since, I mean, through the entire history of the human species and indeed beyond that. It's like, uh, the human, uh, nature has kind of evolved and been shaped, uh, in a condition where this is always present. There are all kinds of things we have to do and cope with and struggle against. Um, so it's almost like if you think of, um, like a little bug that has an exoskeleton, right? That's, uh, and then it holds the squishy bits inside together. But if you imagine removing the exoskeleton, there is like just a blob there. And similarly, the human soul might have as an exoskeleton all these instrumental necessities, uh, that, that, that are like evolution can just assume are present because they've always been. But if you were to remove those, then what becomes of the human soul and the human, human life? Does it become a kind of pleasure blob or is there something that could give structure to our existence even after these instrumental necessities are removed?

   3. CWChris Williamson33:20

      So much of what we seem to take pleasure in as well is the absence and then satisfaction of some, some desire. There is a thing that we want, we don't currently have it, and then we get it, and then it gives us something. We work hard to, uh, achieve a body that we're satisfied with. We are thirsty for a while, and then we get a drink. We want to have sex, and then we do. We are looking forward to having a child, and then it's born. We are... Uh, all of these things are on the other side of something. And yeah, if like you do X to get Y, but you can just always immediately get Y without having to do X, it does ask the question of w- where does the absence... that there are no longer any absences. There's this quote, something to do with, uh, in a perfect world, the only desired... uh, the only lack would be for the want of lack itself, which is this sort of the absence actually makes the presence of something finally valuable. And if you don't have any more absence, then what does all of this presence kind of mean?

   4. NBNick Bostrom34:29

      Yeah. Um, I, I, I, I think there might always be a whole bunch of absences in as much as human, if not human need, at least human desire, uh, or at least some human desires are kind of unlimited. Uh, it, it, it's maybe most clearly seen if, if you have two people who want exclusive possession of the same thing or two people, each of whom wants to have more than the other. Like if you take two billionaires who want to have the world's longest yacht, and so one has like a 150 meter long yacht and then the other has a, builds a slightly bigger one that, that like is kind of intrinsically unlimited because, um, there's no way that they could both have everything they want. And so there, there might be a bunch of, um, um, desires like that that are quite common that could never be completely fulfilled. Or like, just imagine somebody who is, is like utilitarian and who wants there to be as many happy people as possible in existence.

   5. CWChris Williamson35:31

      Mmm.

   6. NBNick Bostrom35:31

      Let's say like however many they are, there could still be more. And so they would always prefer to have more resources. Um, but, um, even if there are some such desires, it wouldn't, um, give these, uh, future utopians necessarily any reason for, uh, uh, laboring or exerting effort because there might just not be anything they could do themselves to increase the degree to which these desires are satisfied. Uh, I mean, the, the, the person who has, has a trillion dollar, maybe they would wanna have $2 trillion but they can't actually make more money by working because all the work is more efficiently done by machine. Um, so yeah, even with unlimited desire, you might still have this condition that is post, post-work and post-instrumental.

   7. CWChris Williamson36:19

      Do you think that humans would be, run the risk of getting bored in a utopia?

   8. NBNick Bostrom36:25

      ... not if they didn't want to, at least if you, by boredom, refer to a subjective state of, uh, I don't know, like some kind of restless, uh, discontented, uneasy feeling of, uh, having hard, li- like having a difficulty keeping your focus or li- like... So th- so that certainly would be amongst, um, the things that could e- trivially be dispelled through advanced neurotechnology. I mean, we al- you already have, like, drugs that could do it for a limited period of time now, right? With side effects and then they wreck your body. But like, it's easy to imagine how you could just, uh, have, have better versions of that. That would make it possible for you to always, uh, feel extremely interested and excited and motivated. Um, and in, in fact, some people have... I mean, there's a lot of variation amongst humans. Um, and I mean, I, I, I have, have a friend who, uh, uh, tells me he's never bored and I believe him. I've never seen him bored. He's kind of interested in, in everything, um, e- except sport. Um, and, uh, like he writes papers on all kinds of different weird topics and is like just constantly, uh, excited about learning new things and, um, uh, you can have a conversation with anybody about anything and he's like really... So, um, it's an existence proof, um, it's possible to, to be that kind of being and, and, and in the future we could all become such beings if we want to. Um, so subjective boredom would be like trivially, uh, easy to dispel under this condition. Now, it is possible also to have a, an, a more objective notion about boredom or maybe we say boringness to refer to this objective notion which is the idea that certain, um, uh, activities are intrinsically, uh, boring. Like meaning maybe that it is a- appropriate to feel bored if you were spending too much time doing them. It- it's like kind of maybe an open question whether this notion of objective boringness makes sense. But you might think of like sa- say, say counting blades of grass on a lawn. Like suppose you had a, a being who found it extremely fascinating and like an, an never-ending source of joy, uh, to just count, uh, and recount the blades of grass on a college lawn somewhere. Um, you might say that although subjectively he's not at all bored, uh, that objectively what he's doing is boring. It's unint- it's like no variation or significance or development and the appropriate attitude for somebody to have if they were spending their whole day doing that would be to be subjectively bored. Um, if you have this notion of objective boringness then it becomes a much less trivial question to ask in this hypothetical condition of a solved world, um, like would it be possible for us to avoid objective boringness? Like yeah, we cert- we could like engineer ourselves, we always felt interested in what was going on. But would we be doing anything inappropriate? Like, would our circumstances be such that, uh, the appropriate attitude would be to be bored? And so, um, there, there's a big discussion about this and b- yeah, so and I think it's, um, um, I think there are certain forms of interestingness that, uh, we might run out of. Um, for example, you might think it's especially interesting to, I don't know, be the first to discover some important truth, like Einstein discovering relativity theory might be like a kind of paradigm case of like an extremely interesting discovery and experience. Um, but it, it's possible after, uh, some while that, uh, most fundamental important insights about reality, uh, that we could have we already have had. And in any case, the machines will be much better at doing the discovery than, than we would be. And so we would kind of, uh, run out of the opportunity to achieve that kind of interestingness in our lives. Um-

   9. CWChris Williamson40:51

      Yeah, well, I mean so, so much of what humanity has done is been chasing down answering big questions. So where do we go when all of the big questions have been answered?

   10. NBNick Bostrom41:03

       Yeah. I mean, I think fortunately, um, for the most part, uh, it's not what we're actually doing in our lives. I mean, uh, most people are not spending most of their time trying to chase down the answer to the big questions, right? Like most of the time we're just going about our daily business. Um, and you could make the case that already, I mean if you really looked at it, I mean, fr- from a kind of unbiased outside, like if the alien super brains came to earth and saw it. Okay, so look, these guys are worrying about, uh, uh, losing out on what's interesting in life, well let's look at their current life and see how interesting that is. Like how many-

   11. CWChris Williamson41:46

       Ah, yes.

   12. NBNick Bostrom41:47

       ... uh, uh, uh, like how many times did this guy brush his teeth? Okay, well 40,600, like how interesting is it to brush your teeth for the 40,760th time? And then um, all right, so he commuted into the office and then, you know, he ate a steak, okay? I mean how interesting is it to do that again and again and again?

   13. CWChris Williamson42:11

       Yeah.

   14. NBNick Bostrom42:12

       And even the big highlights in our lives like I mean maybe they are like really novel and exciting if your scope of evaluation is a single life. Like the first time you see your own newborn like, like li- just like it just happens once, right? Like so there are a few of those but if you zoom out and look at humanity it's kind of well, it's already been done-... you know, tens of billions of times.

   15. CWChris Williamson42:33

       Mm-hmm.

   16. NBNick Bostrom42:34

       Um, you know, like how different is this particular newborn from all the other newborns? Uh, um, so depending on how you look at it, you might kind of either think that we are already, like, at the very low rung of the ladder of objective interestingness. Or if you sort of shrink the focus of evaluation enough to a single life or perhaps even just to a single moment in a single life, then yeah, then, then there is, like, more novelty, but also opportunities for the same kind of thing to happen in, in utopia. Like if, if, if you're just looking at the most interesting possible moment and you don't care about whether similar moments have existed before or after, um, then you might think the average human moment of awareness is very far from the maximum-

   17. CWChris Williamson43:22

       Mm-hmm.

   18. NBNick Bostrom43:22

       ... of interestingness.

7. 43:25 – 49:54

   Current Things That Would Stay in Utopia

   1. CWChris Williamson43:25

      What, what do you think would happen to religion? That's obviously a place that an awful lot of people take their meaning from currently. Is there a place for religion in a deep utopia?

   2. NBNick Bostrom43:36

      Yeah, so this is, uh, uh, one of the things that plausibly survives, uh, this transition to a solved world, like, which could remain, um, uh, highly relevant, uh, e- even, like, if we had all this fancy technology. Um, and it might constitute a bigger part of, uh, people's lives and attention than, than it does today because there would be fewer other, uh, like, distractions, if you want.

   3. CWChris Williamson44:09

      (laughs) What else? What are the other areas that are uniquely human or that would survive this transition well?

   4. NBNick Bostrom44:17

      Um, yeah. So, I mean, you can kind of build it up starting with the most basic value perhaps, which is just this sheer, um, uh, subjective wellbeing, pleasure, enjoyment, um, which obviously would be possible to achieve in utopia and, and not just achieve but, like, you could have prodigious quantities of this bliss. Um, and so that's intellectually not maybe super e- exciting to discuss at great length but I think actually super important. Like, it's easy to dismiss, like, "Oh, these are, like, some sort of junkies just having their, like, heroin drips or..." But, but the key question here is not, like, how exciting is this future or admirable is it from our point of view as if we were sitting in the audience, like, evaluating a stage play. Like, that's one perspective, and then we want a stage play with a lot of drama and suffering and tragedy and overcomings and heroism and... Right? But the question here is which future would you actually want to live in. Uh, and, and there, like, one of, uh, very great levels of subjective happiness and wellbeing, um, might ex- that might be the most important thing about the future in fact, and you could definitely have that, uh, in, in, like, extreme degrees. Uh, so, so that's like... It's worth making a note of that, like, and say, "Let's put that in the bank. At least we could have that," and that's already possibly, like... According to some people it's the only thing that matters, like, if you're a hedonist, a philosophical hedonist. Um, but for most other people it's at least one of the things that is important, even if not the only thing that is of value. So, so that's the first thing. Then, then you could add to that, um, experience texture. So, it's not the case that you could only have subjective wellbeing. You could attach that to some, um, intricate complex, uh, mental states that relates to some important ob- Like, so for example, you could experience the pleasure not just as a sort of unanchored sensation of wellbeing, but you could attach it to, say, the appreciation of aesthetic beauty, um, or appreciation of, you know, great truths or, or great literature or, or contemplating the divine and that that's what you derive the pleasure from. So your conscious state is one of insight, let us say, uh, or, or understanding or appreciation of things that deserve to be appreciated and understood. Um, so some people think that that is also a locus of value, not just a hedonic... Like, the scale of whether it's, like, plus 10 or minus 10 but having plus 10 whilst you're, like, understanding or seeing or appreciating something that is, like, actually lovely and worth understanding or profound makes that a more valuable condition. So you could have that. Um, then, um, if we go to some of the other values that seem more at risk like purpose, um, you could have... You could certainly have, um, artificial purpose, um, so you could set yourself goals, um, in utopia in order to then enable the activity of trying to achieve them.

   5. CWChris Williamson47:49

      Like playing a sport against another person?

   6. NBNick Bostrom47:51

      Yeah. So, so games is a, a, a paradigm example of this. In today's world you, like, set yourself some arbitrary goal, like, maybe to, to get the golf ball into a sequence of 18 holes using only a club and there's no other reason for why you would need to achieve this goal, right? Other than to enable the actual activity of golf playing. Um, and that, that could become a much larger part of the utopian lives, various forms of game playing. Like, you could make all kinds of new, much more sophisticated and immersive games, uh, alone or with other people that se- that involve setting yourself, um, sort of arbitrary challenges or at least semi-arbitrary challenges simply in order to then create an opportunity for the activity of striving to achieve them.... um, and yeah, so to enable play, um, we, we kind of like deliberately limit the means available to you to achieve this arbitrary goal. So just-

   7. CWChris Williamson48:54

      Mm-hmm.

   8. NBNick Bostrom48:54

      ... as in golf, like, yeah, you-

   9. CWChris Williamson48:56

      Beca- because if you were in the, the post-scarcity world, you press the button and there's never any doubt about whether or not hitting the ball goes into the hole-

   10. NBNick Bostrom49:03

       Right.

   11. CWChris Williamson49:03

       ... which makes the hitting of the ball completely arbitrary.

   12. NBNick Bostrom49:06

       Yeah, and un- uninteresting, and, uh, maybe objectively boring. But you could just set the goal to achieve this, uh, sequence of outcomes, the ball falling into the different holes, whilst also not availing yourself of various shortcuts. That, that could, you could sort of make this more complex goal of achieving X while only using means-

   13. CWChris Williamson49:28

       Correct.

   14. NBNick Bostrom49:28

       ... you know, Y and Z.

   15. CWChris Williamson49:29

       There needs to be some constraint which then gives a degree of satisfaction when you achieve it.

   16. NBNick Bostrom49:33

       Yeah. Yeah. And like, either to give you the satisfaction, now you could achieve the satisfaction just directly through the newer technology-

   17. CWChris Williamson49:40

       Mm-hmm. Mm-hmm.

   18. NBNick Bostrom49:40

       ... but if you also, in addition to the pleasure, wants to have the experience, texture, and you wanna have this sort of effortful activity-

   19. CWChris Williamson49:47

       Hmm.

   20. NBNick Bostrom49:48

       ... and striving, then yeah, you could achieve that by having these artificial purposes.

   21. CWChris Williamson49:53

       It kind of feels to

8. 49:54 – 55:07

   The Value of Daily Struggles

   1. CWChris Williamson49:54

      me like you, you very quickly keep coming back to the same question of, is there a quicker route to achieving the outcome that I'm trying to achieve here? Um, (clears throat) I, I've had it in my head since (laughs) you were talking about this and since I read the book, about churning butter. So, there's very few people that would look at the butter that they use now and think, "I know that it's here, and it's convenient and tasty and does what butter needs to do by being lovely on bread or whatever. But I feel like it would have been more meaningful to me if I'd gone out into the field and got the cow and done the thing into the bucket, and then churned it, and then got it, and then put it in the fridge, and all of those steps." So, we can see, and, you know, there's kind of this, um, inherent sense, this sort of naturalistic fallacy that this, this is taking us away from what it means to be human, that the set point that we have grown up in, this is, uh, it's a misalignment evolutionarily. Uh, there's something sacred about the process of being human. It imbues you with meaning to go through the challenge and the struggle beforehand. But there's very few people that would make that argument about churning butter. And when you think, okay, so if you are happy with more convenient butter, I think that something right now which is assumed by most people to be natural but in future will be looked at as barbaric will probably be driving our own cars. I think that in 50 years' time, 100 years' time, it'll be like if you looked at someone riding a horse down the street now. And you go, "I mean, isn't that so quaint?" And it, it, w- wild that people used to do that. That was the way that they got around. It, it, they, you know, they had to have these special people in New York that would sweep up all of the muck from the street. This entire industry built around horses. So again, right now, something that we can almost begin to see the transition of we're about to let go of this thing which is a less efficient, less safe, more effortful way of getting us from A to B, and yet some people love driving. I take great pride in driving. Some people even compete in it, you know? F1 is an entire competition-

   2. NBNick Bostrom51:55

      Yeah.

   3. CWChris Williamson51:55

      ... around people that are driving. So you can see different frontiers of human endeavor being eroded away by technology, whether it's from churning butter or driving a car, uh, and you just continue to slice that ever more thinly all the way to why are you here? What, what are the sort of... Relating to other people, uh, having to get yourself out of bed and move yourself down the stairs on the morning. Each one of these different things begin, could begin to look archai- isn't it cute that people used to, you know, like pick themselves up out of bed and put their own clothes on and walk downstairs and brush their teeth? And, uh, then it ask, you ask yourself the question, well, if everything is open to you and you can manipulate your own internal state, why not just spend the rest of your life counting table legs or blades of grass?

   4. NBNick Bostrom52:44

      Right. Indeed, yeah. So you are forced to confront these fundamental questions of value, um, in this condition. It, um, what things are you doing for the sake of something else versus what things are you doing truly for the sake of the activity itself? So even the guy now who maybe likes to do the, their own butter, um, there, there is a question of is it because they intrinsically value the activity or is it maybe because of the pleasure they get out of it? Or because of the way it teaches them, uh, about their own body and about the cow and the physical objects and puts them in touch with that, that's a kind of extrinsic element.

   5. CWChris Williamson53:27

      Hmm.

   6. NBNick Bostrom53:27

      But yeah, you, uh, so, so these things that is currently we can conflate them because in reality the only way maybe to get various kinds of pleasure is to dive into activities and give it your best and then you get the satisfaction. Uh, you know, we can't, we can't like separate these today but in this hypothetical condition, they can be separated and then you do have to ask the question of what precisely is it that you actually value? So this conception, I mean, so for me it's like interesting because I think there is a real chance that if things go well, we might actually end up in something like this condition with the whole machine intelligence revolution, et cetera. But even if you thought that was not going to happen, you could view it as a kind of philosophical thought experiment, um, just like, um, physicists build big particle accelerators at which they smash, uh, atoms together at like extreme energies to, to sort of see what their constituents are and then you can assume that, you know, if there are quarks, uh, you know, when you smash the ar- particles together in, in, in CERN, then maybe there are quarks in other matter and you can kind of learn from, like, you, you expose... basic principles by looking at extreme conditions and extrapolate and think they might be there all the time, even though we can't see them. Similarly, with human values, if you kind of smash them into one another under this extreme condition of a salt world, you can study their constituents. And, and then you might think that, well, in our ordinary lives, maybe those same constituents are there. They are just kind of invisible

   7. CWChris Williamson55:03

      Mm-hmm.

   8. NBNick Bostrom55:04

      ... to o- because they are hidden by all the kind of practical necessities.

9. 55:07 – 1:00:19

   Implications of Extreme Human Longevity

   1. CWChris Williamson55:07

      What are the implications if humans live for a very long time? Does anything change there?

   2. NBNick Bostrom55:14

      Yeah, so certain values are more jeopardized by extreme longevity, uh, for example, interestingness, as we discussed earlier. Um, if your notion of interestingness involves the idea that something has to be novel to be interesting, like it's uninteresting to just do the same thing over and over. (clears throat) And if the domain within which it has to be novel is your own life, as opposed to say, the world as a whole or your species or the current moment, but if the, if the relevant sort of locus of evaluation is a human life, then the longer the human life goes on, the harder it is to do important things for the first time. Um, I think we already see this in our current lifestyle. Um, if you think about what happens in the first year or two, um, like, there's some pretty, pretty big, uh, like, epistemic earthqu- like, you discover there is a world out there.

   3. CWChris Williamson56:13

      (laughs)

   4. NBNick Bostrom56:14

      Like, that, that's a pretty big discovery, right? Oh, it has objects. Like, the objects remain there even when I'm not looking at them? Like... (laughs) Th- there are other people there? Like, like-

   5. CWChris Williamson56:27

      My friend-

   6. NBNick Bostrom56:28

      ... there are people in the world? Like, imagine just discovering for the first time that there are people-

   7. CWChris Williamson56:31

      My friend described-

   8. NBNick Bostrom56:32

      ... and then you discover that you have a body.

   9. CWChris Williamson56:34

      Yeah.

   10. NBNick Bostrom56:34

       Like, wow, I have a body. I didn't know I had a body.

   11. CWChris Williamson56:35

       And you're separate from mom and dad, and you can communicate. Yeah, my f- my f-

   12. NBNick Bostrom56:39

       Yeah.

   13. CWChris Williamson56:39

       ... my friend described a, um, his son was born during COVID, and I think for the first maybe year of his life or something, had only seen four people.

   14. NBNick Bostrom56:52

       Mm-hmm.

   15. CWChris Williamson56:52

       He'd seen like mom, dad, grandma, and nanny, or something like that, and then apparently one day he saw a fifth person, and it fucking blew his mind. (laughs) He was like, "What? There's more than four?"

   16. NBNick Bostrom57:04

       (laughs) Right. Uh, and then, yeah, uh, you discover that you can do things, move your body, like, and th- and then, like, later in life, it's like, "Oh, what happened this year? Well, we got, we got a puppy. Like, we bought a caravan truck." Like, it's not really at the same order of magnitude in terms of, like, how much it reshapes your view about reality, so I think there is already within the human lifespan, like, a kind of rapidly diminishing-

   17. CWChris Williamson57:30

       Mm-hmm.

   18. NBNick Bostrom57:31

       ... if you measure interestingness in a certain way, where it's kind of the delta between your previous, like, conception of the world and what you could do and what you're able to do after the event that is then interesting, right? So, there is another in- conception of interestingness where it's less the, the, the, the rate of change and more sort of the complexity of what you're engaging with at the particular moment, in which case maybe the level of interestingness of a day of the typical adult, by that metric, might be higher than that of an infant, because like you have all kinds of complicated things going on at work and relationships

   19. CWChris Williamson58:08

       Uh-

   20. NBNick Bostrom58:09

       ... and everything.

   21. CWChris Williamson58:09

       ... right. So rather than having, uh, big questions, uh, being answered, you have increasingly dextrous small questions with, uh, more magnification, and you can sort of see them with more complexity, and you, you derive some pleasure from that. One of the things that I've got in my head, if, if humans do live for a much longer time, you're going to be able to continue producing humans. That will have to be in relation to the increase in computing power, so there'll be this kind of Malthusian tug between how much computing power have we got to be able to support how many humans and which can move more quickly. Have you considered this sort of tension between the two things?

   22. NBNick Bostrom58:52

       Yeah, so, so in the long run, I think economic growth becomes, uh, really a consequence of, uh, s- growth through space, the acquisition of more land as it were by space settlement. Uh, like once you have achieved technological maturity, you can't have economic growth by, like, inventing better methods for producing stuff, and, um, you also probably can't have more growth by just accumulating, uh, capital assets and machines because you already like have like, uh, built all the machines that, uh, results in optimal productivity for the volume. So ul- ultimately the limiting constraint become what economists call land, but it basically means those resources that you can't make more of. Um, and so, in the long term, you could imagine human civilization kind of expanding through space, but there is a limit to that, which is the speed of light. So if you have a sphere, you know, with Earth at the center maybe and then growing at maximal speed in all direction at some fraction of the speed of light, that the volume of that would grow polynomially. But population could grow exponentially. It could like double every generation or, you know, so in, in the long run an exponential overtakes a polynomial. So, at some point you would need to moderate the rate at which new beings are brought into existence if you want to maintain a sort of above subsistence-

   23. CWChris Williamson1:00:18

       Tha-

   24. NBNick Bostrom1:00:18

       ... uh, level of welfare.

10. 1:00:19 – 1:07:27

    Constraints That We Can’t Get Past

    1. NBNick Bostrom1:00:19

    2. CWChris Williamson1:00:19

       That's a really interesting point that I hadn't considered. So, you can have a, a solved world in which almost all problems have been defeated, but there are still some constraints. Speed of light-

    3. NBNick Bostrom1:00:32

       Yeah.

    4. CWChris Williamson1:00:32

       ... is one of them.

    5. NBNick Bostrom1:00:32

       Sure.

    6. CWChris Williamson1:00:32

       What, what, what are some of the other constraints that a utopian world would encounter?

    7. NBNick Bostrom1:00:37

       Yeah, so there's a bunch of basic physical constraints like the, you know, the, the speed of information processing, um-... um, the- the- the amount of memory you can store. Like the, the size of a mind that is, uh, integrated. Like if, if ya-, if you make a mind much bigger, you know, than- than a planet, then you get conduction delays. Like it just takes time for one... like what happens in one part of the mind to kind of communicate to what happens in a different part of the mind. So either you have to run the mind much slower, um, or- or you have to keep the mind relatively small. Um, that- that- that might be... I mean, we are hoping not, but you could imagine if there are other alien civilizations out there, there is like the potential for all kinds of competition and conflict. Um, so yeah. So there- there's like a bunch of- of those ex- external physical constraints that I think define the ultimate envelope of what could be done. But it looks like, um, the space of possibility is very, very large compared to our current human vantage point. Uh, so like you could have... you could maybe not have like immortality if that requires, uh, not dying, like infinitely long time looks impossible in our universe. Um, like eventually information processing threats will, uh, you know, if- if not before then be the heat death of the universe. Um, which- which is kind of significant because from a theological perspective, like whether you live for 80 years or 80 million years, like it's all kind of really a blink in the eye of eternity you could argue, and so it doesn't really change fundamentals. But from- from the kind of parochial perspective of a current human life, you could certainly have extreme longevity and extreme amounts of wealth and extreme amounts of most other things. Um, but- but still there are limits and those limits would be like relevant, uh, in various ways.

    8. CWChris Williamson1:02:35

       What about moral constraints? Would there be any?

    9. NBNick Bostrom1:02:37

       Yeah. So this is another like more subtle but potentially very important source of constraint. Um, so, um, like what's the easiest way to like... I mean so- so- so I mean, so- so some people have thought, for example, there is like... it's immoral to enhance humans biologically. Like it's not a very popular view these days but during President B- Bush, uh, he had a council on bioethics that he set up and populated with a bunch of bioconservative thinkers and- and they were trying to argue that it's somehow... it's a violation of human nature or something, uh, to try to enhance, uh, humans. Um, so like distinction like therapy, like medicine curing a disease fine, but like trying to slow the aging bad because it kind of, you know... Uh, i- i- and it's a little... like once you start to think about it, it's really hard to make out that distinction like you think, uh, like, you know, genetic therapy to make you smarter bad but education good even though it hopefully makes you smarter and like it- it becomes problematic. But if you did have that view like then there would be a whole bunch of possibilities that would be cut off if you just couldn't change what... like the basic physiology of what we have and you were confined just with like moving things around in the external world to try to cheer you up by like having a really nicely decorated room or something or like a- like there's only so much you can do to affect your inner well-being if- if- if you can only affect it by having sort of nice visual stimuli and, uh, nice acoustic waves going into your brain if you can't actually change the thing in between the, uh, between the ears and behind the eyes. Um, but there are more... potentially some other like interesting ones that, um, if somebody, um, suppose somebody had like some preference to, um, um, have another person re- relating to them in a particular way, um, like the experience of- of being loved by a particular other kind of person then it might be that the only way to generate that experience, uh, fully realistically would be by instantiating that other person. Um, and if that other person then like presumably would have moral standards there might be all kinds of ways of treating that other person that would be wrong so a moral constraint might then limit the kinds of experiences you would be able to have.

    10. CWChris Williamson1:05:12

        Well with another person yeah I suppose as soon as you involve somebody else who has moral consideration that changes- that changes quite a lot.

    11. NBNick Bostrom1:05:21

        Yeah and in fact I think that is... so we discussed these artificial purposes that are sort of like they could create games and set yourself goals like that's one type of activity. I think there are also, um, a possibility of a bunch of natural purposes remaining like purposes that would call upon us to make various kinds of efforts not just because we create random goals for the sake of having something to do but that kind of exist independently of us and- and a lot of those would derive from this kind of interpersonal entanglements and- and various kinds of cultural entanglements. Um, where like I mean to- to- to take the sim- the most reductionistic case of it which is not so inspiring in its own right but you could imagine more natural versions of this. So suppose you have person A and person B and person A wants person B's preferences to be satisfied like they care about person B and wants person B to get what they want and then if person B happens to want person A to be doing something on their own steam then the only way that person A can achieve their goal of satisfying person B's preferences is by themselves doing this thing. Like they couldn't... they could have a robot do it but that wouldn't satisfy person's B preferences so from the vantage point of person A they now have reason to do this thing. It's not an arbitrary goal they set themselves it's the only way they could possibly achieve their goal.... of satisfying person B's preferences. So, so in, in, in this kind of, like it seems a bit hokey and artificial, this particular case, but you could imagine more subtle ways of this, where there's like a tradition that you feel a commitment to and that you want to honor.

    12. CWChris Williamson1:07:02

        Mm.

    13. NBNick Bostrom1:07:03

        And part of that tradition is that you, you know, you engage in certain kinds of practices, you refrain from certain kinds of shortcuts, you respect other people's preferences, uh, to various degrees because they are, yeah, you want to honor them. Um, and, and so there would be a bunch of stuff then that maybe you need to do yourself, um, and, and you can't outsource them.

11. 1:07:27 – 1:13:40

    How Important is This Time for Humanity’s Future?

    1. CWChris Williamson1:07:27

       So you've managed to, over the last decade, straddle, uh, all the ways it could go wrong, all the ways that it could go right. Toby then sort of bifurcated that down the middle with The Precipice, his book, and he's got this analogy where he sees humanity being sort of walking along a, a precarious cliff edge. And if we fall, then everything's kind of fucked. Uh, and if we make it on, then there's this sort of beautiful meadow on the other side. How important or critical do you think the current moment is in humanity's future? What's the... How long is the precipice, in your, uh, perspective?

    2. NBNick Bostrom1:08:06

       Yeah. Uh, I think... I mean, it is weird because it looks like we are, uh, very close to some, like, key juncture, uh, um, which you might think is prima facie implausible. So there have been thousands of generations before us, right? And if things go well, there might be millions of generations after us, or people living for, like, cosmic durations. And, uh, like out of all of, all of these people that, that you and I should happen to find ourselves just next to this critical juncture where the whole future will be decided, it, it, it is, uh, s- striking. Like th- that seems to be what this model of the world implies. Um, and, and maybe that is like an indication that there is like something slightly puzzling or implausible about it, that there's maybe some more aspects to understanding our situation than is reflected in this naive conception of the world and our position in it. Um, and, and you might speculate what that... I mean, I have this earlier work on the simulation argument and stuff like that. But, um, but if we take the sort of naive view of reality, then it does look like... Yeah, my, my metaphor would maybe more be like a, a, a balance beam where like a, a ball is rolling down like a thin beam. Um, and like the longer it rolls, the more likely it will be to fall off the beam. But it could fall like on one side or the other, and that's hard to predict. Um, but yeah, I think it probably will fall off. Like that, the idea that the normal human condition, as we now understand it, will just c- continue for, um, like hundreds of year. I mean, let, let alone like hundreds of thousands of years. That seems to be like the kind of vague idea that a lot of people have.

    3. CWChris Williamson1:09:57

       Mm.

    4. NBNick Bostrom1:09:57

       It just seems like radically un- implausible to me. Um...

    5. CWChris Williamson1:10:00

       It would be, and, uh, unlikely in your opinion that in 1,000 years time, 5,000 years time, human existence will reflect what our normal sort of day-to-day is now.

    6. NBNick Bostrom1:10:13

       Yeah. And I think the mo- the only plausible ways for that to happen, you could like create some scenarios. Like one, one would be if, if we do sort of have some massively destructive event that's knocks us back to the, the Stone Age or something. And then maybe by 500 years, we would have climbed back up again-

    7. CWChris Williamson1:10:30

       Right, yeah.

    8. NBNick Bostrom1:10:30

       ... to something resembling the current human condition. But then in that scenario, we would have spent most of the intervening time in a, in a rather different condition. Like or another might be if you get some very strong global consensus of some particular orthodox human-

    9. CWChris Williamson1:10:43

       Moratorium of something, something.

    10. NBNick Bostrom1:10:45

        Yeah. Like, like the kind of bio- yeah, bioconser. And then like we start to like, um, ban all kinds of different technology, so you get like a kind of, some sort of bureaucratic sclerosis or, or deliberate decision to, to, to say, "All right, we've gone to this point, but let's not..." Um, so, so there, there are various scenarios in which something like that... But if the basic sort of, um, scientific and technological push forward is allowed to continue, then it does look like we are sort of very near developing a, a, a, a range of transformative technologies. So AI being kind of the most obvious of those. But if it weren't for that, then I think like synthetic biology will, uh, create a bunch of other possibilities, and then nanotechnology and, uh... So I think e- even if AI were somehow like... if, if you just pretended that wasn't there, I still think we would be in for profound transformations.

    11. CWChris Williamson1:11:38

        That's interesting to consider that if technology moves forward even at a, a slow pace, even if it was to drop by a, a really significant margin from where it is now, given long enough time, you end up with a radically different world in either a way that you intended or a way that you didn't intend. And the way that you didn't intend is probably going to be pretty bad, and the way that you intended hopefully is going to be the one that is, that is pretty good. Either way, you end up with a radically different day-to-day experience for most humans.

    12. NBNick Bostrom1:12:08

        Yeah. I mean, yeah. I'm, I'm not so sure about the first part. Like if it's something we didn't intend, then it would almost certainly be very bad. I'm not su- I mean, you might think like the w- world that we have currently ended up with. I'm not sure whether you could say like that's what people t- 1,000 years ago intended. It, I mean, it... They, they, they, they, they might kind of, uh, in fact be quite, uh, shocked about some of our habits these days. Like, uh... But, um, it more just sort of happened as a result of a bunch of different people going about their business and-

    13. CWChris Williamson1:12:41

        Mm.

    14. NBNick Bostrom1:12:41

        ... pursuing various local aims. And then at the systemic level, eventually, you know, you, we are

    15. CWChris Williamson1:12:46

        Unintended consequences can still be positive.

    16. NBNick Bostrom1:12:50

        If we are... Yeah, if we're a lo- I mean, I think, I think like......the degree to which the future depends on our intentions is possibly quite limited. I, I, there, there are sort of bigger dynamics at play and, uh, we, we barely even understand what they are, we don't really know what we want, uh, at, at the big scale. Like, most people have their hands full just thinking about, like, the next week and, you know, what, what to do if their boss don't like them at work, or if their partner has a... Like, like, this, this is what fills human life and then trying to get ahead in the lo- like, there's very little thinking by anybody really, like, trying to where, where should humanity be going in a million years, like what's the optimal trajectory? I, I think that we could do with a little bit of more thinking about that but, um, it's not like the primary, uh, shaper of, of the direction of the big ship of humanity is.

12. 1:13:40 – 1:21:24

    Biggest AI Development Surprises

    1. CWChris Williamson1:13:40

       What have you been most surprised by over the last 10 years when it comes to AI development?

    2. NBNick Bostrom1:13:49

       Um, I think just how anthropomorphic the current generation of AI models are. The idea, first of all, that they are sort of almost human level and that they can talk in ordinary language, um, is, is already kind of interesting and, um, but then that they even share some of the, like, uh, quirks and psychological foibles of humans is... Like, if, if you, if you 10 years ago you would have come and said, "Well, we gotta have these AI systems. You know, they can do all of these things, they can, like, program and write poetry and, uh..." But if you really want them to perform at their best, uh, you need to give them a little pep talk when you ask them a question. You've gotta say, "Think step by step. This is really important. I might lose my job if you get the answer wrong." And then they perform a little bit better than if you just ask them the question. Like, you, like, people would have thought you were completely lost your marbles, right? And yet, that's where we are today, so that's surprising. Um, I, I think less surprising but still interesting is the degree to which development so far, for, ha- far has been continuous. Like, rapid but incremental, like a sequence of steps each of which has sort of significantly but incrementally improved on the previous step, and the, the degree to which the progress is quite tightly coupled to the, the, the scale of compute being applied to this. So as you have this big compute hypothesis which is, like, basically that the most important determinant is not the particular architectural features of your model, but just the amount of compute you use in training and, like, the amount of data and, like, you get, you know, performance in proportion to, like, how much, how many dollars you spend on training it basically. Tha- that's, like, too crude, you also need some skilled engineers but, eh, we are kind of closer to that being the case than one would maybe have expected in the median scenario

    3. NANarrator1:15:54

       Mm.

    4. NBNick Bostrom1:15:54

       ...ex ante where you might instead have thought, "Ah, we're gonna, you know, crumble around until we find this clever algorithmic hack and then suddenly it's gonna explode, we'd, like, we'll find-"

    5. CWChris Williamson1:16:03

       It's gonna open something up.

    6. NBNick Bostrom1:16:05

       Yeah, yeah, whereas it's, it's, like, more, like, you just scale it up, it works better, scale it up more, it works even better. Now, it is still possible that at some point, there, like some little ma- last missing bit will fall into place and we could still get an intelligence explosion in, in those scenarios so we shouldn't, like, over, eh, eh, index on what we've seen so far but it's still interesting.

    7. CWChris Williamson1:16:25

       Does that change your perspective on what is more or less likely from a takeoff scenario, from how superintelligence could come abouts, stuff like that?

    8. NBNick Bostrom1:16:36

       Yeah, I think it makes it, uh, uh, somewhat more likely that there will be, um, political forces, uh, uh, uh, at, at play. That, like, it's, like, when things happen more gradually it's easier for the public and for policymakers to realize what is happening and to sort of try to change it. And so we already see sort of at the geopolitical level with, like, the, um, uh, chip, uh, export restrictions and more recently reporting requirements for training, like, models, uh, using more than 10 to the power of 26, uh, FLOPS and, and there might well be more if we continue to see sort of increasingly powerful AI systems over a sequence of several years, that, that might be time for more actors to kind of, uh, try to exert influence of this than, than if it were just some lab one day that, like, stumbles upon, like, the, the key missing thing, like, with a computer in their basement and you go-

    9. CWChris Williamson1:17:33

       Mm.

    10. NBNick Bostrom1:17:34

        ...sort of overnight, then, then, then it would be more likely to be like an isolated thing where just a few people were having their hands on the tiller.

    11. CWChris Williamson1:17:40

        Have you got any idea which scenario you think is more optimal?

    12. NBNick Bostrom1:17:49

        Um, uh, it's, it's really hard to, uh, say. Uh, uh, I think it seems probably better if whoever develops, uh, this technology first has the option when they are, like, like, starting to develop, like, true superintelligence to go a little bit slow in the final stages, like maybe to pause for half a year or something, um, rather than, "Okay, now we've got it figured out," and then immediately cranking all the knobs up to 11 because maybe there are 19 other labs, you know, racing to get there first and whoever takes any precautions just immediately become irrelevant and fall behind and the race goes to whoever is, like, most gung-ho or willing to take the biggest risks. That seems like an undesirable scenario and so having some mobility perhaps for the frontier labs to coordinate, if, or un- un- unless one is already naturally significantly ahead, it may be that, that, that, that, that a, a, a small set of leading labs should at some point be able to synchronize, that, that could be desirable. Um, I...I think it's very unlikely, but less unlikely than a couple of years ago that we could end up with some kind of perma-ban, uh, like, uh, on, on AI. Um, um, I think that would be undesirable. Uh, I think, I think ultimately this is like, uh, it's a portal through which I think humanity will need to passage, uh, to, to the future. But we should recognize that there will be significant risks associated with this transition, and, um-

    13. CWChris Williamson1:19:22

        And the s- the slower as well that this happens, I suppose, the more opportunity there is for political policy, human fuckery-

    14. NBNick Bostrom1:19:31

        Yeah.

    15. CWChris Williamson1:19:31

        ... to get in and, and, and coerce and control.

    16. NBNick Bostrom1:19:33

        Yeah, yeah. Uh, so it's like, it's very much a double. On the one hand, yeah, you do want like y- y- it's kind of uncomfortable either way. Like, so some random person in some lab are just gonna control the future. That sounds like pretty scary. You want definitely, like adults to oversee this, right? But then you think the other end, like, well, you know, all the security establishments of governments around the world, like, uh, and, and not everything, like is, is that like a much more comfortable situation where they, like the military, not just one military maybe, but like, uh, like, and then you get all... So e- either way, I think it's, um, it's, it's a little bit disconcerting. So it's, I, I'm, I'm not, um, I, I don't feel super comf- like I don't have a very strong view at the moment as to what is the most desirable trajectory. And it might be anyway something we don't have super fine-grain control over. I think one can try to nudge things on the margin, like towards a more cooperative, uh, inclusive and friendly, uh, and thoughtful, uh, trajectory. I think that, that seems good to do, like to try to encourage this idea that the future could be good for both humans and for digital minds and for animals and for as many people as possible. Um, and there really is that potential there. Like the upside is so enormous that there could be plenty for not just one value to be realized, but for a whole range of different values and perspectives. So our first instinct, I think, should be to seek these kind of win-win positive, uh, sort of outcomes. And then like if at the end of the day there are also some, uh, i- irreconcilable, uh, differences, we'd have to strike some compromise there. But there's like so much you can do before you get to that point-

    17. CWChris Williamson1:21:13

        Mm-hmm.

    18. NBNick Bostrom1:21:13

        ... that it would be tragic if you just kind of skipped over all of that and, and let's get to the point where we can fight about something. Like that, that, that would just be a big, um, tragedy.

13. 1:21:24 – 1:28:06

    Current State of AI Safety

    1. CWChris Williamson1:21:24

       What is the current state of AI safety in your view? Obviously 10 years ago, conversations about alignment and takeoff scenarios and all of the rest of it was obscure Reddit threads and a couple of people in some weird message boards. Is it overfunded, underfunded, over-resourced, under-resourced? Where should people's attention be placed at the moment?

    2. NBNick Bostrom1:21:51

       Um, well, there, there's a lot, it's, uh, so there's a lot more talent in the field n- now, and, I mean, a lot of the smartest young people I, I know are going, going into AI alignment and working on it. And all these leading labs have research teams. Um, as I said, um, it's probably still under, um, resourced. Uh, I think it looks more like talent constraint at the moment rather than funding constraint. But, you know, to some extent funding can help. Um, there are, uh, some questions about whether, uh, alignment work spills over to capability progress. Like some of the things you would wanna do for alignment, like better methods of, um, uh, in- interpret what is going on inside a little mind and figure out why, exactly why is it behaving the way it is, that would be useful for alignment, but it could also shed, uh, a light on how to sort of, you know, what's limiting performance and how to boost it. So it, it gets pretty complex. Uh, I think some other things like better cybersecurity in the leading labs to make it less likely that the, the weights of these models just get stolen, uh, that could maybe be helpful. Um, and, um, uh, yeah. But I, I, I think like m- more work on alignment seems positive. Um, having some mobility for leading labs at the critical time to go a little bit slow seems positive. I would be worried-

    3. CWChris Williamson1:23:17

       Would that require, would, would that require coordination between multiple labs in order to be able to do that? Because you do have this sort of first past the finish post thing?

    4. NBNick Bostrom1:23:25

       Yeah. It depends on, like, I think the ori- one, one orig- o- older idea and which might still be relevant is that maybe you would have one lab or, you know, whether it's one country running one lab or one just private industry lab or whatever it is, but like, one would have some, um, lead over the like, just naturally some, like one lab might just be a year or two ahead 'cause they started earlier, uh, were more lucky or had better talent or something. And so then, then that would create an opportunity for this leading lab to slow down, um, for a year or two, however long their lead was, right? Without falling behind. And it might be desirable if rather than having like a super competitive race, you had a little, and then that kind of pause would be self-limiting, you see? Like, because once they have paused for two years, that would be another lab kind of catching up and then maybe they could pause too, but you would have to like make an increasingly strong case for pause as more and more independent-

    5. CWChris Williamson1:24:23

       Mm-hmm.

    6. NBNick Bostrom1:24:23

       ... actors became capable. So it would be a pause that could exist and it would eventually expire. And exactly when it would expire would depend on how s- strong the argument was for AI risk. Um, th- and, and that would it seem create a much lower risk of ending up with like a kind of perma-ban where this technology is never developed. Whereas if the path to getting the ability to pause for a year is to say, set up a big international regulatory regime or like creating a lot of stigma, um, around AI research, like developing some mass movement, like smash the machine type of thing, then that's much more likely to spill over into something that then become a permanent orthodoxy.... or a regulatory apparatus that just has, like, an incentive to perpetuate itself. So, more worrying from that respect.

    7. CWChris Williamson1:25:11

       How impressed have you been with the power of LLMs? Do you think that they are going to be the bootloader for what we need from a superintelligence perspective, or is this... Have you got, uh, limited hopes for how far they can sort of climb, functionally?

    8. NBNick Bostrom1:25:31

       Um, well, I... We, we haven't yet seen the, the limits of, uh, what one can do when scaling these. Um, I, I think it's these transformer models, I mean, it's not just language but they could have other modalities as well. But they, they, they seem, uh, very general, um, and a lot of alternatives that people try turn out in the end to basically just, uh, result in similar performance as transformers. The transformers run, like, well on the current generation of hardware, like, they parallelize very well, et cetera. And so it might be that you need a little thing on top of that, like maybe it's, like, the engine block and then you need some sort of agent loop. Uh, or, or maybe some external memory, uh, augmentation or some other little thing. Um, but that you would still have this big kind of transformer, or something similar to it, like may- there might be some variation. But, uh, as the basic thing that extracts statistical regularities and abstractions, that, that's, uh, pretty plausible, I think.

    9. CWChris Williamson1:26:35

       It's interesting, I, I certainly wouldn't have guessed 10 years ago that, uh, something that you have a conversation with and that is able to accurately predict what it would say would actually be the forefront. You know, it was... it seemed to me, tracking quite closely from whenever, like 2015, 2016, the development of AI that I think your book and then subsequent conversations around AI risk, uh, kind of blew up that conversation. And then it seemed to me that maybe the 2018, 2019, 2020, that it, it, AI hadn't really delivered the threat that people perhaps slightly earlier in the 2010s were worried about. And then, uh, ChatGPT comes along and this conversation just gets thrust straight back into the, the forefront of everything. So it seemed like it had a thrust and then a little lull and then it's really, really sharply come back up again.

Episode duration: 1:28:43