Skip to content

By playing, you agree to uListen's Terms and Privacy Policy and YouTube's Terms of Service.

a16za16z

Why Nick Bostrom Thinks Superintelligence Could End Aging

The question isn’t simply whether humanity should race toward superintelligence or stop.

Nick BostromguestElena BurgerhostEddy Lazzarinhost
Nov 19, 202638mWatch on YouTube

Summaries, chapters, transcripts, and FAQs on this page are made by uListen, not YouTube.

Great conversations run long. Their best ideas don't have to.

uListen is free on iPhone, iPad, Android, Chrome, and Claude.

See how

Transcript

  1. 0:00 – 0:56

    Intro

    1. NB

      Humanity as a whole will need to actually make a decision about what future we want.

    2. EB

      A comment that you made earlier, it could be really bad for humanity if we don't develop superintelligence and really advanced systems.

    3. NB

      Ninety-five percent of everybody who ever lived are already dead. Successfully developing safe superintelligence, I think, could lead to enormous breakthroughs in biomedicine and in anti-aging interventions. Whatever positive impact we are hoping to have in our lives, it's the golden age now for purpose and positive impact.

    4. EL

      Given that these decisions will be made by people in these labs, how do you think about accountability?

    5. NB

      You have AIs that become situationally aware. It might seem more positive for something like this to be developed in a civilian context. It's more possible for many more people to participate now in AI. We have more transparency into these commercial developments.

    6. EB

      Is there an AI model that's so good that all of the important philosophical questions just kind of get obviated?

    7. NB

      Um-

  2. 0:56 – 2:51

    What we’ve learned about AI

    1. EL

      What would you say are the components that have changed most? Uh, of course, like, your, you know, your, your work has gone toward, uh, I would say maybe a, a finer grained analysis of incentives and governance structures of things and sort of the long-term consequences of organizational choices. But would love to hear your characterization.

    2. NB

      Um, yeah. So I think we have a lot more puzzle pieces, uh, today than we had decades ago. Um, puzzle pieces of what you might say the big picture, uh, like the overall macro strategic situation we are in. Um, some set of facts or considerations or insights that would allow us to orient ourselves with respect to which directions are, you know, good and which are bad. Like, do we want, you know, faster progress in some areas, slower, more government involvement, less government involvement, more transparency, less transparency? Like on these kind of basic dimensions, I think, um, we understand more than we did. However, uh, all those little puzzle pieces, I think, still don't really add up to, uh, an ability to form a reliable opinion here. Um-

    3. EL

      Yeah

    4. NB

      ... it's as if you had to, like, um, an, an analogy might be, hmm, suppose you had to, um, determine whether, um, the sum of ten binary numbers was even or odd. Somehow if it's even, you should head north. If it's odd, you should head south.

    5. EL

      Yeah.

    6. NB

      So I'd say maybe, uh, decades ago, we knew three of these binary digits, and now maybe we know seven of them.

    7. EL

      Yeah.

    8. NB

      So in some sense, we know a lot more, but as long as you remain uncertain about one of these, you are still kind of completely in the dark.

    9. EL

      Yep. Yep.

  3. 2:51 – 4:12

    Misconceptions about Bostrom’s views

    1. EB

      Um, I, I was looking at your website, and what I think is funny is there's a section where you list a lot of misconceptions about your work. Um, and I'll just quote a couple. You say, "People think I'm a gung-ho transhumanist or that I'm anti-AI, or that I'm a consequentialist fanatic who would favor any policy purported to mitigate some existential risk." Um, which misunderstandings do you think you're spending the most amount of time correcting right now?

    2. NB

      It depends a lot on, on which specific audience. It's a kind of an evolving set, unfortunately. It's, uh-

    3. EB

      [chuckles]

    4. NB

      Um, I-I-- Well, one, one common one is because the most recent book I wrote was on Deep Utopia, which kind of looks into what happens if things go right. Um, people assume maybe I have radically changed my views and now no longer think there are significant risks with AI. And, um, but, but it-it's more that with any particular, you know, work, you're trying to not, uh, write like a manifesto covering everything. Uh, rather to make some intellectual progress, you need to focus on some specific question and, um, place where you can get traction. Um, so it's more that the scope of the books were different rather than that the underlying view has changed.

  4. 4:12 – 7:29

    Why he wrote Deep Utopia

    1. EB

      How, how did you decide that that's something like deep u-- like the concept of deep utopia or different paths for what utopia would look like? How did you decide that that was a worthy thing that you wanted to write a book about?

    2. NB

      Um, well, at a, uh, superficial level, there was, I guess, some, uh, frustration with how superficial the conversation seemed to be to me, uh, when it started looking at, um, what would actually happen if we do develop machine superintelligence. What's the future like? Um, it often sort of reached a point where people might consider, well, you know, maybe there would then be some impact on the labor market, and so what would we do? Maybe retrain people or something. Uh, but like if you actually think through what it would mean as sort of technology runs to its logical culmination, uh, it involves much more profound, uh, questions about human meaning and so forth. Um, and then another, uh, deeper rationale was that at some point, it seemed to me there might be some person or group of people or maybe humanity as a whole that will need to actually make a decision about what future we want. Um, so imagine going into some room to try to deliberate and determine, uh, uh, all these different paths that get unlocked with more advanced technology. We could transform ourselves in all kinds of way. It's a pretty daunting challenge. Um, and so it seemed maybe useful if there were some, um, as it were, preparatory reading material that such hypothetical future deliberators, um, could go through that might equip them with various ideas and concepts and questions to ask and, and ideally also maybe put them in the right frame of mind, um, to, to approach this, you know, with a, in the spirit of sort of Uh, uh, generous curiosity, open-mindedness while still feeling, uh, in touch with, um, all different kinds of human values. Um, I think maybe the deliberation would be more likely to go better if, if you come at it with some-something like that spirit. And so instilling that was another motivation for writing the book.

    3. EL

      Implicit in what you're saying is when you hear the conversation and it sounds a little more shallow than you'd like to hear, that means it, it, it also implies that you think that it's something valuable and worth, uh, you know, worth edifying, right? And if, uh, you know, when you write about-- when you wrote about superintelligence, many people interpreted that as a sign that you felt that that was something possible, obviously. And when you wrote [chuckles] Deep Utopia, it sounds like that's something that you think is possible. So I, I, I, I get what you're saying about the superficial layer, but there's, there's an implicit recognition that it's a, it's a possibility for the world. Uh-

    4. NB

      Oh, yeah.

    5. EL

      Yeah.

    6. NB

      Yeah, yeah. I mean, and, and what, what superintelligence certainly is possible and, and, and at a wide range of outcomes. Um-

    7. EL

      Yeah

    8. NB

      ... the, the, the, the uncertainty back when I was writing Superintelligence was not whether it was possible to build superintelligence, but what the outcomes of, of that would be.

  5. 7:29 – 11:00

    How humanity should navigate superintelligence

    1. EL

      Right.

    2. EB

      The kind of specificity with which you think through things is a methodology that kind of-- but that, that might be helpful in terms of helping think through our present moment. I think where this, like, uncertainty really comes through, both in Superintelligence and in Deep Utopia, is like, I think you, you grapple a lot with contingency and sort of the idea that a decision or an intervention that is made at one point in time might have a radically different outcome if you do that same decision or that same intervention at a very, very different point in time.

    3. NB

      Yeah. With respect to the AI situation, I think we'll need to sort of feel our way through. Uh, it, it's a rapidly evolving situation, um, and we need to be able to respond as new information comes to light or political realities change around. Um, I haven't been a huge enthusiast of this, uh, stopping AI or pausing AI. Um, I do think the, uh, pacing the frontier, uh, framework has something to be said for it. I could easily imagine there being some point, perhaps in the not distant future, where it would be extremely useful to be able to, um, either have a very time-limited pause, uh, to allow extra safety work to be done on some system, um, before the knobs are all turned up to 11, or, or maybe to have some ability to coordinate, uh, a slight reduction in the pace of capability advances if it looks like safety and alignment i- is trailing behind. Um, it seems that developing this extremely consequential technology in a context where, say, whichever company or product is doing it first, uh, is unable to slow down for more than a few weeks, because if they do, they just become irrelevant and sort of the initiative goes to maybe a less scrupulous, uh, less risk-aware competitor. That, that seems maybe not the best way for humanity to approach this. Um, it, it does seem worth kind of taking a little bit of extra time to make sure we get it right. Um, I, I would start to worry if the method by which we were, uh, planning to achieve that ability to moderate the pace were such that it had a large likelihood of resulting on us, us never developing superintelligence or, or like kind of putting it off. You could imagine if the method is create so much, um, public animosity and antagonism to AI that it gets completely vilified and stigmatized and then set up some extremely powerful regulatory apparatus with global scope to supervise all AI data centers and so forth, that once you start to move to such extreme solutions, then you start to have to worry that they wouldn't just serve this temporary function, but that they would become entrenched in some kind of unchallengeable orthodoxy. And so then it looks far-fetched today, but you know, if that were to happen, then, you know, the pendulum might swing towards kind of another kind of existential risk, which would be a, a permanent failure ever to, to develop superintelligence, which I think it itself would be a, a kind of, um, an existential catastrophe.

    4. EB

      Yeah.

    5. NB

      Um, so, so this doesn't, like, seem very realistic, but, like, look at the trend line of how much anti-AI sentiment has grown just over the past, uh, 18 months, two years. And this is before anything bad actually has happened. Um, so we've, uh, you know-

    6. EL

      Yeah,

  6. 11:00 – 15:51

    Aligning superintelligence

    1. EL

      I don't think it sounds unrealistic at all. At the risk of asking a difficult to form question, uh, couldn't you always construct because of the sort of flexibility and programmability of these things and because of the great improvements in our mechanistic interpretability skills, couldn't you always design a payoff matrix with an AI where you can give it like an infinite femtosecond duration reward in exchange for retiring it peacefully and sort of resolve a lot of the more complex payoff scenarios with humans through the direct manipulation of the, of the model's intelligence capabilities? I mean, may-maybe to put it a, a different, more extreme way is, like, are you, are you genuinely concerned about the construction of a utility monster? And do we need this-- do we need superintelligence capabilities to actually be able to construct this, an, a utility monster today? Like, is that something that you, you, you think we ought to be genuinely concerned of?

    2. NB

      Um, well, I don't think at the moment we have such, um, precise techniques in alignment to, uh, reliably be able to engineer in a precise set of values or goals that, that we would prefer for one of these AIs to have. If we did, then It would be kind of home and dry with at least respect to one of the big challenges of this

    3. EL

      Well, uh, we, we, well, we d- we don't have the alignment capabilities that I, that I'm aware of, but I don't know if you saw something that's been going around X the last couple days. A very ex- very interesting home hacker example of someone who's able to take one of the Qwen models and manipulate only a few neurons at a time and completely change it into a totally, a reportedly, you know, self-describedly, you know, blissful or evil or, you know, psychotic or even, like, Christian or etc. AI. In other words, the single neuron ability to manipulate these and to create what seem like authentic displays of extreme behavior implies that as capabilities improve, we will only gain more fine-grained control over exactly these types of things, including the neurons that could imply some kind of utility payoff, which is not something-

    4. NB

      Yeah

    5. EL

      ... that is presently possible for humans, obviously.

    6. NB

      It, it is, uh, plausible, um, that, um, we will get much better at alignment. Um, in some ways, the problem gets easier the smarter the AIs are, in that they are able to sort of understand what it is that we had in mind, and they can sort of see what we are pointing to. Uh, rather than the literal definition, they might sort of get the underlying gist of a command.

    7. EL

      Yeah.

    8. NB

      Um, in some ways, however, it also gets harder. Um, with less capable AIs, you can rely on, uh, behavioral tasks. So you create your AI, you run it in a different test situations, and then if it behaves well, then well, you know now it's a well-behaving AI, uh, and you can deploy it. Uh, at, at a certain level of cognitive sophistication, you have AIs that become situationally aware that, that can detect whether they're in a test environment or in a deployment environment, and it can reason about that and sometimes choose to act differently because they think they are probably being monitored. And we already see this with, with current, uh, frontier systems. If we look at their chains of thought, they, they do often kind of strategize about how they wanna behave given that, you know, they might be watched or that different things might happen. Um, and so such systems, they might behave one way in a test environment where they realize they are being tested, and then when they think nobody's watching or later when they are in a position really to grab power, uh, their strategic calculation might change radically. Um, this more capable AI system might also be able to subtly influence their own, uh, training process by, um, do- doing different kinds of, um, uh, maneuverings, uh, during the training process to shape their own motivation. So there are-- And, and of course, there's more stuff going on inside it. You know, m- maybe they learn to reason in ways that are not visible in the chain of thought that could then lose us visibility into what they're thinking. Um, so on balance, I think the, uh, alignment problem probably gets harder, um, with these more capable systems. Certainly, the consequences are, are more severe. Nevertheless, I think the main hope is that, you know, we will sufficiently well quasi-align some slightly superhuman systems, that we can mostly get them to do what we want and then have those help us sort of design the next iteration of more aligned AIs and then so on and so forth, so that you sort of have a, hopefully an attractor basin where you just need to get close enough initially, and then the things will kind of turn

  7. 15:51 – 23:09

    The cost of delaying superintelligence

    1. NB

      out okay.

    2. EB

      I wanted to, to go back to, uh, a comment that you made earlier about the risks of postponing superintelligence, um, and, and the idea that, you know, it, it could be really bad for humanity if we don't develop superintelligence and really advanced systems. Um, and it's an idea that, that you have been exploring for a while. Um, you have a paper on the optimal timing for superintelligence. Um, can you explore some of the ideas in there?

    3. NB

      Um, yeah. So this paper, relatively recent one, um, it's, um, it looks at one specific question, which is if, if the goal is to op- maximize the life expectancy of currently existing people, um, how fast would we want AI development to unfold? Um, now, I, I should say it's not the only question that we should be asking. There are also other values and considerations that I think in an all things considered, um, decision should play a big role as well. For example, we might be worried about future generations, uh, like existential risks, uh, fairness, all, all kinds of things, right? But this is one thing that one might look at. Um, and, um, so, so, so Eliezer Yudkowsky, uh, and Nate Soares had, had this book, right? "If anyone builds it, everyone dies." Um, and, and, and it seemed to me that we have a- at least as strong evidence for the claim that if, if nobody builds it, everyone dies. Uh, it, like in fact, 95% of everybody who ever lived are already dead, and, uh, the rest of us are kind of on track to follow them within a few short decades. Um, whereas, um, successfully developing safe superintelligence, I think, um, could lead to, um, enormous breakthroughs in biomedicine and in anti-aging interventions that could radically extend the lifespan of currently existing people. Um, so sort of, uh, if, if you, if you think about what our life expectancy would be if we weren't aging, it's something on the order of, you know, 1,200, maybe 1,400 years or so. Um, and that maybe is a quite conservative lower bound because in this situation you might also have other ways of reducing other, uh, mortality risks like accidents and so forth. Um, m- So you then have, uh, a, a number of different competing factors. On the one hand, you might think that if you took some extra time with this AI transition, maybe we could, uh, um, build AI more safely. So you might think that if we spend more time doing AI alignment research, the, the existential risk from developing AI goes down at some rate. And then on the other hand, um, every year that we delay it, we have, um, millions and millions of people dying. Um, it, I think it's like, um, what is it? Like, uh, like 100-

    4. SP

      I think it's like 150,000 people dying

    5. NB

      ... people every day.

    6. SP

      Yeah.

    7. NB

      Um, so it, it's, you don't, you don't wanna delay a, a single day longer than you really need to. But then you can, on different assumptions, kind of model at what rate does the risk have to decline for it to be worth sort of, m- you know, waiting longer.

    8. SP

      Yeah.

    9. NB

      Uh, and you can add different things. You might, for example, some people might have diminishing, um, marginal return in additional life years, so they might be not happy to take, like, a 51% chance of living to 200 rather than 100% chance of living to 100 and so forth. So, and you might have time discounting. But for a wide range of parameters, it looks like the optimal, uh, delay is, um, relatively short, uh, as in sort of maybe months or a few years as, as opposed to decades.

    10. SP

      Yeah.

    11. NB

      Um, and, and you can see this, that if the, if the rate at which, uh, the risk level goes down from, from a delay is, is very fast, like you, you develop some sort of almost superintelligent system, you pause. If at that point the risk goes down very quickly, um, then you don't need to wait very long, right? You just wait for a few months, and then the risk is so low that you can go ahead. If, on the other hand, the, the rate goes down very slowly, uh, then you don't wanna wait any time at all, right? Because more people are dying per year, uh, from natural causes than are, like, the expected number of people saved by waiting. So it's only if you have a sort of intermediate rate of safety progress that you would wait. Now, I think plausibly the, the, uh, the very early days after you have the ability to create AI would be especially valuable for AI safety research. You have this almost final system that you can study and test and sort of slowly deploy, watching extra carefully. I think having a little bit of extra time there could yield, like, large safety gains. But then I think the, the, uh, rate of additional safety gains will sort of start to decline after, uh, like a few years certainly.

    12. SP

      It's funny to think we're, we're actually, like, living versions of, of this question already. If you look at Waymo or if you look at Ozempic or something, it's like Waymos are empirically much safer than, you know, humans driving cars, and yet we still have humans driving cars. Um, you know, theoretically, every obese person should be on Ozempic, but they're not. Um, so, so there-- we, we do have versions of these like, oh, we actually do have interventions for things that historically have been, you know, extremely deleterious for humanity or deleterious for humanity and, and we just like-- like it, it seems like there are other things that hold us back even after we have the technology that's been developed.

    13. NB

      Yeah. Um, I mean, I think life expectancy has been increasing by about two or three months per year, so we, we need to get it above one year per year.

    14. SP

      Yeah.

    15. NB

      Um, and so, like it-- I, I-- it, it seems very doubtful that absent superintelligence, we would... I mean, you might hope for independently just some kind of breakthroughs in synthetic biology and biomedicine, like ways of actually s-seriously slowing down the aging process. I think if we got that or maybe from some early applications of not yet radically superintelligent AI systems that could sort of fix aging and, and take care of some of the other, like, particularly pressing problems in the world, then, like, maybe we would then become more patient. Um, you could, could afford to take it slow with the more radical versions if, um... But, but as things are. So the life e-e-expectancy is one. Of course, there are other pressing problems as well. Des-desperate poverty, suffering of all kinds amongst humans and animals, and, uh, m-many, many things where, where, like, these, these AIs, if, if developed well, could really, like, you know, become like the, the cavalry that rides into the rescue at, at a, at a critical moment in, in the battle. Um, so, um, so, so the, the stakes are high either way. As I said, I think that's only one of many different considerations that would come into an all things considered, um, evaluation here.

  8. 23:09 – 27:10

    Building superintelligence safely

    1. SP

      Yeah. To, to what extent do, does your thinking in the paper rely on being able to identify something like a discrete threshold? Because the way that it's framed is, you know, we, we have the ability to release, like, we have the cap- we have the ability to release or deploy a superintelligence, which of course relies on some kind of a definition of a superintelligence. And then we are able to measure, in some sense, the risk of its releasing and whether or not we are able to decrease that risk. Now, I, I'll grant that I think-- I, I don't think it's re- we have to have an ultra-discrete, you know, bright line beyond which in realities these things often play out as a patchwork of capabilities are analyzed separately and specific behaviors, specific risk scenarios are evaluated, right, and tests are performed to understand each of them independently as opposed to as this whole. Right? But to what extent do you need to be able to identify and define it? How do you think we ought to identify and define which are these risk thresholds at which we should consider pausing?

    2. NB

      Yeah. So in practice, that's of course what we need to do. So, so this, this particular paper is just a small paper, but it-it's just meant to create a sort of simple model to illustrate the way that the urgency dynamic, uh, of People dying from natural causes kind of interacts with the risk parameter.

    3. EL

      Yeah.

    4. NB

      Um, just to kind of intellectually be able to l-look at that particular aspect of it, because that does seem to be a kind of point where different people arrive at very different views about AI, depending on, you know, where they sit on that. Uh, but then, of course, in practice, like we face not just some hypothetical binary decision about-

    5. EL

      Right

    6. NB

      ... number of years delay, but we f-f-- like each different actor is confronted with like a complex situation with uncertainty and particular things they could do. Um, and so there, I would say, um, uh, there might, for example, be different ways of increasing the capability of the AI systems that, uh, uh, have different implications for safety and, and there might be ways of pushing towards greater capabilities that are comparatively safer than others. So, for example, one way to increase capability might be to increase the recursive depth by moving to looped transformers or other architectures that allow, um, the model to do more thinking within one forward pass or before output-- emitting a, a token in its chain of thought. Um, uh, and that has the disadvantage that it reduces our ability to see what they are thinking. Like, they can do more thinking between each output that we can see. And then you might imagine a different way of increasing the capability, maybe giving the same performance boost, but maybe by increasing the number of parameters in, in the network or something like that. So may-maybe that, that is relatively less costly from a safety point of view than, uh, you know, increasing, uh, recursive depth. Or, or training these systems to, uh, think in neuralese, m-maybe that would make them more efficient at thinking, but then-

    7. EL

      Yeah. So you-

    8. NB

      Not get caught

    9. EL

      ... you're talking about different design setups that create trade-offs between interpretability and capability.

    10. NB

      Yeah. Or between some other risk-relevant dimension-

    11. EL

      Right

    12. NB

      ... and, and capability.

    13. EL

      Right.

    14. NB

      And so, uh, I think a lot of the actual action would be more at, at that kind of level of granularity, where perhaps if there were some ability, uh, for frontier labs to coordinate, to, say, refrain from using some specific way of increasing capability that seem particularly problematic from, like, loss of control point of view and focus on competing, uh, along the other dimensions. That, that might be the kind of thing that would be the right trade-off. Um, sacrificing a little bit of, like, maybe it costs a little bit more to pursue capability in these other dimensions, but, like, we gain, hopefully, greater

  9. 27:10 – 30:27

    AI research isn’t the Manhattan Project

    1. NB

      safety.

    2. EL

      So given that these decisions will be made by people in these labs, at least, uh, assuming it's set up the way it is now, how do you think about accountability? Because you, you-- and governance of these things. You, you, you know, you talk about nationalization as a risk, right? Presumably because it could-

    3. NB

      Well

    4. EL

      ... create delay. But what are the-- how do you think about accountability and governance?

    5. NB

      I mean, whether nationalization is a risk, um, I, I don't really have a strong-- It's, it's one of those things where I think we need to feel our way through, and the, the judgment of whether you would want that or not might sort of change depending on, like, who, who-- if, if it's not nationalized, then exactly who is in the lead and doing what, and if it were nationalized, like, in what precise way would it be? I think, I mean, the case for would be, wow, this is like a crazy, it's more powerful than nukes. Isn't it, like, insane that some random corporation trying to maximize shareholder profits should do this? Like, with, with nuclear weapons, that was a Manhattan Project. You could have much better information security. It's like one big project rather than different competing ones. On the other hand, um, in favor of the current, uh, situation, I think, um, nuclear weapons were developed in a military context. It might seem more positive for something like this to be developed in a civilian context, not focused on, like, maximizing our ability to kill people, but on providing benefits to consumers. Um, uh, we have more transparency, um, into these commercial developments than a Manhattan Project. Like, even the vice president at the time was unaware that the Manhattan Project even existed. So you could imagine extreme sort of concentration of power dynamics. It's run by these kind of national security types. You know, maybe you think these idealistic tech nerds are more, you know, less scary. Um, uh, it's more possible for many more people to participate now in AI, like, either as consumers of the technology, um, you know, or, you know, like think tanks or, or external evaluators looking at these models, investors, investors in the model-building companies or in the semiconductors apply-- Like, the kind of the benefits and participation is much more broadly diffused among society and around the world that might be preferable to having some, like, very concentrated project that's secret with, you know, some, um, you know, small number of people, nobody really know who they are, sit on top of, like, some people have a lot more money than others. But if you compare, say, the Gini coefficient of global capital ownership, uh, or stock ownership, both directly and through pension funds and sovereign wealth funds and stuff like that, it actually is a slightly less unequal distribution than a distribution where, say, you had a, a US national project where even if in, like, you granted the very implausible assumption that every US person would have exactly equal say and stake and benefit from that, and everybody else would have zero. That's actually a more unequal distribution globally than, than one where it's in proportion to current capital, uh, assets. Um, so even though it's not the most ideal from a distributional point of view, this open global investment model, it actually scores better on that than the most likely alternative.

  10. 30:27 – 32:49

    Why our vision of the future matters now

    1. EB

      Are there any kind of questions relevant to humanity today that you're exploring right now that feel particularly salient?

    2. NB

      I think like our main focus probably at this point should be, uh, how to get from there, here to there. Um-

    3. EB

      Mm-hmm

    4. NB

      ... like the, it looks like we're entering this really critical period, uh, where maybe the long-term future for Earth-originating life will, will be determined one way or the other. And so getting that right should, should be a huge priority. Um, you know, I think these can be somewhat relevant in, um, choices we currently make. Uh, for example, the level of, uh, existential risk you're willing to accept from the development of superintelligence might depend in part on how you value the opportunity later on to become a post-human, uh, type of entity. Um, if you think that the best we can possibly achieve is basically status quo human beings, you know, um, with, with slightly higher, you know, uh, standard of living and some new medicines, but that's like the ceiling, then, you know, things could still get better relative to where they are now, and it would be worth taking some risk for achieving that sort of uptick, but not like an enormous risk, maybe. Especially if you're already sitting, you know, in a rich country with, you know, well-educated, a lot of money, privileges of all kinds. You think, "Well, it's pretty good." You know? Like on the other hand, if you think that, you know, we ain't seen nothing yet. Like, okay, we have been this kind of monkey-like creature hopping around here and doing our little thing, but we have no clue, like if you unlock this larger space of possible modes of being, the kind of, uh, super felicity, like the cognitive uplift, the emotional depth of, um, appreciation for what existence could mean, different ways of relating and creating together. Like, if you think there's like a large, a huge space of sort of unfathomed upside there, then you might be more inclined to risk more to have a chance to have a shot at that. Um, so, so I do think even though the main focus should be on these more sort of pragmatic trade-offs and, uh, technical improvements that we're able to make, like I do think that kind of background picture of what you ultimately think is possible does have some relevance as well into how we should approach the choices in front of us.

  11. 32:49 – 38:28

    Does life have a meaning, and can AI answer it?

    1. EB

      Two, two, two ones, two questions for Nick. Now, I don't mean to make them sound quick. Th-this is a weird one. This is a weird one, Nick. It's just, do you think that there is a telos to the cosmos?

    2. NB

      Um, well, what, what do you mean by the cosmos? [chuckles]

    3. EB

      Creation, so to speak. I mean... [chuckles]

    4. NB

      Well, I mean, if it's a creation literally, then, uh, probably-

    5. EB

      Yeah

    6. NB

      ... the creator had some purpose for creating it. Um, now the reason why I'm asking is, so I have this, one of my early work, like, uh, on this, uh, introduced simulation argument, um, which is maybe a different can of worm. But, um, if, if you take, um, the hypothesis seriously that, um, uh, we are living in a computer simulation, then there, there's an important distinction about the part of reality that we see, our simulation, and the totality of physical existence. And if we're in a simulation, then probably the people creating the simulation had some reason for doing that, which is, I guess, depending what you mean by purpose, but there would exist some purpose in that sense. Um, the totality of existence, then it's more of a question of maybe theological, um, uh, considerations, whether you would think that the sort of totality somehow was purposely created. Now, d- asking about purpose could mean many different other things as well.

    7. EB

      Yeah.

    8. NB

      Like different people have different purposes for different... Uh, I, I think like, um, one question one might ask, uh, is whether, um, and I'm, uh, well, I think, uh, yeah, is whether there is some set of facts about the world such that if we understood them clearly, we would basically all come to the same conclusion about what we ought to be doing. And if there is such a fact, then maybe you could say that's a kind of objective purpose in some sense. Um, if either everybody or maybe, uh, in the, for, for a wide range of possible goals and motivations you have, it could be the case that they have this kind of convergence in, in that if you properly understood all the facts, they would sort of point to the same practical next step. Uh, and that, I think, is a lot more of an open question.

    9. EB

      Yeah.

    10. NB

      Um-

    11. EB

      And maybe something I could-

    12. NB

      Like one way that c-

    13. EB

      Oh, sorry, go ahead.

    14. NB

      Uh, uh, so one way that that could be, of course, is in a traditional theological, uh, um, context, right? If, if, if like a divine being has this goal and, and like, um, re- rewards people are not depending on, like, then that might just be such that if that is true and if you understood that, then basically almost independently of what your particular goals are, you would have the most reason to sort of follow the prescriptions there. Um-

    15. EB

      Yeah

    16. NB

      ... if, if, if we are sort of in a, uh, you know, purely materialistic, like the kind of worldview that a lot of secularists have, where there's laws of nature just rattling around, then that might not be such a convergence. Like what you have most reason to do depends on your particular goals, and somebody else has different goals, so there is no higher fact that sort of would cause you to have reason to do the same thing. Um, but I think there's like, yeah, a lot, um, of, of possibilities, uh, between those two that we don't fully understand yet that could make it such that there is in some sense a purpose, even if we are not quite clear on what it would be.

    17. EB

      Yeah. Um, I wasn't gonna ask this question, but, but you brought up, you know, if there, if there is, you know, the, the one answer that all of humanity, uh, could be guided toward and, and that would kind of like answer everything. I, uh, so, so now the natural follow-up is, let's say AI gives us that answer. Superintelligence gives us that answer. Do you stop practicing philosophy? Like is there, is there an AI model that's so good that, you know, all of the important philosophical questions just kind of get obviated?

    18. NB

      Um, yeah, I, I think that we, uh, will have, uh, sort of, um, AI set surpass us in, in phi-philosophy, um, just as we now have in math and, and, um, in probably soon in other fields. Um, um, there might be a subset of questions that have the form more of us making up our minds about what we want, um, where it might be the case that we need to do some of the deliberation and deciding ourselves. Um, um, but broadly speaking, yes, uh, I, I do, I do think that our own intellectual efforts in that, uh, realm also will become sort of obsoleted by, by these superintelligences. And so which means that right now, um, whilst we still have something to contribute is, is the time to sort of really, uh, try to have our-- whatever positive impact we are-

    19. EB

      Yeah

    20. NB

      ... we are hoping to have in our lives. It's the golden age now for, for, for, for purpose and positive impact. And then, you know, once it's out of our hands, then, you know, it's time to maybe, uh, kick back a little bit and, uh-

    21. EB

      [chuckles] And enjoy

    22. NB

      ... and, and speaking of, is there anything that you, uh, wanted to discuss or that we should have covered that we, we haven't so far? I, I mean, it's, like, so many different directions we could go with. Like, each, each of the topics we touched on in itself is a kind of-- could almost be like a one-hour conversation, right? So you can-

    23. EB

      Yeah.

    24. NB

      Um, but yeah, no, may-may-may-maybe at some point in the future we could do another one or something. But, um-

    25. EB

      Yeah

    26. NB

      ... there's nothing in particular right now.

    27. EB

      Would love that. Um, well, thank you, thank you so much for, for joining us. Uh, really, really appreciate you taking the time. This was so much fun.

    28. NB

      It was fun, yeah.

    29. EB

      Yeah.

Episode duration: 38:48

In the uListen app, the transcript lights up word by word as you listen. Get uListen

Transcript of episode gDW9An30_c0

Get more out of YouTube videos.

Read along on your phone, search it beside the video, or ask your favorite AI app what was said.