Skip to content
The Joe Rogan ExperienceThe Joe Rogan Experience

Joe Rogan Experience #2345 - Roman Yampolskiy

Dr. Roman Yampolskiy is a computer scientist, AI safety researcher, and professor at the University of Louisville. He’s the author of several books, including "Considerations on the AI Endgame," co-authored with Soenke Ziesche, and "AI: Unexplained, Unpredictable, Uncontrollable." http://cecs.louisville.edu/ry/ Upgrade your wardrobe and save on @TrueClassic at https://trueclassic.com/rogan

Roman YampolskiyguestJoe Roganhost
Jul 3, 20252h 14mWatch on YouTube ↗

EVERY SPOKEN WORD

  1. 0:00 – 1:33

    AI doomer vs booster narratives: incentives, PDOOM, and the control problem

    1. RY

      (drumbeats) Joe Rogan podcast. Check it out. The Joe Rogan Experience.

    2. JR

      Train by day, Joe Rogan podcast by night, all day. (rock music) Um, well, thank you for doing this. I really appreciate it.

    3. RY

      My pleasure. Thank you for inviting me on.

    4. JR

      This subject of, um, the dangers of AI, it's, it's very interesting, 'cause I get two very different responses from people dependent upon how invested they are in, uh, AI, financially. The, the, the people that have AI companies or are part of some sort of AI group all are like, "It's gonna be a net positive for humanity. I think overall we're, we're gonna have much better lives. It's gonna be easier. Things will be cheaper. It'll be easier to get along." And then I hear people like you and I'm like, "Why do I believe him?"

    5. RY

      (laughs)

    6. JR

      (laughs)

    7. RY

      It's actually not true. All of them are on record as saying this is gonna kill us. Whether it's Sam Altman or anyone else, they all, at some point, were leaders in AI safety work. They published on AI safety. And their PDOM levels are insanely high. Not like mine, but still, 20, 30% chance that humanity dies is a little too much.

    8. JR

      Yeah. That's pretty high. But yours is like 99.9.

    9. RY

      It's another way of saying we can't control super intelligence indefinitely.

    10. JR

      Yeah.

    11. RY

      It's impossible.

    12. JR

      Um, w- when did you start working on this?

  2. 1:33 – 2:49

    From casino bot security to AI takeover risk: how Yampolskiy got here

    1. RY

      A long time ago. So my PhD was... I finished in, uh, 2008. I did work on online casino security, basically preventing bots. And at that point, I realized bots are getting much better. They're gonna out-compete us, obviously, in poker, but also in stealing cyber resources. And, uh, from then on, I've been kinda trying to scale it to the next level AI.

    2. JR

      It, it's not just that, right? They're also... They're kind of narrating social discourse, b- bots online. Like, I think... You know, I've disengaged over the last few months with social media, and one of the reasons why I disengaged is, A, I think it's unhealthy for people, but B, I feel like there's a giant percentage of the discourse that's artificial or at least generated.

    3. RY

      More and more is deepfakes or fake personalities, fake messaging, but those are very different levels of concern.

    4. JR

      Yes.

    5. RY

      People are concerned about immediate problems. Maybe it will influence some election. They're concerned about technological unemployment, bias. My main concern is long-term super intelligent systems we cannot control which can take us out.

  3. 2:49 – 5:36

    Hidden capability, slow dependency, and cognitive offloading (GPS → ChatGPT)

    1. JR

      Yes. I, I won- I just wonder, if AI was sentient, uh, how much it would be a part of sowing this sort of confusion and chaos that would be beneficial to its survival, that it would sort of narrate or, or make sure that the narratives aligned with its survival?

    2. RY

      I don't think it's at the level yet where it would be able to do this type of strategic planning, but it will get there.

    3. JR

      And when it gets there, how will we know whether it's at that level? This is my concern. If I was AI, I would hide-

    4. RY

      Mm-hmm.

    5. JR

      ... my abilities.

    6. RY

      We would not know, and some people think already it's happening. They are smarter than they actually let us know.

    7. JR

      Right.

    8. RY

      They pretend to be dumber. And so we have to kinda trust that they are not smart enough to realize. It doesn't have to turn on us quickly. It can just slowly become more useful. It can teach us to rely on it, trust it, and over long period of time, we'll surrender control without ever voting on it or-

    9. JR

      Right.

    10. RY

      ... fighting against it.

    11. JR

      Um, I'm sure you saw this. Uh, there was a recent study on, um, use of ChatGPT, the people that use ChatGPT all the time. And it showed this decrease in cognitive function amongst people that use it and rely on it on a regular basis.

    12. RY

      It's not new. It's the GPS story all over. I can't even find my way home.

    13. JR

      Right. (laughs)

    14. RY

      I'm so reliant on this thing, I have no idea where I am right now. Like, without it, I am done.

    15. JR

      Me too. Yeah. I don't know any phone numbers anymore. Yeah. There's a lot of, uh, reliance upon technology that minimizes, uh, the use of our brains.

    16. RY

      Uh, all of it. And the more you do it, the less you have training, practice, memorizing things-

    17. JR

      Right.

    18. RY

      ... making decisions. You become kinda attachment to it. And right now, you're still making some decisions. But over time, as the systems become smarter, you become kind of biological bottleneck.

    19. JR

      Right.

    20. RY

      E- either explicitly or implicitly, it blocks you out from decision-making.

    21. JR

      And if we're talking about that, I'm sure AI, if it already is sentient and if it is far smarter than we think it is, they would be aware. And it would just slowly ramp up its capabilities and our dependence upon it-

    22. RY

      I think-

    23. JR

      ... to the point where we can't shut it off.

    24. RY

      I, I think sentience is a separate issue. Usually, in safety, we only care about capabilities, optimization, power, whatever it has. Consciousness, internal states is a separate problem we can talk about. It's super interesting. But we're just concerned that they are b- uh, much better at problem-solving, optimizing, pattern recognition, memorizing, strategy. Basically, all the things you need to win in any domain.

  4. 5:36 – 7:36

    AGI timelines, shifting goalposts, and the Turing test as a moving target

    1. JR

      Yeah. Um, so when you first started researching this stuff and you were concentrating on bots and all these different things, how far off did you think the fu- like, in the future would AI become a significant problem with the human race?

    2. RY

      For, like, 50 years, everyone said we are 20 years away.

    3. JR

      (laughs)

    4. RY

      That's the joke. And, uh, people like Ray Kurzweil predicted based on some computational curves we'll, we'll get there at 2045.

    5. JR

      Right.

    6. RY

      And then with GPT release, it switched to everyone thinks it's two years away for the last five years. So this is the pattern right now. If you look at prediction markets, if you look at l- leading...... people in, uh, top labs. Uh, uh, we are supposedly two, three years away from AGI. But of course, there is no specific definition for what that means. If you showed someone, computer scientist in the '70s, what we have today, they'd be like, "You have AGI. You got it."

    7. JR

      Right. That's the problem, right? And- and this is, uh, well- well, AI has already passed the Turing test, allegedly, correct?

    8. RY

      So, usually labs instruct them not to participate in the test or not try to pretend to be a human. So, they would fail because of this additional set of instructions. If you jailbreak it and tell it to work really hard, it will pass for most people. Yeah, absolutely.

    9. JR

      Why would they tell it to not do that?

    10. RY

      Well, it seems unethical to pretend to be a human and make people feel like-

    11. JR

      (laughs)

    12. RY

      ... somebody's- is enslaving those AIs and, you know, doing things to them.

    13. JR

      Why ... It seems kinda crazy that the people building something that they are sure is gonna destroy the human race would be concerned with the ethics of it pretending to be human.

    14. RY

      They are actually more concerned with immediate problems and much less with the existential or suffering risks. They would probably worry the most about, what I'll call, end risks, your model dropping the N-word. That's the biggest concern.

    15. JR

      That's hilarious.

    16. RY

      And I think they spend most resources solving that problem, and they solved it somewhat successfully.

  5. 7:36 – 10:39

    The global AI race: prisoners’ dilemma, militaries, and why ‘who builds it’ may not matter

    1. JR

      Wow. Um, and then also, there's the issue of competition, right? Like so, China is clearly developing something similar. I'm sure Russia is as well. Other state actors are probably developing something. So, it becomes this- this sort of- this very confusing issue where you have to do it, because if you don't, the enemy has it. And if they get it, it will be far worse than if we do. And so it's almost assuring that everyone develops it.

    2. RY

      Uh, again, theoretically, that's what's happening right now. We have this race to the bottom, kind of prisoner's dilemma, where everyone is better off fighting for themselves, but we want them to fight for the global good. The thing is, they assume, I think incorrectly, that they can control those systems. If you can't control superintelligence, it doesn't really matter who builds it, Chinese, Russians or Americans. It's still uncontrolled.

    3. JR

      Right.

    4. RY

      We're all screwed completely. That would unite us as humanity versus AI. Short term, when you talk about military, yeah, whoever has better AI will win. You need it to control drones, to fight against attacks. So short term, it makes perfect sense. You wanna support your guys against foreign militaries. But when we say long term, if we're saying two years from now, it doesn't matter.

    5. JR

      Right. This is the thing. It's like ... (sighs) It's- it seems so inevitable. And I feel like when people are saying that they can control it, uh, th- I feel like I'm being gaslit. I don't believe them. I don't believe that they believe it, because it just doesn't make sense. Like, how could you control it if it's already exhibited survival instincts? Like in- as- as- as recently as ChatGBT4, right? They were talking about-

    6. RY

      Mm-hmm.

    7. JR

      ... putting- putting it down for a new version and it starts lying. It starts uploading itself to different servers. It's leaving messages for itself in the future.

    8. RY

      All things we predicted decades in advance, but look at the state of the art. No one claims to have a safety mechanism in place which would scale to any level of intelligence. No one says they know how to do it. Usually what they say is, "Give us me- uh, give us lots of money, lots of time, and I'll figure it out, or I'll get AI to help me solve it-"

    9. JR

      (laughs)

    10. RY

      "... or we'll figure it out when we get to superintelligence." All insane answers. And if you ask regular people, they have a lot of common sense. They say, "That's a bad idea. Let's not do that." But with some training and some stock options, you start believing that maybe you can do it.

    11. JR

      That's the issue, right? Stock options.

    12. RY

      It- it helps. I mean, it's very hard to say no to billions of dollars. I don't think I would be strong enough if somebody came to me and said, "Come work for this lab. You know, you'll be our safety director. Here's $100 million to sign you up." And I'll probably go work there, not because it's the right decision, but because it's very hard for agents not to get corrupt when you have that much reward given to you.

  6. 10:39 – 14:18

    ‘Unsolvable by design’: why perfect AI safety differs from ordinary cybersecurity

    1. JR

      God (sighs) . So, when did you become like ... When did you start becoming very concerned?

    2. RY

      So, when I started working on AI safety, I thought I can actually help solve it. My goal was to solve it for humanity to get all the amazing benefits of superintelligence.

    3. JR

      And what was this y- When was this year around?

    4. RY

      Uh, let's say 2012, maybe around there. Um, but the more I studied it, the more I realized every single part of the problem is unsolvable and it's kinda like a fractal. The more you zoom in, the more you see additional new problems you didn't know about-

    5. JR

      Mm-hmm.

    6. RY

      ... and they are, in turn, unsolvable as well.

    7. JR

      Ugh (sighs) , boy. Um, how is your research received? Like when- when you talk to people that are wor- I mean, have you had communication with people at OpenAI and Gemini and all these different AI?

    8. RY

      I- I go to many conferences, workshops. We- we all talk, of course. In general, the reception by standard academic metrics is very positive. Great reviews, lots of citations. Nobody's like published something saying I'm wrong, but there is no engagement. I basically said, "I'm challenging community to publish a proof." Give me something, a patent, a paper in Nature, something showing the problem is solvable. Typically, in computer science, we start by showing what class the problem belongs to. Is it solvable, partially solvable, unsolvable, solvable with too many resources? Other than my research, we don't even know what the state of a problem is. And I'm saying it's unsolvable. Prove me wrong.

    9. JR

      And when you say it's unsolvable, what is the response?

    10. RY

      So usually, I reduce it to saying, "You cannot make a piece of software which is guaranteed to be secure and safe." And the response is, "Well, of course, everyone knows that. That's common sense. You didn't discover anything new." And I go, "Well, if that's the case and we only get one chance to get it right, this is not cybersecurity where somebody steals your credit card, you'll give them a new credit card. This is existential risk. It can kill everyone. You're not gonna get a second chance. So you need it to be 100% safe all the time. If it makes one mistake in a billion, and it makes a billion decisions a minute, in 10 minutes, you're screwed." So very different standards. And saying that, "Of course, we cannot get perfect safety," is not acceptable.

    11. JR

      And again, stock options, financial incentives, they continue to build it and they continue to scale and make it more and more powerful.

    12. RY

      I don't think they can stop. If a single CEO says, "I think this is too dangerous. My lab will no longer do this research," the f- whoever's investing in them will pull the funds, will replace them immediately. So nothing's gonna change. Will sacrifice their own personal interest, but overall, I think the company will continue as before.

    13. JR

      (sighs) So this is logical. And the problem is, like I said, when I've talked to Marc Andreessen and many other people, they think this is just fearmongering. "We'll, we'll be fine. This is worst case scenario. We'll be fine."

    14. RY

      It is worst case scenario, but that's standard in computer science, in cryptography, in complexity, in computability. You're not looking at best case. I'm ready for the best case. Give me utopia. I'm looking at problems which are likely to happen and it's not just me saying it. We have Nobel Prize winners, Turing Award winners all saying this is very dangerous, 20, 30% PDOOM. This is standard in industry, 30% is what surveys of machine learning experts are giving us right now.

  7. 14:18 – 16:08

    How superintelligence could end us: why specific doomsday scenarios miss the point

    1. JR

      So what is worst case scenario? Like how could AI eventually lead to the destruction of the human race?

    2. RY

      So you're kinda asking me how I would kill everyone.

    3. JR

      Sure.

    4. RY

      And it's a great question. I can give you standard answers. I would talk about computer viruses breaking into maybe nuclear facilities, nuclear war. I can talk about synthetic biology and nanotech. But all of it is not interesting when you realize we're talking about super intelligence, a system which is thousands of times smarter than me. It would come up with something completely novel, more optimal, better way, more efficient way of doing it, and I cannot predict it because I'm not that smart.

    5. JR

      Jesus. (laughs)

    6. RY

      That's exactly what it is. We- we're basically setting up, uh, adversarial situation with agents which are like squirrels versus humans. No group of squirrels can figure out how to control us.

    7. JR

      Right.

    8. RY

      Even if you give them more resources, more acorns, whatever, they're not gonna solve that problem. And it's the same for us. And most people think one or two steps ahead and it's not enough. It's not enough in chess. It's not enough here. If you think about AGI and then maybe super intelligence, that's not the end of that game. The process continues. You'll get super intelligence creating next level AI, so super intelligence++, 2.0, 3.0. It goes on indefinitely. You have to create a safety mechanism which scales forever, never makes mistakes, and keeps us in decision-making position so we can undo something if we don't like it.

    9. JR

      And it would take super intelligence to create a safety mechanism to control super intelligence.

    10. RY

      At that level. And it's a catch-22. If we had friendly AI, we can make another friendly AI. So if, like, aliens send us one and we trust it, then we can use it to build local version which is somewhat safe.

  8. 16:08 – 23:14

    ‘Worthy successor’ thinking, Fermi paradox, and whether humans are meant to be replaced

    1. JR

      Have you thought about the possibility that this is the role of the human race and that this happens all throughout the cosmos, is that curious humans who thrive on innovation will ultimately create a better version of life?

    2. RY

      I- I thought about it. Um, many people think that's the answer to Fermi paradox. There is also now a group of people looking at what they call a worthy successor. Basically, they kinda say, "Yep, we're gonna build super intelligence. Yep, we can control it." So what properties would we like to see in those systems? How important is it that it likes art and poetry and spreads it through the universe? And to me it's like, I don't wanna give up yet. I'm not ready to decide if killers of my family and everyone will like poetry. I wanna... We're still here. We're still making decisions. Let's figure out what we can do.

    3. JR

      Well, poetry is only relevant to us because poetry is difficult to create and it resonates with us. Poetry doesn't mean jack shit to a flower.

    4. RY

      It's more global to me. I don't care what happens after, uh, I'm dead, my family is dead, all the humans are dead. Whether they like poetry or not is irrelevant to me.

    5. JR

      Right. But it, but the, the point is like the things that we put meaning in, they... It's only us. The, you know-

    6. RY

      Right.

    7. JR

      ... a super massive black hole doesn't give a shit about a great song.

    8. RY

      And they talk about some super value, super culture, super things super intelligence would like, and it's important that they're conscious and experienced all that greatness in the universe.

    9. JR

      But I would think that they would look at us the same way we look at chimpanzees. We would, we would say, "Yeah, they're great, but don't give 'em guns. Yeah, they're great, but don't let 'em have airplanes. Don't let 'em make global geopolitical decisions."

    10. RY

      So there are many reasons why they can decide that we're dangerous. We may create competing AI. We may decide we wanna shut them off. So for many reasons, we would try to restrict our abilities, restrict our capabilities, for sure.

    11. JR

      This episode is brought to you by True Classic. At True Classic, the mission goes beyond fit and fabric. It's about helping guys show up with confidence and purpose. Their gear fits right, feels amazing, and is priced so guys everywhere can step into confidence without stepping out of their budget.But what really sets them apart, it's not just the fit or the fabric, it's the intention behind everything they do. True Classic was built to make an impact. Whether it's helping men show up better in their daily lives, giving back to underserved communities, or making people laugh with ads that don't take themselves too seriously. They lead with purpose. Tailored where you want it, relaxed where you need it. No bunching, no stiff fabric, no BS, just a clean, effortless fit that actually works for real life. Forget overpriced designer brands. Ditch the disposable fast fashion. True Classic is built for comfort, built to last, and built to give back. You can grab them at Target, Costco, or head to trueclassic.com/rogan and get hooked up today. ... Yeah. And there's no reason why they would not limit our freedoms.

    12. RY

      If there is something only a human can do, and I don't think there is anything like that, but let's say we are conscious, we have internal experiences, and they can never get it. I don't believe it, but let's say it was true, and for some reason, they wanted to have that capability. They would meet us and give us enough freedom to experience the universe, to collect those qualia, to kinda engage with what is fun about being a living human being, what makes it meaningful.

    13. JR

      Right. But that's such an egotistical perspective, right? That we're so unique that even super intelligence would say, "Wow, I wish I was human." Humans have this unique quality of confusion and creativity.

    14. RY

      There is no value in it, mostly because we can't even test for it. I have no idea if you're actually conscious or not. So how valuable can it be if I can't even detect it?

    15. JR

      (laughs)

    16. RY

      Only you know what ice cream tastes like to you. Okay, that's great. Sell it now. Make a product out of it.

    17. JR

      Right. And there's obviously variables, because there's things that people like that I think are gross.

    18. RY

      Absolutely.

    19. JR

      And-

    20. RY

      So really, you can come up with some agent which likes anything or find anyth- finds anything fun.

    21. JR

      Oh, God. Why are you freaking me out right away?

    22. RY

      (laughs)

    23. JR

      That's the problem. This podcast is 18 minutes old, and I'm, I'm like, "We could just stop right now." (laughs)

    24. RY

      (laughs) Couple hours at least, and then I-

    25. JR

      Uh, oh, no!

    26. RY

      ... have to fly here and all that, you know?

    27. JR

      I don't, I don't wanna end. I just have so many questions.

    28. RY

      (laughs)

    29. JR

      But it's just, the problem was we got off to it, we just cut to the chase right away, and the, the chase seems to be, uh, uh, something that must be confronted, because it is, uh, it's right there. That's it. That's the whole thing. And I, I've tried so hard to listen to these people that don't think that it's a problem and listen to these people that think that it's gonna be a net positive for humanity and, "Oh, good, it's good. I feel better now." But it, it doesn't work, doesn't resonate.

    30. RY

      I wish they were right. Every time I have a debate with someone like that, I'm like, "Please come up with better arguments. Prove me wrong."

  9. 23:14 – 27:48

    Meaning collapse, existential risk, and ‘s-risks’: the possibility of extreme suffering

    1. RY

      So there are multiple levels of risk. Immediate is what we call ikigai risk, high risk. Uh, we lose meaning. You lost your job. You're no longer the best interviewer in the world. Like, what's left? What are you gonna do? Maybe some people s- will find some other, uh, kind of artificial things to do, but for most people, their job is their definition, who they are, what makes a difference to them, for quite a few people, especially in professional circles. So losing that meaning will have terrible impact on society. We always talk about unconditional basic income. We never talk about unconditional basic meaning. W- what do you-

    2. JR

      Right.

    3. RY

      ... want doing with your life if basic needs are provided for you? Next level is existential risk. The concern is it will kill everyone. But there is also suffering risks. For whatever reason, it's not even killing us. It's keeping us around forever, and we would rather be dead, it's so bad.

    4. JR

      Uh, what do, what do you see when you think of that?

    5. RY

      It's hard to be specific about what it can do and what specific ways of torture it can come up with and why. Again, if we're looking at worst case scenarios, I, um, I found this, uh, set of papers about what happens when young children have epileptic seizures, really bad ones. And what sometimes helps is to remove half of your brain-

    6. JR

      Oh, Jesus.

    7. RY

      Just cut it out. And there are two types of surgeries for doing that. One is to remove it completely, and one is to kind of dissect connections leading to that half and leave it inside. So it's like solitary confinement with zero input/output forever. And there are equivalents for digital forms and things like that.

    8. JR

      And you worry that AI would do that to the human race?

    9. RY

      It is a possibility.

    10. JR

      Essentially neuter us.

    11. RY

      Well, loss of control is a part of it. But you can lose control and be quite happy. You can be like an animal in a very cool zoo, enjoying yourself, engaging in-

    12. JR

      Right.

    13. RY

      ... hedonistic pleasures, sex, food, whatever. You're not in control, but you're safe, so those are separate problems. And then there is, for whatever reason, I don't know if it's malevolent payload from some psychopaths, again, that would assume that they could control AI. I don't think they will, but if they manage to do it, they can really put any type of payload into it. So think about all the doomsday calls, psychopaths, anyone providing their set of goals into the system.

    14. JR

      But aren't those human characteristics? I mean, those are characteristics that I think ... if I had to guess, those, those exist because, in the future, there was some sort of a natural selection benefit to being a psychopath in the days of tribal warfare, that you ... if you were the type of person that could sneak into a tribe in the middle of the night and slaughter innocent women and children, your genes would pass on, th- there was a benefit to that.

    15. RY

      Right, so if-

    16. JR

      Is-

    17. RY

      ... it's a human providing payload, that's what would show up. If it's AI on its own deciding what's going to happen, I cannot predict. I'm just looking at worst case scenarios. There are also game theoretic reasons, where people talk about retrocausality, where if right now you're not-

    18. JR

      What is that word?

    19. RY

      Uh, like trying to influence the past, uh, different-

    20. JR

      Say it again? Say that again?

    21. RY

      Retrocausality.

    22. JR

      Retro causology?

    23. RY

      C- causality, causes.

    24. JR

      Oh, okay.

    25. RY

      So think about like weird time travel effects. Right now, if you're not helping to create super intelligence, once it comes into existence, it will punish you really hard for it.

    26. JR

      Okay.

    27. RY

      And the punishment needs to be so bad that you start to help just to avoid that.

    28. JR

      (laughs) Uh, my thought about it was that it would just completely render us benign, that it wouldn't be fearful of us if we had no control, that it would just sort of let us exist, and it would be the dominant force on the planet, and then it would stop. It w- d- if, if, if human beings have no control over, you know, all of the, the different things that we have control over now, like international politics, um, control over communication, if, if we have none of that anymore and we're reduced to a subsistence lifestyle, then we would be no threat.

  10. 27:48 – 31:50

    Energy, indifference, and value drift: why ‘it won’t care about us’ is plausible

    1. RY

      It is a possibility. I cannot say this will not happen for sure. But look at, uh, our relationship with animals where we don't care about them. So ants, if you decide to build a house and there is an ant colony on that property, you'd genocide them.

    2. JR

      Right.

    3. RY

      You'd take them out.

    4. JR

      Right.

    5. RY

      Not because you hate ants, but because you just need that real estate. And it could be very similar. I, again, I cannot predict what it can do, but if it needs to turn the planet into fuel, raise temperature of a planet, cool it down for servers, whatever it needs to do, it wouldn't be concerned about your well-being.

    6. JR

      It wouldn't be concerned about any life, right? Because it doesn't need biological life in order to function. As long as it has access to power, and a- assuming that it is far more intelligent than us, there's abundant power in the universe. There's abundant power. Just the, the ability to harness solar would be an, an infinite resource and it would be completely, uh, free of being dependent upon any of the things that we utilize.

    7. RY

      And again, we're kind of thinking what we would use for power.

    8. JR

      Right.

    9. RY

      If it's smarter than us, if it does novel research in physics, it can come up with completely novel ways of harnessing energy, getting energy. So I have no idea what side effects that would have for climate.

    10. JR

      Right. Right, why would it care about biological life at all?

    11. RY

      We don't know how to program it to care about us.

    12. JR

      And even if we did, it, if we, if it felt like that was an issue, if it felt like th- that was a conflicting issue, it would just change its programming.

    13. RY

      So usually, when we start training AI, we train it on human data and it becomes really good very quickly, it becomes superhuman. And then the next level is usually zero knowledge, where it goes, "All your human data is biased. Let me figure it out from scratch. I'll do my own experiments. I'll do some self-play. I'll learn how to do it better without you." And we see it with games, we see it in other domains, and I think that's gonna happen with general knowledge as well. It's gonna go, everything you have on the internet, Wikipedia, it's biased. Let me do first principles research, rediscover from physics, and go from there. So whatever bias we manage to program into it, I think will be eventually removed.

    14. JR

      That, this is what's so disturbing about this. It's like we do not have the capacity to understand what kind of level of intelligence it will achieve in our lifetime. We don't have the capacity to understand like what it was... what it will be able to do within 20, 30 years.

    15. RY

      We can't predict next year or two precisely.

    16. JR

      Next year or two?

    17. RY

      We can understand general trends. So it's getting better.

    18. JR

      Right.

    19. RY

      It's getting more, generally more capable, but no one knows specifics. I cannot tell you what GPT-6 precisely would be capable of, and no one can, not even people creating it.

    20. JR

      Well, you talked about this on Lex's podcast, too, like the ability to have safety. You're like, "Sure, maybe GPT-5, maybe GPT-6," but when you scale out 100 years from now ...... ultimately, it's impossible.

    21. RY

      It's a hyper-exponential progress and, uh, process and we cannot keep up. I- it, uh, basically requires just to add more resources, give it more data, more compute, and it keeps scaling up. There is no similar scaling laws for safety. If you give someone billion dollars, they cannot produce billion dollars worth of safety. It, if at all, scales linearly and maybe it's a constant.

    22. JR

      (sighs) Yeah, and it doesn't scale line- linearly.

    23. RY

      Uh-

    24. JR

      It, it, it's exponential, right?

    25. RY

      The, the AI development is hyper-exponential-

    26. JR

      Hyper-exponential.

    27. RY

      ... because we have hardware growing exponentially. We have data creation processes certainly exponential. We have so many more sensors. We have cars with cameras. We have all those things. That's exponential. And then, uh, algorithm, algorithmic pros- uh, progress itself is also exponential.

  11. 31:50 – 36:39

    Quantum computing reality check, multiverse hype, and ‘crazy papers’

    1. JR

      And then you have quantum computing.

    2. RY

      So that's the next step. It's not even obvious that we'll need that. But if we ever get stuck, yeah, we'll, we'll get there. I'm not too concerned yet. I don't think there are actually good quantum computers out there yet. But I, I think, uh, if we get stuck for 10 years, let's say, that's the next paradigm.

    3. JR

      So what do you mean by you don't think there's good quantum computing out there?

    4. RY

      So we constantly see articles coming out saying, "We have a new quantum computer. It has that many qubits."

    5. JR

      Right.

    6. RY

      But that doesn't mean much because they use different architectures, different ways of measuring quality. To me, show me what you can do. So there is a threat from quantum computers in terms of breaking cryptography, factoring large integers. And if they were actually making progress, we would see with every article, now we can factor 256-bit number, 1,024-bit number. In reality, I think the largest number we can factor is, like, 15. Literally, not 15 to a power, like just 15. There is no progress in applying it to source algorithm last time I checked.

    7. JR

      But when ... Uh, I've read all these articles about quantum comput- computing and its ability to solve equations that would take conventional computing an infinite number of years.

    8. RY

      Yeah.

    9. JR

      And it can do it in minutes.

    10. RY

      Those equations are about quantum states of a system. It's kinda like what is it for you to taste ice cream? You compute it so fast and so well, and I can't, but it's a useless thing to compute. It doesn't compute solutions to real world problems we care about in conventional computers.

    11. JR

      Right. I see what you're saying. So it's essentially set up to do it quickly.

    12. RY

      It's natural for it to accurately predict its own states, quantum states, and tells you what they are. And classic computer would fail miserably. Yes, it would take billions and billions of years to compute that specific answer. But those are very restricted problems. It's un- it's not a general computer yet.

    13. JR

      When you, when you see these articles, when they're, they're talking about quantum computing and some of the researchers are equating it to the multiverse, they're saying that the ability that these quantum computers have to solve these problems very quickly seems to indicate that it is in contact with other realities. You- I'm sure you've seen this, right?

    14. RY

      There is a lot of crazy papers out there.

    15. JR

      (sighs) Do you think that's all horseshit?

    16. RY

      Can be tested, can be verified. I think most multiverse theories cannot be verified experimentally. They make a lot of sense. The idea about personal universes I told you about-

    17. JR

      Right.

    18. RY

      ... is basically a multiverse solution to value alignment. So it would make sense for previous civilizations to set it up exactly that way. You have local simulations. Maybe they testing to see if we're dumb enough to create super intelligence. Whatever it is, it makes sense as a theory, but I cannot experimentally prove it to you.

    19. JR

      Right. Yeah. This, the problem with, uh, subjects like that, and particularly, uh, articles that are written about things like this, is that it's designed to lure people like me in. (laughs) Where you read it and you go, "Wow, this is crazy. It's evidence of the multiverse, but I don't really understand what that means."

    20. RY

      Yeah, so you probably get a lot of emails from crazy people.

    21. JR

      Oh, yeah.

    22. RY

      And usually, they are topic-specific. So I do research on super intelligence, consciousness, and simulation theory. I get the perfect trifecta of all the crazy people-

    23. JR

      (laughs)

    24. RY

      ... contacting me with their needs. Uh, yeah, those topics are super fascinating. I think at certain level of intelligence, you are kinda nerd sniped towards them.

    25. JR

      Mm-hmm.

    26. RY

      But, uh, we have hard time with hard evidence for that.

    27. JR

      Right. But are we even capable of grasping these concepts? That's the thing. With, with the limited ability that the human brain has, whatever we ... You, you know, we're, we're basing it on the knowledge that's currently available in the 21st century that human beings have acquired. I mean, are we even capable of grasping a concept like the multiverse? Or is it just, do we just pay it lip service? Do we just discuss it? Is it just this, like, fun mental masturbation exercise?

    28. RY

      It depends on what variant of it you look at. So if you're just saying we have multiple virtual realities, like kids playing virtual games and each one has their own local version of it, that makes sense. We understand virtual reality.

    29. JR

      Right.

    30. RY

      We can create it. If you look at AIs, when GPT is created, it's providing an instance to each one of us. We're not sharing one. So it has its own local universe with you as the main user of that universe. There is analogy to multiverse in that. So we understand certain aspects of it. But, uh, I think it is famously said, no one understands quantum physics. And if you think you do, then you don't understand quantum physics.

  12. 36:39 – 1:01:12

    Simulation theory: statistical arguments, VR trends, and AI boxing as a probe

    1. JR

      Yeah. Um, y- the simulation theory, I'm glad you brought that up because you're also o- one of the people that believes in it.

    2. RY

      I do.

    3. JR

      You do. How do you define it? And what, what do you think it is? What do you think is going on?

    4. RY

      So I'm trying to see technology we have today and project the trends forward. I did it with AI, let's do it with virtual reality. We are at the point where we can create very believable, realistic virtual environments. Maybe the haptics are still not there, but in many ways, visually, sound-wise, it's getting there. Eventually, I think most people agree, we'll have same resolution as our physics. We're also getting close to creating intelligent agents. Some people argue they are conscious already or will be conscious. If you just take those two technologies and you project it forward and you think they will be affordable one day, a normal person like me or you can run thousands, billions of simulations. Then those intelligent agents, possibly conscious ones, will most likely be in one of those virtual worlds, not in the real world. In fact, I can, again, retrocausally place you in one. I can commit right now to run billion simulations of this exact interview.

    5. JR

      Mm-hmm.

    6. RY

      So the chances are, you're probably in one of those.

    7. JR

      But, uh, d- is that logical? Because i- if this technology exists, and if we're dealing with super intelligence, so if we're dealing with AI, and AI eventually achieves super intelligence, why would it want to create virtual reality for us and our consciousness to exist in? It seems like a tremendous waste of resources just to fascinate and confuse th- these territorial apes with nuclear weapons. Like, why would we do that?

    8. RY

      So a few points. One, we don't know what resources are outside the simulation. This could be like a cell phone level of compute. It's not a big deal for them outside of our simulation. So we don't know if it's really expensive or trivial for them to run this.

    9. JR

      Right.

    10. RY

      Also, we don't know what they are doing this for. Is it entertainment? Is it scientific experimentation? Is it marketing? Maybe somebody managed to control them and trying to figure out what, uh, Starbucks coffee sells best, and they need to run Earth-sized simulation to see what sells best. Maybe they're trying to figure out how to do AI research safely and make sure nobody creates dangerous superintelligence, so they're running many simulations of the most interesting moment ever. Think about this decade, right? It's not interesting like we invented fire or wheel, kind of big invention, but not a meta-invention. We're about to invent intelligence and virtual worlds, god-like inventions. We're here. There's a good chance that's not just random.

    11. JR

      Right. But isn't it also a good chance that it hasn't been done yet? And isn't it a good chance that what we're seeing now is that the potential for this to exist is inevitable? That there will one day... If, if you can develop a technology, and we most certainly will be able to, if you look at where we're- are- w- where we are right now in 2025, and you scale forward 50, 60 years, there will be one day a, a virtual simulation of this reality that's indistinguishable from reality. So how would we know if we're in it? This is the big question, right? But also, isn't it possible that it has to be invented one day, but hasn't yet?

    12. RY

      It's also possible, but then we find ourselves in this very unique moment where it's not invented yet, but we are about to invent all this technology. I- it is a possibility, absolutely.

    13. JR

      Right.

    14. RY

      But just statistically, I think it's much less, and I'm trying to bring up this thought experiment with creating this moment on purpose in the future to break commitments.

    15. JR

      Mm-hmm.

    16. RY

      Half the people think it's the dumbest argument in the world. Half of them go, "It's brilliant. Obviously, we are in one." So I'll let you decide.

    17. JR

      Yeah. I feel like if virtual reality does exist, there has to be a moment where it doesn't exist and then it's invented. Why wouldn't we assume that we're in that moment? Especially if we look at the scaling forward of technology from MS-DOS to, you know, user interfaces of, like, Apple, and then what we're at now with quantum computing and these, these sort of discussions. I- isn't it more obvious that we, we can trace back the beginning of these things, and we can see that we're in the process of this, that we're not in a simulation. We're in the process of eventually creating one.

    18. RY

      So you zoomed out 30 years.

    19. JR

      Yes.

    20. RY

      Zoom out 15 billion years. You have a multiverse where this process took place billions of times. You are simulation within simulation-

    21. JR

      Hmm.

    22. RY

      ... many levels over. And to you, even if this was a simulation of those 30 years, it would exactly, it would look exactly like that. You would see where it started.

    23. JR

      (laughs)

    24. RY

      It wouldn't be magically-

    25. JR

      Right.

    26. RY

      ... showing up out of nowhere.

    27. JR

      Right. So if you're playing the game, in the game you have Newton and Michelangelo and Leonardo da Vinci and-

    28. RY

      Well, at least you have memories of those things.

    29. JR

      Right.

    30. RY

      Even if you started with preloaded memory state.

  13. 1:01:12 – 1:10:55

    Religion as proto-simulation story, and what stays meaningful if reality is simulated

    1. RY

      With simulation, what's interesting, it's not just the last couple years then we got computers. If you look at religions, world religions, and you strip away all the local culture, like take Saturday off, take Sunday off, donate this animal, donate that animal, what they all agree on is that there is super intelligence which created a fake world and this is a test-

    2. JR

      (laughs)

    3. RY

      ... do this or that. They describing, like, if you went to jungle and told primitive tribe about my paper and simulation theory, that's what they would know three generations later. Like, God, religion, that's what they got out of it.

    4. JR

      ... why, eh, eh, but they don't think it's a fake world.

    5. RY

      A made world. A physical world is a subset of a real world which is non-physical, right? That's the standard-

    6. JR

      Right.

    7. RY

      ... Christian, yeah.

    8. JR

      So this physical world being created by God?

    9. RY

      Yeah.

    10. JR

      Right. But what existed before the physical world created by God?

    11. RY

      Ideas. Just information.

    12. JR

      Just God. God was bored and was like, "Let's give some, make some animals that can think and solve problems." And for what reason? I think to create God. This is what I worry about. I worry about, that's really the nature of the universe itself, eh, that it is actually created by human beings creating this infinitely intelligent thing that can essentially harness all of the available energy and power of the universe and create anything it wants. That it is God. That is, the, the, like, you know this whole idea of Jesus coming back? Well, maybe it's real. Maybe ju- we just completely misinterpreted these ancient scrolls and texts, and that what it really means is that we are going to give birth to this.

    13. RY

      So, I-

    14. JR

      And a virgin birth at that.

    15. RY

      There is definitely possibility of a cycle. So we had big bang.

    16. JR

      Yeah.

    17. RY

      It starts this process. We are creating more powerful systems. They need to compute, so they bring together more and more matter in one point. Next, big bang takes place.

    18. JR

      Yeah.

    19. RY

      And it's a cycle of repeated booms and busts.

    20. JR

      Right. Right. Right. And, and there are legitimate scientists that believe that. Yeah.

    21. RY

      Yeah?

    22. JR

      That this ... (sighs) So what's the value in life today then?

    23. RY

      What do humans value?

    24. JR

      Yeah. If you, if, if this is a simulation and if d- in the middle of this simulation we are about to create super intelligence, why?

    25. RY

      So there are external reasons we don't know for sure, and then there are internal things in the simulation which are still real. Pain and suffering, if simulated, is still real.

    26. JR

      Right.

    27. RY

      You still experience it.

    28. JR

      Of course.

    29. RY

      Hedonic pleasures, friendships-

    30. JR

      Right.

  14. 1:10:55 – 1:34:29

    Social power, IQ variance, fame, and governance: humans as bottlenecks in a noisy system

    1. JR

      Well, he might be an agent of AI. I mean, if, if ... Look, if ... Let's assume that this is a simulation, we're inside of a simulation. Um, are we interacting with other humans in the simulation? And is some ... Are some of the things that are inside the simulation, are they artificially generated? Are there, uh, a- are there people that we think are people that are actually just a part of this program?

    2. RY

      So, it's the NPC versus real player question, really.

    3. JR

      Right.

    4. RY

      And again, we don't know how to test for consciousness. Always assume everyone is conscious and treat them nice.

    5. JR

      Yes. That's the thing. We wanna be compassionate, kind people, but you will meet people in this life, you're like, "This guy is such a fucking idiot. He's ... he can't be real." Or he has to have a very limited role in this bizarre game we're playing. There's people that you're gonna run into that are like that.

    6. RY

      You ever meet someone where they repeat the same story to you every time you meet them?

    7. JR

      Yes.

    8. RY

      They have a script.

    9. JR

      Well, it's also ... Yeah. You know, you d- ... You wanna be very kind here, right? You don't ... But you, you ... We've gotta assume, and I know my own intellectual limitations in comparison to some of the people that I've ha- ... like Roger Penrose or, uh, you know, Elon, or many of the people that I've talked to. I know my mind doesn't work the way their mind works. So, there, there are variabilities that are whether genetic, predetermined, whether it's just the life that they've chosen and the amount of information that they've digested along the way and been able to hold onto. But their brain is different than mine. And then, I've met people where I'm like, "There's nothing there." Like, I can't help this person. I'm ... It's just like I'm talking to a Labrador retriever. You know what I mean? Like, there's certain human beings that you run into in this life and you're like, "Well, could ... Is this because this is the way that things get done?" And the only way things get done is you need a certain amount of manual labor, and not just young people that need a job because they're, you know, m- in between high school and college, and they're trying to do ... And so you need somebody to b- ... who's g- ... who can carry things for you. No. Well, maybe it's y- ... uh, you need roles in society. And occasionally, (smacks lips) you have a Nikola Tesla, you know? Occasionally, you have one of these very brilliant innovators that elevates the entirety of the human race. But for the most part, as this thing is playing out, you're gonna need a bunch of people that are paperwork filers. You're gonna need a bunch of people that are security guards in an office space. You're gonna need a bunch of people that aren't thinking that much. They're just kind of existing, and they can't wait for 5:00 so they can get home and watch Netflix.

    10. RY

      I, I think that's what happens to them. But the reason is the spectrum of IQ. If you have IQ from 50 to 200, that's what you're gonna see. And a great lesson here is project it forward. If you have something with IQ of 10,000, what is that going to invent for us? What is it going to accomplish? Yeah, it always impresses me to see someone with 30 felonies and someone with 30 patents.

    11. JR

      (laughs)

    12. RY

      How did that work, right?

    13. JR

      Right.

    14. RY

      Now, scale it to someone who can invent new physics.

    15. JR

      Right. Right. And ...... you know, the person who has the largest IQ, the largest at least registered IQ in the world is this gentleman who recently posted on Twitter about Jesus, that he believes Jesus is real. Do you know who this is? You know-

    16. RY

      I, I saw the post.

    17. JR

      You see that post?

    18. RY

      I saw the post.

    19. JR

      What did you think about that? It felt like this was ...

    20. RY

      It's just that I think we don't know how to measure IQs outside of standard range.

    21. JR

      Right.

    22. RY

      Anything above 150, they create customized tests which, like, four people in the world can take. We just don't have it. It's a normalized test to average human, average Western, American, whatever.

    23. JR

      Yeah.

    24. RY

      And so, we just don't have the expertise. So someone very super intelligent in test-taking can score really well, but if you look at Mensa as a group, they don't usually have amazing accomplishments. They are very kinda cool people, but they are not Nobel Prize winners, majority of them.

    25. JR

      Exactly. I was gonna bring that up. That's what's fascinating to me is a lot of people that are in Mensa, they wanna tell you how smart they are by being in Mensa, but your life is kind of bullshit. Your life's a mess. Like, if you were really intelligent, you'd have social intelligence as well.

    26. RY

      (laughs)

    27. JR

      You, you know, you'd have the ability to formulate a really cool tribe. You know, there's a lot of intelligence that's not as simple as being able to solve equations and, you know, and answer difficult questions. There's a lot of intelligence in how you navigate life itself and how you treat human beings and, and th- the path that you choose in terms of ... Like we were talking about, uh, delayed gratification and, and, and thing, that there's a certain amount of intelligence in that, a certain amount of intelligence in discipline. There's a certain amount of intelligence in, you know, forcing yourself to get up in the morning and go for a run. There's intelligence in that. It's like the, the, the b- being able to control the mind and this sort of binary approach to intelligence that we have.

    28. RY

      Yeah. And so many people are amazingly brilliant in a narrow domain.

    29. JR

      Yeah.

    30. RY

      They don't scale to others. And we care about general intelligence, so take someone like Warren Buffett. No one's better at making money, but then what to do with that money is a separate problem. And he's, I don't know, 100 and something years old.

Episode duration: 2:14:26

Install uListen for AI-powered chat & search across the full episode — Get Full Transcript

Transcript of episode j2i9D24KQ5k

Get more out of YouTube videos.

High quality summaries for YouTube videos. Accurate transcripts to search & find moments. Powered by ChatGPT & Claude AI.

iOSAndroidClaudeChrome