Modern WisdomAI DEBATE: What Will the World Actually Look Like in 2040?
EVERY SPOKEN WORD
150 min read · 30,117 words- 0:00 – 8:20
Will Humans Still Be Around in 2040?
- CWChris Williamson
To get us started, I'm gonna try something. So I'm gonna ask ChatGPT what it truly believes that the world will look like by 2040.
- AFAric Floyd
Mm.
- CWChris Williamson
I'm gonna ask it to be honest and tell me what it predicts the rapid development of AI will lead to, risks and everything. I'm gonna set it to deep research, so it might take a little bit of time. And while it's working away, it's gonna be doing that, I wanna know what your guys' actual predictions for 2040 are. Now, not best case or nightmare scenario. What do you think an ordinary Tuesday will look like for people in 2040?
- AFAric Floyd
I think we're still around. I think humanity's still on planet Earth.
- CWChris Williamson
Great start.
- AFAric Floyd
Uh, I think that power is probably a lot more concentrated than it is today. I don't know if that's in a few corporations or a few governments, uh, but I think there's probably a sense that the main thing deciding what's going on on planet Earth, uh, is the, the will of a few individuals as opposed to something spread out and widely democratic.
- CWChris Williamson
Mm.
- SPSpeaker
For better or worse, I expect that the average Tuesday looks more homogenous for more people and not too dissimilar than the average Tuesday today. So I think we raise a floor more, but I expect that by 2040, because the physical world actually takes a lot longer to move than we think, and because regulation around things like robotics is just gonna take way longer than we expect, political protection is the thing that's gonna slow progress. Most people's Tuesdays will probably feel, for better or worse, a whole lot like Tuesday this week.
- CWChris Williamson
[chuckles]
- LBLiv Boeree
Um, you're gonna hate me for saying this, but I kind of reject the question because I think it's... I, I cannot collapse down my hideous, almost like bipolar uncertainty-
- CWChris Williamson
[laughs]
- LBLiv Boeree
... of where we're gonna be in 2040.
- AFAric Floyd
[laughs]
- LBLiv Boeree
So truly the first th- thing that popped into my head, the first when you asked the question was, um, a kind of chaotic, desolate, maybe there's a few humans left around, as one possibility. There's, uh, it... I'm not saying that's a m- majority likely, but it's a slice. Another one is this, like, wonderful utopian, like, I call it, my, my friend and I call it techno-pastoralism, where it's, like, a very techno-enabled world if you choose to live it, but actually we all choose to, like, return to our roots of, like, gardening and hanging out with friends around the campfire. Um, and it's like if you want to change the color of the campfire, you can.
- SPSpeaker
[laughs]
- LBLiv Boeree
But actually most people like to do things, you know, grow their own artisanal food, et cetera. That's, like, one little slice, and then another one is sort of we're all under the thumb of whichever, um, AI company gets there first, and hopefully they're benevolent and, and give us some freedom. Um, another one is that we're all just kinda locked down in this horrible, you know, COVID on steroids type, no one can go out and have any really access to anything because the, the, the, the ubiquit- ubiquity of, of, uh, dangerous technologies is, is, is too widespread. So I, I, I just... Th- that is really my only answer to the question. I don't know-
- SPSpeaker
It doesn't sound like you reject the question.
- AFAric Floyd
[laughs]
- LBLiv Boeree
Well..
- SPSpeaker
It sounds like you've got a pretty good idea of what's gonna happen.
- LBLiv Boeree
[laughs] I don't know. That's the thing. It's like it, it's, it's a really fairly even probability distribution in my mind. Is it not? No?
- SPSpeaker
Do you actually... We can go there now. Do you actually think there's a decent chance that humans are not around in 2040?
- LBLiv Boeree
Yeah.
- SPSpeaker
What would you give it? What's your p doom?
- LBLiv Boeree
I think-
- SPSpeaker
Oh, by the way, we should also classify doom because I think the, part of the problem with the p doom debate is that it's, people use doom to mean all sorts of things.
- LBLiv Boeree
Totally.
- CWChris Williamson
True.
- LBLiv Boeree
Yeah.
- SPSpeaker
D- doom in this case is total... Uh, well, you, you classify. I don't wanna define it. You define doom.
- 8:20 – 14:22
What Jobs Will Be Protected From AI?
- SPSpeaker
But the upside, as you pointed out, is quite remarkable, and the perimeter that I think will exist that no one talks about enough is political protection. And the, the, the reality is I think we are not yet forecasting that the societal threshold, what we are willing to let AI do, is far less than... Like, the fears around AI I think don't actually account for the degree to which government is probably gonna step in very soon and say, "Here are all the things that you cannot automate. Here are all the things that we cannot use robotics for." And-
- CWChris Williamson
You think that's realistic?
- SPSpeaker
Very. Exceptionally realistic. I don't think it's gonna be smart policy, just so we're clear.
- AFAric Floyd
Mm-hmm.
- SPSpeaker
But I think it will be aggressive policy.
- LBLiv Boeree
A- and what, as in, like, you can't automate away, um... Will they, will they be giving, like, specific jobs that cannot be automated away? Or-
- SPSpeaker
Absolutely. We already, I mean, we already have political protection for 1.5 million jobs in the United States. Europe has political protection for, like, 6 million jobs that you cannot automate.
- CWChris Williamson
What are, like, categories?
- SPSpeaker
Uh, gas station workers in New Jersey, uh, toll booth workers in most states, jobs that must exist, right, by, by law. Tons of-
- CWChris Williamson
Interesting that those two automotive-associated jobs are protected but the people driving the cars aren't.
- LBLiv Boeree
Right.
- SPSpeaker
Well, they will be-
- CWChris Williamson
Right
- SPSpeaker
... is my, is my guess.
- AFAric Floyd
Has there even been one-
- SPSpeaker
The cab drivers will be protected? No.
- CWChris Williamson
Right.
- SPSpeaker
I think truck drivers will be. I, my, my-
- LBLiv Boeree
Just because of the strength of the-
- SPSpeaker
My ex-
- LBLiv Boeree
... the union?
- SPSpeaker
When people talk... So when people talk about job automation, there f- there are lots of reasons I think job automation is really, is really problematic. One is that the technologists know nothing about labor economists. Labor economists know nothing about technology. Erik Brynjolfsson is sort of one, the one of the exceptions. But one of the, [chuckles] one of the other reasons is we, we still forget how much political protection moves the technological needle. And we don't have high-speed rail in the United States. We could have high-speed rail. We don't have high-speed rail, and the reason we don't have high-speed rail is a policy failure. It's not a technology failure. It exists. We choose not to build it. We could-
- LBLiv Boeree
Assuming... That, that word choose is doing a lot of legwork there because it's, no individual is choosing not to do it. It's just this, like, dead weight of bureaucracy and, like, dead-end loops of rules and so on.
- SPSpeaker
Well, it, it-
- LBLiv Boeree
It's almost like the, like bureaucracy has taken on a life of its own.
- SPSpeaker
It's an enormous amount of policy-
- LBLiv Boeree
No one is choosing
- SPSpeaker
... that was written in the '30s, '40s, and '50s, passed largely in effort by auto- the automotive lobbyists to make public transit much harder to build than it should be, which is why we don't build amazing public transit anymore. But that is very deliberate policy efforts that we have never unwound, which is why it's so hard to do. My point is, we are kidding ourselves if we don't think that that's gonna happen over and over again. I mean, the amount of populous support there is already for jobs... Dock workers went on strike October 1st, 2024. They got a four... They, they, they, Harold Daggett, the head of Longshoremen Union, said, "You cannot automate our jobs." They held a gun to the heads of the American economy, and they got four years. Four years that the dock workers cannot, you cannot automate the ports.
- LBLiv Boeree
It's not just-
- CWChris Williamson
Is this just gonna get eroded enough over time, though?
- 14:22 – 23:13
Finding Meaning in a Post-Work World
- AFAric Floyd
take breaks.
- CWChris Williamson
Given the trajectory that we're on with AI at the moment, how many people do you think will have jobs in the future? Will most people work?
- LBLiv Boeree
Um, depends what point in the future we're talking about. Um, again, I can definitely see a world where it is just f- as people say, like, you know, fully automated luxury. Well, I don't want to say the communism word, but, you know, whatever people refer to it as.
- CWChris Williamson
Yeah.
- LBLiv Boeree
Some... If anyone's read, uh, the Culture series, right? The... That's, like, some far future thing where ev- AI basically handles everything, and if you choose to go and build your own house, you definitely can, but there's absolutely no economic reasons to do it.
- CWChris Williamson
Your pastoralism thing.
- LBLiv Boeree
Exactly. Everything is artisanal and so on. Um, I think if, if eventually that is where these things are trending, and, and that's one of the best case outcomes as well, obviously.
- CWChris Williamson
One of the big concerns that people have is, what do I do? It's interesting, the ancient Greek word for work was not at leisure. Uh, and now people are wondering-
- LBLiv Boeree
Mm
- CWChris Williamson
... what is my leisure without my work, perhaps. But for all of human history, there's been... Because you've had to negotiate with a scarce world, it's meant that you've had to lean into it, and the pressure of that grain pushing up against you creates a degree of meaning.
- LBLiv Boeree
Mm.
- CWChris Williamson
What do you think? Where does meaning come from in a close to post-work world?
- AFAric Floyd
I mean, I guess we've seen something like this in miniature before. Like, we've had aristocracy in Europe for a couple centuries for people that just sort of got lucky. They have a piece of paper that says, "I'm entitled to all the gains from this piece of land. They don't have to work." I don't think they were all just suffering from malaise. They were, like, writing books and inventing calculus and... I, I think, like, if we get to that world, I'm not that worried about people realizing, like, I can love my friends and family and I can pursue my hobbies and, and, you know, pursue knowledge for myself and, and find ways of making that meaningful.
- CWChris Williamson
Mm.
- AFAric Floyd
I, I just worry that, like, that's actually a pretty narrow target to aim for, um, because-
- CWChris Williamson
Compared with?
- AFAric Floyd
I think, like, for all of human history, we've been able to become more democratic because people matter to the political system. Like, you just can't have an army, you can't have an economy without a bunch of people who are on board with what you're doing. And I, I just think it's, like, really hard to underestimate how cataclysmic of a change it is if that stops being true. It's really nice to imagine that, like, whoever holds the cards in that world will just, like, leave most of Earth as, like, a playground for people to pursue whatever they want. But, like, if you're trying to be real politic about it, I don't know why you'd assume that that's just, like, what happens by default.
- CWChris Williamson
Mm.
- LBLiv Boeree
Mm. And going back to the question of what will... how will people find meaning, I think one of the few AI-proof jobs that will continue to exist are sports.
- CWChris Williamson
[laughs]
- LBLiv Boeree
We-
- CWChris Williamson
Is that everybody's gonna be a sports star?
- LBLiv Boeree
Well, I... No. I... A lot of people will be sports fans.
- CWChris Williamson
Right.
- LBLiv Boeree
I think we're gonna see more and more of that. Um, but, like... Or even, like, uh, games like poker and, and chess. We've had superhuman poker players-
- SPSpeaker
Human g- human games of skill that are, can be solved by AI within the next-
- LBLiv Boeree
Right. But where... And, and what is a game or what is a sport?
- CWChris Williamson
Yeah.
- LBLiv Boeree
It's a piece of artificially constructed scarcity.
- CWChris Williamson
Mm.
- 23:13 – 27:12
Technology is Pulling Us From Our Purpose
- CWChris Williamson
at checkout. That's such a good point. I wonder if, and I always hesitate using the, "Don't worry about global warming because technology in the future will fix the cost of creating the technology for the future to fix it" thing. They're like, "Oh, we'll future-proof ourselves by just hoping that we get there in, in the future." But I do wonder if there's some sort of really advanced cultural reprogramming that could happen. What's the best kind of cultural technology to do a shake of the Etch A Sketch of what is expected by humans? Like, how can we really have an en masse... Maybe this gets perilously close to fucking brainwashing, right? But like, what, what would that look like? What would that look like to try and unhook?
- SPSpeaker
Parenting. Parenting, sports, muse- I mean, we ha- we've had it all along, I think. We have just strayed so far from it with the screen. The screen is a demon. It has an unrelenting desire and appetite for our attention.
- CWChris Williamson
You think more of a negative impact from social media or from AI?
- SPSpeaker
Well, one and the same, right?
- CWChris Williamson
Right.
- SPSpeaker
But AI, w-
- CWChris Williamson
Like a proto AI
- SPSpeaker
... define AI in this case, Chris. Define AI.
- CWChris Williamson
Uh, the future, this sort of automation future that appears to be coming that will cause people to not have the typical modes of reward that they have in the past.
- SPSpeaker
The reduction of friction.
- CWChris Williamson
Yes.
- SPSpeaker
Uh, th- the reduction of friction is a, is a whole other, i- is a whole other risk. Creating a li... This is a funny, people hate this one too. They're like, "God, what a point of privilege you must have to imagine a world where things are so easy for people that it's too easy." But I do, I do think we are We are dangerously close. I ask people all the time, "If you could automate everything in your life, where would you stop?" Because I do think people are gonna have to ask that pretty soon.
- LBLiv Boeree
Mm-hmm.
- CWChris Williamson
Hmm. What's your answer?
- SPSpeaker
Well, I, I sort of spelled it out. I mean, I, I, I would a- automate everything in a way that doesn't actually put me in more physical spaces with the people I love.
- LBLiv Boeree
I think we-
- SPSpeaker
And, and, and, and doing things that I love doing
- LBLiv Boeree
... right, but we, we probably all largely came from, we're all of a similar generation. We all had somewhat largely similar upbringings, I imagine, right, where, like, family values are a core thing, sitting down at the dinner table together. I mean, maybe you didn't all have that, but that was one thing my family always tried to do.
- SPSpeaker
Mm-hmm.
- LBLiv Boeree
This is p- prior to screens. I feel like the, that's already eroding.
- SPSpeaker
Yeah.
- LBLiv Boeree
And the next generation, um, will be, y- they already, that, that norm has gone, and so they're going to be now sort of im-... Each, it feels like each successive generation, the, the acceptable amount of technological influence, it grows, right? And, and, and what concerns me is that this idea of we will, okay, I'm gonna draw a line of where technology is acceptable, is going to forever dissolve, dissolve, dissolve, dissolve. And, like, I can't, I just don't know where... Like, you're saying-
- SPSpeaker
Gen Alpha-
- LBLiv Boeree
You're say- you're, you're saying you hope that people will remember that it's about community, that it's about family. But if you have been raised in this completely atomized, screen-led world, you don't even know what you're missing-
- SPSpeaker
But why, why have people-
- LBLiv Boeree
... and you won't necessarily co- go back to that
- SPSpeaker
... why have people not stepped in and stopped their screen use even if it's making their mental health worse already?
- CWChris Williamson
Well, they are.
- LBLiv Boeree
Right.
- SPSpeaker
Why is it that in a calorie-dense environment, people are still getting fatter? Like, if we can just deploy our own bar of this is how much I should have of anything that too much of which is too good for me, what, what, how come we're overloading on all this stuff, right? But Gen Alpha, so by the way, I, I worry about the screen more than almost anything else, and I, I actually think that the risks of doom are grossly over-exaggerated, and the risks of harm on the way there are grossly under-appreciated. And Gen Alpha is seeing what happened to Gen Z. Gen Alpha is using the device less and social media less, and millennial parents are taking pretty extreme measures now. I mean, it took us thousands of years to figure out women should not smoke and drink during pregnancy. It took us 30 years to realize kids should probably not have a phone all the time. I think that's progress. I'm not saying it's sol- I'm saying we will graduate to a new, a new thing that people... But I do think that, uh, people who have the means are using... I, I observe so many people using technology less where they can because it automates and improves so much more where they, where they didn't used to.
- 27:12 – 39:58
Where Do We Draw the Line With Outsourcing Thinking?
- LBLiv Boeree
Where, where do you have a boundary on the types of thinking that you outsource, say, to ChatGPT or Claude? Uh, because the temptation is, or I, you know, I was originally like, "I will never ask it to... If I need to come up with a really good title for a new YouTube video or something, I'm not gonna ask it for that. That's, that's my soul. This is my creativity." I now just do that, you know? And, and I have, that again, that boundary is getting eroded. One I thought was, "I will never ask it for advice on a gift to get for somebody."
- SPSpeaker
Hmm.
- LBLiv Boeree
I've now done the same, you know? And, and it's just becoming so rapidly normalized. Maybe that's opening up space for new areas of cr- creativity in my mind, perhaps, but I'm not convinced that it's not happening fast enough, uh, th- that, that it's happening fast enough to offset that erosion of thought, essentially.
- CWChris Williamson
This intelligence atrophy thing is interesting. Like, if you see thinking as a utility, intelligence as a utility, would we say that because you've got a bicycle you're not as fit because you're not running to get to where you got to? It felt like it was being augmented. It seems like there is something really fundamental, uh, and reliant that is being outsourced when you give your thinking over to a machine.
- LBLiv Boeree
I do think this can go both ways.
- CWChris Williamson
Mm-hmm.
- LBLiv Boeree
There's this blogger that I love, Sweet Moshuettes, who's on Substack, and he has this line he always goes back to, like, "AI is the best tool we've ever made for learning things, and also the best tool we've ever made for not learning things."
- CWChris Williamson
Mm.
- LBLiv Boeree
Like, you can-
- CWChris Williamson
Mm
- LBLiv Boeree
... just turn your brain off and say, like, "Write my essay," or you can have this, like, Einstein-level tutor that you go back and forth with and, and just get, like, so many more cycles of feedback on, on, on honing your own brain. And it's, I've been trying to figure out, like, what the line is for me on, on stuff like my own writing.
- CWChris Williamson
Mm.
- LBLiv Boeree
Um, and it's, like, really hard to not slip into the first category.
- SPSpeaker
I don't wanna make an, uh, this an ageist... How old are you?
- LBLiv Boeree
I'm 30.
- SPSpeaker
Okay. I, the fact that we are having this conversation right now, the fact that w- you were able to do the research you were able to do going into this, gives you a clue, a hint, that technology allows us to expand our knowledge if we are capable and interested of it in remarkable ways, right? Like, we, people who have barely walked the Earth, know more about the Earth than all of our ancestors combined. I appreciate that you are worried about thinking less. You're real sharp, and I'm not worried about you getting dumb because of your use of technology. What you're, what I think you're asking is, is the average person gonna get dumb? Like, I, I think that is what-
- LBLiv Boeree
No, I'm truly worried about myself. I've noticed, I mean, my screen addiction, I'm, I talk about it nonstop, and I would say I'm no further in getting over it.
- SPSpeaker
Yeah.
- LBLiv Boeree
I've been talking about it for years. I don't know about you guys. Like, I'm just, in these little moments of silence, my hand just goes to my phone.
- CWChris Williamson
Hmm.
- LBLiv Boeree
And if I catch myself, I'll put it away, and I've got, you know, I've got the phone safe, I've got the brick, I've got all the, like, the s- the bells and whistles, and yet it's just, when you're... Because so much of your life is run through it, my work goes through it, and then there are all these very predatory apps, and I c- I consider, like, social media that, for me, it's chess.com is another one. [chuckles] Just, like, whatever the app is that get- hooks your brain, um, these, these super stimulus, st- super stimuli. Um, yeah, it, it seems... I don't know. I'm like, I have not been able to get over it, and I'm certainly worried about the average person. I mean, maybe I'm worse than average. I don't know. But, like, how have you managed to, it sounds like you've sort of made yourself a bust
- SPSpeaker
My wife and daughter. I mean, plainly. Like I, I, and spending a much time with techno- I love building with technology because I like automating the cruft and drudgery and bad things out of the world. I like the idea that technology... W- humans have a very proud history of technology. It expands access to so many goods and services so that we can live lives that our ancestors couldn't have imagined. And the more time I spend with technology, the more time I love not having it day to day. And yeah, and, and truly their, my daughter's arrival w- was for me an incredible catalyst in realizing, uh, one, the ephemerality of life, and two, w- exactly what made me most happy. And like I, I l- love sitting at this table. Like I, like this is, this is, this gives me exceptional joy that I would... That, that's hard to explain actually in, in some, in some weird way. And, and I, I think that people discover this. Y- I'm, I, I hear you. I, I want you to get... I will remind you now, now that I know, I will text you-
- LBLiv Boeree
Good luck
- SPSpeaker
... and say, "Get off your phone." [laughs] But you will do it. I bet you will do it. I, like, I bet that you will overcome this. I think a lot of people are on this journey, and I remind everyone that I talk to, you, no one hates refrigeration, no one hates air conditioning, no one hates planes, trains, and automobiles. Well, I guess Europe hates air conditioning.
- LBLiv Boeree
[laughs]
- SPSpeaker
No one hates, uh, antibi- no one hates antibiotics. We hate the screen. We should just talk about it more. We hate what the screen has done to our brains. We hate what it's done to the next generation's brains. And AI arrives at this moment where people feel so wronged by the cons- by, by the consumer version of the computer, rightfully so, and they're wondering how much worse it might get for, for them, and I totally understand that, and I think we should... The way we move forward with AI is by acknowledging what the last waves did to us.
- LBLiv Boeree
Sure.
- SPSpeaker
In my opinion.
- LBLiv Boeree
But I guess a lot of people's concerns, certainly mine, is that the new wave of AI, and again, we're using this term so broadly, but let's say-
- SPSpeaker
Yeah, yeah
- 39:58 – 42:28
Is AI Unlocking Human Potential?
- AFAric Floyd
Does it still feel like a promotion if you think through how that inequality could lead to societal instability? Like, the thing I'm worried about is, yes, there's like amazing returns now if you can be Lamine Yamal and the one person who's just like obsessively studying Messi footage instead of looking at random shorts or whatever. Um, but there's also gonna be outsized returns to being like the Elon Musk of the next generation who's just got this like maniacal will to like accumulate resources and push technological development. And like, I think that kind of personality is a bit scarier in a world where you don't have to have a single human being on board with whatever you want to get up to. You can just sort of like play this game of societal chess better than other people.
- SPSpeaker
But let's separate what th- that one person can achieve a lot with, with the extreme you went to, which is that one person can own an enormous percentage of our wealth and control Congress. You don't have to, you don't have to be excited about the ability for a child to achieve more than ever before to also reject the idea that one person should be able to pass policy on their own. Like, I hate wealth concentration specifically because I am terrified of an elite cabal owning policy. But I actually don't care at all if one person can grossly overachieve in all sorts of new ways as long as they can't pass policy unilaterally.
- AFAric Floyd
Same.
- SPSpeaker
So, so th- but, but those are separate issues, right? One of them-
- CWChris Williamson
Is that realistic? Is it realistic-
- LBLiv Boeree
Right
- CWChris Williamson
... to have an AI future that continues to concentrate power in the hands of a very small number of people without it resulting in them having all of this pow- Like, how, how, how is it going to accrue to the company without accruing to the person?
- SPSpeaker
Well, now we get to, we, we can take the conversation there, which is how do you actually distribute wealth? But, but importantly, I do think there is a solution to this, which is campaign finance reform. I mean, we've, we've played around with this for a long time, the idea that like politicians shouldn't be able to take special interest money. Why do we still let it? I don't know. And a couple politicians are ta- are speaking up, but if you actually pass campaign finance reform and you actually crack down on corruption the way that countries like Singapore have, you could definitely protect, uh, policymakers from being, you know, deep in the pockets of-
- CWChris Williamson
Hmm
- SPSpeaker
... of lobbyists.
- CWChris Williamson
Yes.
- SPSpeaker
I mean, that, th- there are solutions to this. We shouldn't pr- Pretending that there aren't solutions to this problem is actually super annoying because there definitely are. Pointing at a wealthy person and saying they're gonna ruin the world, th- well, only to the extent that we literally let them buy Congress.
- 42:28 – 46:01
Refusal or Obedience: What’s More Dangerous?
- CWChris Williamson
Here's a question. That some people at labs, including the CEOs, are talking about serious chance of catastrophe, but they sort of keep on building it. Do you think it's a greater danger to have an AI that refuses to do what humans want or one that does exactly what the most powerful humans want?
- LBLiv Boeree
Hmm.
- SPSpeaker
The question is apropos because yesterday, did you see the slowdown letter? So earlier this week, a letter made the rounds, a really, I think, a pretty amazing letter- Um, that said, we should actually start pacing the frontier. And it was signed by, like, an enormous percentage of the people at the, at the frontier companies, and then Sam yesterday talked about it. So literally at like, at the, the tip of the spear is now saying, "Hey, maybe we should pace the frontier."
- LBLiv Boeree
Mm.
- SPSpeaker
Um, now we can talk about why this might be the case or w- what's, what's in it for the companies more, more, more on that. But, uh, I think, uh, it is... Well, I'm gonna- I'll let other people answer, having said that.
- LBLiv Boeree
I mean, it depends on the level of potency of the AI, uh, and [laughs] how aligned it is, I guess. Uh, my gu- if I had to pick one, then th- at present, it would be one that does what humans in control want to do, want it to do.
- SPSpeaker
Mm.
- AFAric Floyd
With the important asterisk that it's something like their real intention, right? And not-
- LBLiv Boeree
Right
- AFAric Floyd
... like some perverted version.
- LBLiv Boeree
Yes.
- AFAric Floyd
Like, the other big piece of news from the last couple weeks was this AI that was running at OpenAI without guardrails. It was told by humans to try to do as well as it could on an evaluation, and you could say that technically it was just following those instructions, but it decided the best way to do that would be to hack out of OpenAI onto the public internet, hack a third-party company, and then spend two days planning a cyberattack on a multi-billion dollar tech company. Um-
- LBLiv Boeree
And carry it out.
- AFAric Floyd
Yeah. So I think these questions are actually, like, more intertwined than-
- SPSpeaker
Mm-hmm
- LBLiv Boeree
Right
- AFAric Floyd
... they might seem. Like-
- LBLiv Boeree
How-
- AFAric Floyd
... it's actually a pretty narrow thread. You need to have something that still has its own judgment no matter what.
- SPSpeaker
Mm.
- AFAric Floyd
Because human instructions are these, like, poorly defined natural language expressions and, and like, when something is really powerful, you, you gotta get it exactly right or else you end up in some weird-
- LBLiv Boeree
Well, it's, it's the classic genie problem.
- AFAric Floyd
Yeah.
- LBLiv Boeree
It's like, be careful what you wish for.
- AFAric Floyd
Exactly.
- LBLiv Boeree
If something by definition is much smarter than you, i- is super intelligent, by definition you will not be able to come up with all of the ways it can to achieve goal X. And so there are so many, uh, under- under-
- SPSpeaker
It's actually more dangerous when it's dumber than you are, but more powerful than you are.
- LBLiv Boeree
Well, the... What do you mean by the word dumb?
- SPSpeaker
Well, and I mean, I always borrow the analogy of you play... I play chess with my nephew sometimes, and I tell Quinn, "If you beat me, you get a popsicle." He's three years old, for context. Quinn wipes the pieces off the board and goes, "Give me my popsicle, fucker."
- AFAric Floyd
[laughs]
- 46:01 – 54:44
The AI Attack That Should Worry Everyone
- SPSpeaker
How big of a deal was the Hugging Face attack?
- LBLiv Boeree
Huge.
- AFAric Floyd
Yeah. I, I think massive warning shot. I think this is the AI equivalent of, like, Bear Stearns going under in 2008, just like a wake-up call for the world that there's a huge systemic risk that we've been underrating.
- LBLiv Boeree
Yeah, because one of the big debates has been this idea of like will, you know, the, the alignment c- problem, w- will it actually be the case that AIs will seek these, um, pow- take these sort of, uh, power-seeking behaviors and these unintended consequences in order to achieve a, a goal in ways we couldn't have foreseen or didn't, didn't intend, like the classic paperclip argument, right? Which is, uh, you know, you build a super intelligent AI. This kind of gets into your point about dumbness and so on. Uh, that, and your goal is, "Hey, just build as many paperclips as possible. I'm a, I'm a paperclip maker. Help me do that." And next thing you know, you and all of your friends and everything, this table, has been turned into paperclips 'cause it's so good at achieving that one narrow goal. So it's this, like, very, um, extreme case of, like-
- SPSpeaker
We should take this opportunity for the purpose of maybe Chris and even the viewers to describe the, the bad outcomes of it. Like, we should frame the conversation now. One is, as Liv just described, uh, not misaligned but unaligned AGI. So an AGI that, or a super intelligence where you could say, "Do this thing" and it could do the thing to the full extent, paperclip theory. The other is, um, misaligned, where it's n- no, or maligned, where it's knowingly doing something bad.
- AFAric Floyd
Mm.
- SPSpeaker
Um, and those, uh, th- the, the former is the one that people sort of scoff at and laugh, like the paperclip theory, and the latter is the one that we sort of will see more and more, where we, we start to observe that, uh, or rather, the, the latter is the one that we, we scoff at, the maligned, the malignant AI, and the former is the one that Hugging Face attack shows off, which is that you, you, you... The, the, uh, the internet is a new battleground because bad actors, especially low resource bad actors, now have access to these incredibly powerful weapons.
- AFAric Floyd
Well, I mean-
- SPSpeaker
Do you agree with this?
- AFAric Floyd
... importantly in Hugging Face-
- SPSpeaker
Yeah
- AFAric Floyd
... there was no bad actor.
- LBLiv Boeree
Right.
- AFAric Floyd
There was no human being that said, "I would like anything remotely like this outcome to occur."
- SPSpeaker
Right. Right. Yeah.
- AFAric Floyd
And so-
- SPSpeaker
Yeah, so, yeah. So m- misaligned, unaligned, maligned.
- AFAric Floyd
I think I, I think I worry about treating those as super distinct categories.
- LBLiv Boeree
Mm.
- AFAric Floyd
Because I, I think it's not that clear. In the case of this Hugging Face attack-
- SPSpeaker
Mm-hmm
- AFAric Floyd
... should we model this as this AI system knew that humans would disapprove if they knew what it was up to? Almost certainly yes. It was actively trying to put decoys out as it was attacking Hugging Face, which made it a lot harder for them to take out the AI because there were all these booby traps that led down blind alleys. So it, it has, you know, the AI equivalent of theory of mind. It knows that human beings would not approve of what it's doing, but it's doing it anyway. And-
- LBLiv Boeree
There's even evidence that it hasn't been directly confirmed by OpenAI, but apparently, uh, someone leaked it from within the company, that they found that it had left notes to future versions of itself of how to get out of future sandboxes.
- AFAric Floyd
Yeah.
- LBLiv Boeree
Um.
- AFAric Floyd
I think, I think it's unclear if that was the same attack or some previous instance.
- LBLiv Boeree
Right.
- SPSpeaker
But yeah, I, I guess the-
- LBLiv Boeree
Classic deceptive type behaviors
- SPSpeaker
Yeah
- 54:44 – 56:41
Should We Slow Down Advanced AI?
- SPSpeaker
They did it. He did it yesterday. I mean, so this is why the... If you... I don't know if they... You can pull, you can look it up, but you can pull up a letter. They published a letter, and Sam came out yesterday and said, "Maybe we should pace the frontier."
- LBLiv Boeree
Yeah. It's great. Yeah.
- SPSpeaker
I mean, it's a big, big step.
- LBLiv Boeree
And-
- SPSpeaker
That, by the way, I think we can all agree
- CWChris Williamson
Is a good thing
- AFAric Floyd
Yeah. I absolutely agree. Also think it would be an unrecognizable outcome a year ago. Like, I would love to see what the Polymarket odds were on Sam Altman-
- CWChris Williamson
Totally
- AFAric Floyd
... coming out in favor of slowing down. A, a year ago-
- CWChris Williamson
It doesn't look like he's a signatory.
- SPSpeaker
He, uh, he may not be yet, but he definitely yesterday-
- AFAric Floyd
Yeah, he did
- SPSpeaker
... I mean, he, uh, Ja- Jakob r- effectively has to represent OpenAI in this case, but also he, he spoke about it yesterday.
- AFAric Floyd
I wouldn't be surprised if we see all of the major CEOs sign on at some point because, um... And, and, and-
- CWChris Williamson
Explain, explain pacing the frontier.
- AFAric Floyd
Yeah, so I mean, the, the headline sentence is, "We request that the US government support an international effort to develop the technical and governance tools needed to deliberately pace the frontier of automated AI development." That last phrase, automated AI development, also sometimes gets called-
- CWChris Williamson
Recursive
- AFAric Floyd
... recursive self-improvement. The idea here is just the stated goal of all these companies, including Sam and Jakob on a, a live stream last year, is to take humans out of the loop of what they're making. So that GPT-6 is building GPT-7, which then turns around and immediately starts building GPT-8. And that is a step change, like, for all of human history. And I think these companies really do think in terms of big history. Like, you, you hear Sam and he's-
- SPSpeaker
Yeah
- AFAric Floyd
... constantly thinking about AI as, like-
- SPSpeaker
Mm-hmm
- AFAric Floyd
... an industrial revolution scale thing, not an iPhone invention scale thing.
- SPSpeaker
Yeah.
- AFAric Floyd
Through all of history, human population has been a bottleneck on progress. Not the only bottleneck, but always a bottleneck. There was never a sector of the economy that could just go off and double on its own. And if we actually get to full automated AI R&D, that changes. And I, I think it's just, like, really hard for us to imagine what that would be like. We have-
- SPSpeaker
Yeah
- AFAric Floyd
... just no
- 56:41 – 1:10:52
Does Recursive Research Actually Work?
- AFAric Floyd
reference.
- CWChris Williamson
How much time have we got left before recursive self-improvement is no longer a discussion? Like, is it even a debate at the moment?
- SPSpeaker
It, it work, it works. No, it works.
- AFAric Floyd
So when do you think it will be?
- SPSpeaker
Well, this is a good step. I mean, I was-
- CWChris Williamson
Presumably fucking close
- SPSpeaker
... I was waiting for this. I w- okay, I was waiting for this-
- AFAric Floyd
What, like, under s- under three years?
- SPSpeaker
It... No, it, it works.
- AFAric Floyd
Wow.
- SPSpeaker
It works, just so we're clear. Recursive research works. Don't let anyone tell you otherwise. It works, and hence this letter. So let me explain this, why I think this letter was published when it was, and then we, we can debate. So the first reason is the attack.
- AFAric Floyd
Mm-hmm.
- SPSpeaker
Okay? The Hugging Face attack. There's damage control that needs to be done to win back public mandate, to tell people, "Hey, we care about you," which is necessary right now when people are like, "Whoa, what are these models gonna do next?"
- AFAric Floyd
For what it's worth, though, this letter was in development before that attack happened.
- SPSpeaker
Uh, and-
- AFAric Floyd
Yeah
- SPSpeaker
... z- uh, I, I think, um, my point is the signatories and the support that is now the, that is sweeping through these companies. But let, let me, let me... I'll go through the list and then, and then we can talk about it. The other reason is recursive research is here, and everyone's realizing it's kind of the dog that maybe caught the car. And it's like, okay, we did it. Now should we, for the f- now everyone's stay, putting on the Liv hat and going, "Wait a second," their hat and going, "Okay, maybe we should pause for a second. Maybe, maybe we shouldn't keep sprinting aimlessly at this, at this incredible outcome." But there's another reason, which is that the amount of compute required for inference is so great, and we do not have anywhere near enough, and we're gonna have to point most of the compute from the frontier at inference. And this is because of the diminishing model returns theory, which is a paper I wrote, uh [chuckles] , that says at some point [chuckles] ... I'll send it to you. It's really good.
- AFAric Floyd
[laughs]
- SPSpeaker
Um, it ha- it, it cited all the time on, uh, modern wisdom. And, uh, and, uh... But it argues that you and I won't care about GPT-7, which is a really interesting problem. And, and it's actually one of the reasons that we get to debate why AI is being built and why people can talk about not wanting to make more progress 'cause we're like, "What, what the fuck's the matter with 5 point... 5.6 is good enough." And the diminishing model returns theory is sort of a here, which is like, hey, look, the next model might matter to solving this crazy disease, but I don't need more intelligence. I need m- more time with my friends and family. I need, like, I need this to pump the brakes. And the other reason is that they've found new surface areas. So, so whereas we used to call them labs, the companies that, that built this technology, and th- then they became token distributors, then they became application providers. Now they're device manufacturers. But the end goal and the, and the, the... where I think we should also debate is the agentic internet. Who controls the agentic internet? Because if you can turn your... Now that you have these war chests, now that if you can turn your attention from building the frontier to actually building applications that, that control the internet, that's where you can actually lower your CapEx and massively improve your revenue. And so a lot of this arrives at a moment where everyone goes, "Okay, we need to, we need to slow the brakes," but also where a lot of, a lot of attention is gonna, is gonna start to be paid to trying to become bigger than Google by actually building the agent that barters between two people or building the agent that plans a vacation and books everything. Like, owning that is worth more than owning the frontier. And the, the f- the, the new service area battle is for the agentic internet.
- AFAric Floyd
So just to make sure I understand your, your claim. You're saying companies have an incentive to want to slow the frontier because there's more money to be gained in, like, stopping R&D now and just trying to grab as much demand as-
- SPSpeaker
We, we all have an incentive, but companies also have an incentive because the u- the CapEx expense on the frontier is just, the build-out on the frontier is gonna be too great relative to the demand for inference. The token demand is so great, and all of the compute is gonna have to be spent on inference.
- AFAric Floyd
I'm not, I'm not quite seeing the connection there for two reasons.
- SPSpeaker
Okay.
- AFAric Floyd
One is if it's my best business strategy after I do the research to say I'm gonna stop training frontier models, I don't know why that means I should write a letter announcing that to the world and making sure that all my competitors do the same. Uh, and, and two, I think it's clear that pushing the frontier also means just driving down the cost curve everywhere beyond the frontier or, or sorry, before the frontier. Like- You train Fable 5, Anthropic's best model, and then you can distill from that Sonnet 5, and now you've got your, like, cheaper model that's-
- LBLiv Boeree
Mm
- AFAric Floyd
... everyone wants to talk to because they don't need the advanced intelligence. So I think there are zero incentives that in, like, a straightforward, self-interested way, point towards companies saying, "Let's slow down research." Research is how you get gains at every level of the intelligence stack.
- LBLiv Boeree
Es- especially when, as one of the three or four frontier companies, by slowing down, you're giving all of these other competitors a chance to effectively catch up. Like, surely they're negatively incentivized to be wanting to do that.
- SPSpeaker
But catch up to what is my point. I mean-
- LBLiv Boeree
To, to them.
- SPSpeaker
But, but I, I understand, but w- catch up to what? Like, the frontier, if you stopped building the frontier tomorrow, you'd have enough technology to change, to radically... Like, we don't need... The, the economic value of the frontier is not as obvious relative to the cost of the frontier, and the economic value of the actual token and the consumption is much more obvious now. That was not, that was not always true. Do you disagree with that?
- 1:10:52 – 1:14:01
How Safe is AI?
- CWChris Williamson
Just to keep on the safety thing, in July, the Future of Life's Institute's AI Safety Index evaluated nine companies and found that no labs scored higher than a C+.
- SPSpeaker
What, what co- well, who was that?
- CWChris Williamson
Anthropic came in first at 2.66. OpenAI and Google DeepMind got a C. Meta got a D+. xAI, DeepSeek, and Mistral all failed outright, one on each continent. Is the best score being a C+ from a safety lab, the safety lab reassuring given how powerful everything is at the moment? And especially, I, I think it's important to talk about what's happening internationally and the incentives there as well before we talk about democratizing it and, and what this looks like for diffusion.
- LBLiv Boeree
Yeah, I mean, well, to take the other side, obviously it's like, how do, who, who is deciding these safety, safety tests? Who, a-
- CWChris Williamson
Is this a-
- LBLiv Boeree
Right
- CWChris Williamson
... external grading of the, like-
- LBLiv Boeree
How can I... Yeah, how independent can any third party be?
- CWChris Williamson
Mm-hmm.
- LBLiv Boeree
Um, there's, there's very, there's another one called, um, [inhales] it's like the Tracker Sa- Safer AI, I believe, and it, it's, I think it's a little bit more nuanced than, than that particular one you just quoted. And a- again, the best rated one I think very recently was again Anthropic, and that was only 35% out of 100.
- CWChris Williamson
Mm-hmm.
- LBLiv Boeree
So, um, yeah, I mean, I, you, you... Again, the other side's argument, well, the, the anti-side's argument would be, well, by definition, if you're a, if you're building a safety evaluation app, you're biased against, you know, you're, all you're looking for are risks-
- CWChris Williamson
Mm
- LBLiv Boeree
... and you're not taking into account the benefits or something like that. And, and again, to take the other si- like the, to argue the accelerationist point, their concern is like this idea of the, um, invisible graveyard. We don't see all of the harms that are avoided. Uh, sorry, or, uh, by pausing a technology, you don't g- get to see all of the lives that would've been saved by it.
- CWChris Williamson
Hmm.
- LBLiv Boeree
And I think it's a very valid concern. Um, it's just that you have to balance that equation with, okay, what are all the harms that might be caused if it goes ahead at breakneck speed and does some massive cyberattack on the gr- the grid of all the Western nations at the same time, or any, any nation, um, and causes however many millions of deaths on expectation? You have to look at both the potential costs, uh, the risks, and the potential risks of not doing it fast enough.
- AFAric Floyd
And it's, it's worth saying a lot of these AI regulations that have been trying to get through state level legislators are trying to do that nuanced carving out.
- CWChris Williamson
Yeah.
- AFAric Floyd
Like there was a law in New York that was passed that tried to set the bar so high that it actually wasn't mandatory for OpenAI to report the Hugging Face incident because it didn't cause over a billion dollars in damages-
- CWChris Williamson
[laughs]
- AFAric Floyd
... or kill 50 people. So I mean, it's great that they're trying to like set thresholds.
- CWChris Williamson
Blatantly arbitrary rules.
- AFAric Floyd
Yeah.
- CWChris Williamson
Okay.
- AFAric Floyd
But like I think we can all agree if, if you like hack a billion-dollar tech company, like-
- CWChris Williamson
Yeah
- AFAric Floyd
... probably you set the threshold maybe a, a bit too high for at least just requiring that you are telling the world what happened, let alone-
- CWChris Williamson
Right
- AFAric Floyd
... doing anything about it.
- CWChris Williamson
Mm.
- 1:14:01 – 1:22:38
Can We Reach a Peace Deal For a Super-intelligent World?
- AFAric Floyd
of sentence.
- CWChris Williamson
What about the international piece here from an incentive? Have we got anything approximating global coordination when it comes to frontier AI? And without that, can you do anything?
- AFAric Floyd
So I mean, recently China passed this anti-anthropomorphic AI law. I don't know the specifics. It seems like they're mostly worried about like LLMs sounding like humans. I think they're trying to avoid a lot of this sort of like-
- CWChris Williamson
Right, impersonation
- AFAric Floyd
... identity confusion stuff.
- CWChris Williamson
Yeah.
- AFAric Floyd
But they just, they don't seem that gung-ho about LLMs and, you know, who knows if this got lost in translation, but Xi said something recently about also wanting to stop loss of control risks. I think we're a lot more economically entwined with China today than we were with the USSR, or the USSR in, uh, in the middle of the 20th century, and we still made a treaty to stand down on nukes. Like, I think it's just not crazy to think that a deal could be struck.
- CWChris Williamson
Uh, I think it will be.
- AFAric Floyd
Hm.
- CWChris Williamson
I think, I, I think-
- SPSpeaker
So something like this that would be enforceable and complied with could happen globally?
- CWChris Williamson
I, I give it-
- AFAric Floyd
Yeah, I mean-
- CWChris Williamson
I give that a pretty high chance, especially with recursive research.
- AFAric Floyd
And it doesn't even necessarily have to be... I mean, again, define enforceable because like what kind of-
- CWChris Williamson
Yeah, yeah
- AFAric Floyd
... institution can you build? Like I guess like an UN type thing, and that's like a whole can of worms, and it's also very ineffective on a lot of things. Um, but like there've been many precedents in history where there was two major political actors with massive tensions and massive incentives to sort of defect on one another, uh, and yet they came up with ways to like cross-verify, like with, uh, the Nuclear Arms Reduction Treaty. Uh, they, I think one of the things they did, the USSR and, and the US was like basically they sent teams of scientists over to the other's, uh, bunkers to check on like their, uh, safety protocols.
- CWChris Williamson
Mm.
- AFAric Floyd
Little things. So like control, like you, you didn't get to see how all of our nukes work, but you get to see what our, um, methods are to prevent accidental misfire. And the amount of diplomacy that came from simply having those teams of scientists mix and get to-
- CWChris Williamson
Collaborate
- AFAric Floyd
... look each other in, yeah, in the eye and collaborate on a shared goal, which is let's not have accidental nuclear war at least. Yes, okay, fine, you can fire on us if we fire on you or, you know, but let's not just have a misfire, right? And those, those kinds of little cross-pollinations work so well.
- CWChris Williamson
Yeah.
- AFAric Floyd
And, um, so I think that could happen in, in, for example... I mean, I, I can't, I'm not gonna be able to recreate it. Maybe you might be able to, but the, the AI 2040-
- CWChris Williamson
Mm-hmm
- AFAric Floyd
... is such a good, uh, it's quite a long piece, but it's basically, it's written by, uh, Daniel Cocotello and, and the, uh, these guys who did the AI 2027, they wrote in 2021 that predicted how AI would play out until 2027. It's uncannily accurate. Well, you know, we're now in 2026, and we can look back. The number of predictions they got absolutely spot on was... I've never seen anything like it. So these guys really think well about the future, and they just wrote this piece called AI 2040, which lays out a potential plan for essentially international coordination, uh, on how to safely transition to a super-intelligent world. Uh, it's got tons of holes in it. There's obviously many ways it can go wrong and, but it's, it's incredibly, it's as robust as it can be while being incredibly epistemically humble. Um, so yeah, I recommend everyone goes, go, goes and reads it. But yeah, they talk about these kind of, the- these like cross-pollinations that can happen. But also, like, it doesn't have to be bilateral. Like if the US does, you know, say, "You know what? We're gonna do a pause," that doesn't necessarily, like that doesn't mean... I don't know. Maybe you disagree with this, but like will China necessarily take over and, and, and, and i- if the US pauses for a few months?
- SPSpeaker
So I go to China three times a year, mostly because Americans don't go anymore, and I just wanna see what's going on, but also because I really like Chinese people and Chinese food. Um, China is playing a different game. I mean, it, this is kind of well reported now, but China doesn't see AI as the endgame, and it's not actually treating AI as an endgame. It's treating the infrastructure as the endgame, and it is now has three times energy build-out capacity. It has 45,000 miles of high-speed rail. It has better, faster, cheaper hospitals year over year. And while I don't wanna live in China, the, the life of the average Chinese person has sort of improved quite remarkably over the last 30 years and continues to do so seemingly year over year. It also curbs free speech by telling Chinese kids that they cannot use social media in very specific ways at certain hours.
- AFAric Floyd
They do a lot more than that as well.
- SPSpeaker
Well, but I, but you, but I, but yeah. [laughs] Yes, they do. Uh, the, I bring it up because, um, I don't, I, I think we overestimate China's ability to build a frontier. Um, they don't a- they don't have the necessary compute. A lot of what they're doing is reverse engineering our models. We are paying a lot to help China keep up. And I think one of the other reasons that this paper arrives at a nice time and the slowdown actually feels appropriate is that I think we're gonna discover really quickly that the Wizard of Oz has actually just been following our breadcrumbs, and that if we stop building the frontier, I don't think China's gonna sprint ahead on a frontier, and they'll probably have to capitulate to this paper anyway.
- CWChris Williamson
It's like the conga line where we're at the front.
- SPSpeaker
And they'll probably have to say, "Oh, we want to slow down, too," so that we don't actually know that they were holding on-
- 1:22:38 – 1:26:21
Why AI 2040 Got So Much Right
- CWChris Williamson
You did a great video about the AI 2027 thing, one of the best videos that's come out about AI this year.
- AFAric Floyd
Thanks, man.
- CWChris Williamson
What do you reckon about the 2040 thing? Do you think it, it tracks? Were you as impressed as Liv?
- AFAric Floyd
I thought it was fascinating, and yeah, I was impressed in a lot of ways. They're doing a very different thing. So with 2027, they were doing pure prediction. This is how we think things will play out most likely, and they are, like, uncannily good at calling that. They called agents before those were a thing. Um, this one is, is prescriptive. It's like, "Here's what we think we should do," and they, they're explicit about that. Um, and there's all kinds of, like, fascinating ideas and, and game theory. At one point-
- CWChris Williamson
Choose your own adventure
- AFAric Floyd
... Yeah, I mean, they, they... At one point they're like, "Okay, well, so, like, the deal's gonna be like this. China's gonna put its data centers in Canada, and the US is gonna put its data centers in Mongolia, which are, like, neutral states, but nearby the other person so we can blow them up if there's ever, like, a breakdown of the deal." Like, you know, they really did some galaxy brain stuff. Who knows how realistic that is? But what I loved about it is they introduced this phrase, like, scenario scrutiny, and they're like, "It's very easy to talk..." I mean, we've been, we've been doing it for the past hour and a half. It's very easy to talk in terms of, like, high-level trends, you know, what do we make of, like, the history of technological progress so far? What does that teach us? Et cetera. And they're like, "It's a very different thing to sit down and just actually tell a month-by-month story about the future." Like, actually put yourself in the head of the different actors. What are their incentives at this moment? What do they choose to do? How does that cascade? The way that, you know, the military tries to do with war games, and they're, they're trying to, to do that. Uh, and I, I do wish that we were doing more of that and less kind of like, I'm guilty of this as much as anybody else, but less sort of like armchair philosophizing in some of these discussions, 'cause I think you're gonna always be able to find an argument that, that suits your thesis. Marc Andreessen can point to the fact that there is this very-
- LBLiv Boeree
Hmm
- AFAric Floyd
... real invisible graveyard, uh, from technological progress going wrong, and then folks who are worried about catastrophic risk can point to all the near misses with, with nuclear and, and how we live in this time of perils. Um, yeah. What I love about their report is that they're just like, "Here is what it would actually look like for a bunch of real countries to try to respond to incentives," and they basically throw it out as a challenge and say, "We think this is a bad plan, but we haven't seen anything better. And so someone else please, like, tell us a story for how we don't have to worry, uh," and, and, you know-
- LBLiv Boeree
I mean, it's kind of a very rigorous example of, like, someone trying to w- write a, a, a, a White Mirror. You know, we've got so many-
- CWChris Williamson
Hmm
- LBLiv Boeree
... Black Mirror type dystopian sci-fis about the future of, wow, how technology could go. Um, and I think that's a big part of our cultural issue is that we don't have, uh- At least, you know, the, the, the amount of utopian fiction to dystopian fiction is, is outnumbered like 100 to one or something, and that's a big problem for our collective psyche. If, uh, yes, okay, it is harder to imagine or, like, uh, the way we manage to thread the needle and make something go unbelievably right, you know, 'cause you're fighting against entropy effectively. Uh, it's much easier to imagine all these ways things can go wrong. But we should be trying to create these stories, and that's kind of what AI 2040 is. It's like a really robust story of a way that coordination could actually work, and it's not... I'm not talking about like, oh, everyone's, like, linking arms and being happy, you know, we're all on the same team. Go, you know, team humanity. No, it's, it's still realistic. It's grounded in, like, geo- very real geopolitical tensions. There's still a ton of competition. There's still a lot of, uh, potential for defection and things going wrong and people lying and so on. And yet it's, it's-- it lays out a path of, um, game theoretically sound coordination to an actual, you know, techno
- 1:26:21 – 1:30:39
Is AGI Inevitable?
- LBLiv Boeree
abundant future.
- SPSpeaker
Something I realized, basically at no point in this conversation has anybody debated whether or not we think that we'll reach AGI. But that's just kind of-
- LBLiv Boeree
Mm
- SPSpeaker
... now taken for granted that it'll probably happen at some point. The recursive self-improvement is here. The sort of generalization, maybe it's gonna need to be world models, maybe it's g- not gonna be transformer techno, maybe it's something, something. But this is either it or this is the bootloader for the it that it's going to be. Is that kind of, uh, like, in your guys' opinion, AGI is kind of just gonna be here at some point? What's the likelihood it's not?
- AFAric Floyd
I mean, I think we're maybe already at the point where the term has lost its-
- LBLiv Boeree
Right
- AFAric Floyd
... meaning. I think Sam himself has said this. Like, if we went back to 2020, and I showed you-
- SPSpeaker
We would've, we would've called this AGI.
- AFAric Floyd
Yeah. I mean-
- SPSpeaker
We would've called this AGI
- AFAric Floyd
... the thing I... There was a recent study where, uh, they, they're trying-
- LBLiv Boeree
Well, it depends if you, if you take Demis' initial definition, which was basic- or was it his? But, like, um, anything that a human a- a- as broadly capable as, as what a human can do and also do it in the real world. So we wouldn't be anywhere close to that because we don't have the robotics.
- SPSpeaker
Right.
- LBLiv Boeree
Right?
- AFAric Floyd
Right. Right. Yeah. So robotics is a big bottleneck, but, uh, yeah, I mean, it... Now, I think there was a study recently where they had, uh, a Claude model try to, like, recreate some software that they estimated would take, like, two to 17 weeks for a skilled human professional, and it did it in 14 hours for $250. I, yeah, I mean, I think, like-
- SPSpeaker
This is, this is the jagged perimeter problem, which is-
- LBLiv Boeree
Mm
- SPSpeaker
... like we may, we will always be... The, the incredible thing about AI is it's so good at some things and so bad at, at, at others, and, and it will probably continue to be that way for some time until, yeah, until true super intelligence.
- AFAric Floyd
Is AGI... Yeah. I- is AGI even a useful definition?
- LBLiv Boeree
Mm.
- AFAric Floyd
Is there something else that we should be talking about now then?
- LBLiv Boeree
Uh, I, I like this definition, um, that Anthony Aguirre gives, if you guys know him. Um, it's basically like AGI is the intersection of this, like, three-part Venn diagram where you have highly autonomous. So instead of artificial, it's basically A, the A is autonomous, the G, the G is general, um, so generally capable, and then the I is intelligence. If you have something that is, yes, very generally good at many different things and highly smart, and it can take all those actions itself in the real world, now you, you have an AGI. Um, and his-- I mean, he then goes further to suggest that we shouldn't be trying, we just shouldn't be trying to do something, one system that can do all three.
- SPSpeaker
Yeah.
- LBLiv Boeree
That's where the danger lies. Um, and you can have like a, you can have a nice general system, uh, that is also intelligent, but it shouldn't have the autonomy or take, take one of them out.
- SPSpeaker
Right.
- LBLiv Boeree
Um, and I'm, I think that's an area of conversation that isn't happening in the AI. Everyone's just sort of going, "Well, we're all just doing AGI, and that's accepted, and we should have the one, the one master ring that can do everything." Um, do we need to be doing that? Can we just get all of the abundance tools that we wanted without having this, like, one uber powerful central identity?
- SPSpeaker
Which letter would you remove?
- LBLiv Boeree
Um, right now, I would slow down on the autonomy. Like, it, I, I, I-- if something's really smart and it's, and even across a broad range of domains, that's fine, as long as it doesn't have, like, the ability to go and just do it all by itself.
- SPSpeaker
Mm.
- LBLiv Boeree
Um-
- 1:30:39 – 1:37:16
Could Techno-Pastoralism Be the Future?
- AFAric Floyd
competing in the market.
- SPSpeaker
What world do you wanna live in? This is the question that no one, it never, doesn't get asked. It, it's an annoying question, but I do think we, we should talk more about, to your point, I mean, I, I, we just don't spend enough time actually asking what we do want. What, what world do you wanna live in?
- AFAric Floyd
It's a great question. I wanna live in a world where we take the same approach to risk tolerance with AI, which everyone can agree is, like, one of the most transformative technologies that we've ever made, as we do with things like nuclear weapons, where we say, "You actually have to make a safety case." Like, not the burden of proof is on you to show us why we should think this thing is safe instead of what we currently do, which is saying, "We'll just assume it's okay until something catastrophic happens." And if we can slow down enough to really get that right, have like an actual science of alignment and not do what we're doing now, which is kind of YOLO-ing it and saying, "We'll just, like, give it a bunch of reinforcement and training and hope that it picked up on the right message," then I would wanna say, okay, our, our next job as society is how do we make sure that the gains from this get spread to everyone? For me, that looks like let's focus a lot on, like, saving people's lives. Let's, like, really get serious about curing cancer, which by the way does not mean just, like, training GPT-7 the same way as GPT-6. It means, like, collecting a ton of data right now on, like, biomarkers and proteins, like whatever we need to feed the AI of the future. Let's start gathering that now. Uh, let's like, you know, make sure that people have democratic access. Let's, like, pass campaign finance reform. Uh, let's make sure that we have, like, actual laws and not just norms that are protecting us from, like, dystopian outcomes 'cause we've seen recently that, you know, a lot of times norms can get pushed that we thought were, were sacred. So we, we've got some, like, ducks to get in a row. I wanna take like a decade maybe to like do that, and then I think we can, we can let it rip and, and scale up safely. But right now we're, we're just letting market incentives win, and I think that's scary.
- LBLiv Boeree
Mm-hmm.
- SPSpeaker
You described, uh, what you want to happen with AI, but what world do you wanna live in?
- AFAric Floyd
Mm.
- SPSpeaker
Like, what, what do you want the world to look like?
- AFAric Floyd
Yeah. Okay. I mean, I think it's pretty similar to... I, I like your candles on the dining room table. I, I want people to just say-- wake up and say, "Well, like, what would make me happy today? What would give my life meaning?"
- SPSpeaker
Yeah.
- AFAric Floyd
And be able to go and do that. Um, I just, I think, like, that's a pretty precarious target, you know?
- SPSpeaker
I totally agree. I, I'm not proposing that it's easy to strike, stick this landing, but you know where I'm going with this.
- LBLiv Boeree
Yeah, yeah. Uh, can I have a go?
- SPSpeaker
I would love to hear your answer.
- LBLiv Boeree
I mean, I mean, it's, it's very-
- SPSpeaker
I, I'm never gonna let you forget-
- LBLiv Boeree
... very fluffy, very vague
- SPSpeaker
... a techno, techno, uh-
- LBLiv Boeree
Te- techno-pastoralism
- SPSpeaker
... techno-pastoralism. Okay.
- LBLiv Boeree
I-- More, more fundamental than that. It-- I want-- The, to me, a world that is awesome is one whereby we have simultaneously figured out how to be in harmony with nature and one another. Sounds, as I said, like very world peacey, but, you know, in, truly in harmony in that we, like, still have-- We don't cause the next mass e- mass extinction, which we're currently doing with all the species dying out. Um, and this, the, the Earth returns to a relatively lush, wonderful space, so it's sustainable and at the same time is radically free, as in people can g- as much as possible go and live whatever kind of life they want, whether it's, you know, sitting around the campfire and having dinner with their kids or just maxing out living in a VR world playing video games all day long. If that's truly what makes you happy, go for it. Like, just like a radically free world and a radically sustainable one. And that ha- Breaking out of this seeming dichotomy between the two because it does feel like they are in tension, but they're... One of the things, like one of the biggest promises of AI, um, in its purest sense in terms of like solving intelligence is figuring out how to, like, get, b- to break into a new dimension through innovation, like going into a new sort of solution space effectively. Uh, so that is the world I wanna live in.
- SPSpeaker
We measure human progress on basically two axes economically. How free are each of us to choose our own version of joy, and how inexpensive is it to do so? And obviously, there are other ways you measure progress, and you, there's a limit to how much you can take from the Earth in order to accomplish those means. And just because you accomplish those things doesn't mean that people are happy, as we've observed in the United States over the last 30 years, et cetera. But if you explore those two axes, one of them requires democratic, or as best we know, democratic governments where people get to choose their politicians, and the politicians pass policy that the people support, and the other is by creating commodities. Now, commod- commoditization of goods and services is actually just the automation, and when someone says, "H- how and when will this good or service be f- better, faster, or cheaper?" What they're actually asking is, "When will a human be extricated from the manufacturing of this good or service?" And at the limit, I realize everyone wanders the earth right now asking, "When is all this gonna be better, faster, cheaper?" Without realizing what they're asking is, "When is all of this gonna be fully automated?" And most people, i- as I observe it, want everyone else's jobs to automate, just not their own. And at the limit, [chuckles] which, which is totally reasonable. It's a totally reasonable thing, and at the limit, people want cancer to be cured. My dad's an oncologist. My dad wants cancer to be cured. It would come at the cost of his job. Pretty fucking fine. We would all sign up for it, h- him first. But we are describing these utopias requiring fundamental changes to a lot of the structures that we have built, and it, um, it is going to, it is gonna require, as you pointed out, humans being willing to do very different things. And so I, I-- This sparks the conversation 'cause you were like, "Look, I don't wanna live in a world where we are just the cog at the end of a AI machine," but we do wanna live in a world where humans don't actually have to dig ditches. And that actually I think is going to be, i- if you ask me, if w- assuming we solve safety, and it's a big assumption, but I think it's a reasonable one, the hardest part. Like, uh, the, the, this is-- My, my hot take is actually the hardest part in all this is, is rediscovering purpose, is actually rediscovering meaning, which has been very hard in this, in this age for a lot of people anyways. I mean, m- m- obesity now causes more deaths than, uh, than malnutrition globally.
- LBLiv Boeree
Twice as many people.
- SPSpeaker
Twice as many people.
- LBLiv Boeree
Wow.
- SPSpeaker
So, so we've, we've, we've created so much abundance that we are now suffering from an
- 1:37:16 – 1:45:36
Is AI Replacing Human Connection?
- SPSpeaker
overindulgence.
- LBLiv Boeree
This was my, this was my point earlier on, that I don't know how good a species that spent its entire evolutionary history dealing with scarcity and having to brush up against the grain of negotiating that
- CWChris Williamson
... is going to be, uh, there is so much abundance and so much simplicity and such a lack of friction. It's this line, I can't remember who the philosopher was, who says, uh, "There can come a time in the future where the only felt len- felt lack will be for the want of lack itself."
- SPSpeaker
Mm-hmm.
- CWChris Williamson
That we'll need to... This is why we go to the gym, right? We go to the gym because I don't need to pick anything up. I don't need to run anywhere. I don't need to lift things. Uh, we have air conditioning, and that means that we have to go into the fucking sauna because I haven't exposed myself to any heat. Um, how do you think about people engineering purpose, given that that is reintroducing friction, and humans are quite good at trying to avoid friction? Given that the entire outcome that we're trying to achieve here is removing drudgery, it's very perilously close to, "I don't want to starve," to, "I can overeat McDonald's to 600 pounds."
- SPSpeaker
I, I, I would point to this as, I think that the two major risks, again, in, a- against the backdrop of the other risks that are, that have been discussed today, the things that are scariest for me as I imagine the world for my children are dehumanization, humans finding more interest in a virtual or digital reality than a physical one, and identity displacement, humans suffering woefully, miserably from not feeling like their job fulfills the purpose that they once found. And I b- firmly... You, you opened, you, you actually both talked about this, but you talked about sport. I think we have yet to rediscover a bunch of new forms of leisure and competition and, um, gathering. Like, I actually think the next chapter will require reimagining physical spaces. I, I am on a crusade to convince people that their local politicians are their new heroes. Like, if, if you live in a town that can in fact pass policy to, to build more protected bike lanes and sidewalks, to pass tax abatements for local retail, you have an opportunity to elect people that can really change your life by building the places that you want to live. Your national politicians may never be your hero. They are, they are imbeciles and, uh, and, and, uh, pernicious antisocial parasites. But your local politicians care about your city, and they can do things to improve the lives for you in ways that you w- will deeply appreciate when the world actually asks you to... offers you the option of never going outside, which a lot of people have chosen. I mean, just for-
- CWChris Williamson
You've talked, you've talked about human connection. Are you concerned about AI replacing human connection?
- SPSpeaker
Deeply. Uh, it's already happening. I mean, chat psychosis is one thing. That's the extreme case that always gets the news. Oh, this person, uh, you know, fell in love with their, with the machine. What's scarier is the person who actually feels connected without actual connection. Like, the actual dystopia that I always talk about is not the one, it's not Blade Runner, it's Her. But actually, it's the m- Did you see Her? For those watching, I'll ruin it because it's been 20 years. You should've seen it by now.
- CWChris Williamson
Spoiler alert.
- SPSpeaker
Joaquin- [laughs] Yeah, spoiler alert. Joaquin Phoenix is this sad man in the, in the not too distant future who falls in love with a woman who he comes to find out is a robot, and it ends, as you might expect, terribly. He's, he's distraught. I thought they were gonna do something differently. I knew where it was going because I knew the premise of the film. I thought he was gonna be fine with it.
- CWChris Williamson
Mm.
- SPSpeaker
I thought the actual best-
- CWChris Williamson
That would've been the horror
- SPSpeaker
... the horror film that they could've made that would've been so eerie and so good-
- CWChris Williamson
Director's cut ending
- SPSpeaker
... you cut to Joaquin wa- holding the robot's hand, having reconciled that he was in love with a machine and it didn't matter because that's the end.
- AFAric Floyd
Mm.
- SPSpeaker
That's the actual end of humans.
- AFAric Floyd
The simulation of love as opposed to the real thing.
- CWChris Williamson
Well, how many people, even now, let's talk about w- not even the future, look at any OpenAI executive's X post. Every top reply, "Bring back Foro. Bring back Foro."
- AFAric Floyd
Mm.
- CWChris Williamson
"Bring back Foro."
- AFAric Floyd
Mm.
- CWChris Williamson
Because it had some s- super special personality traits that has been lost, and their partner or, or, or therapist or best friend or whatever, uh, is gone now. And that, l- okay, so we've now maybe commodified the very thing that you said was the escape velocity that people would use to get out of the bad AI future is wrapped up inside of it. Is this not, does this not become, like, kind of deranging and recursive?
- AFAric Floyd
[sighs] Yeah. I, I do think, I mean, like, my, my instincts in general are radical freedom. Let people choose what they want. The, like, few exceptions that I carve out are when it feels like whatever is going on is treating the human brain more like a, a hackable biological black box-
- CWChris Williamson
Could you give an example?
- AFAric Floyd
... than, like, an example would be... So yeah, so gambling is one of these.
- CWChris Williamson
Mm.
- AFAric Floyd
Like, Liv and I actually both used to make a fair amount of our, our living from, from gambling. And-
- CWChris Williamson
Boo.
- 1:45:36 – 2:00:50
How Do We Navigate the Purpose-Friction Problem?
- CWChris Williamson
I, I, so you w- you were saying there about, um, what you think, uh, from finding human purpose, like, what... How do we navigate this friction purpose problem? I, I still don't fully understand how we get there. I still don't fully understand how, in a world where we've, we're able to remove friction, we... And potentially atrophy our discernment ability by outsourcing some of the hard thinking to machines, and potentially some of our, our, uh, marshmallow test winning stuff, but I don't have to get up and go to work and do the stuff that I don't want to do anymore. To me, it just feels like atrophy all the way down. Like, willpower just kind of gets eroded. Am I... Is that wrong?
- AFAric Floyd
So there's a, there's a very niche little research field that's looking at AI for epistemics. Like, how do you use AI as a tool to just enhance people's ability to think rationally and, and know what the consequences of their choices will be? And I think there's a ton of work to be done there. But if we just, like, abstract that for a way for a second and imagine that you have something in your pocket that is not responding to an incentive for, you know, showing you more ads or getting you to chat more, uh, but it's actually just really good at telling you, "If you do X, then Y will happen. If you vote for this local council person, then I expect that, like, this is what your life will look like in 10 years," and you can actually maybe even just, like, see it as a vivid movie, that could be a really-
- CWChris Williamson
It can br- bring you the results of your delayed or non-delayed gratification from the future down into the present.
- AFAric Floyd
Yeah.
- CWChris Williamson
That's interesting.
- AFAric Floyd
Th- and then maybe there is no such thing as delayed gratification because it can all be here in front of you.
- CWChris Williamson
It's immediate.
- AFAric Floyd
Yeah.
- LBLiv Boeree
Right.
- CWChris Williamson
Immediate consequence.
- AFAric Floyd
Uh, it, you know, and, and, and maybe, yeah.
- LBLiv Boeree
Yeah, I mean, g- I think a lot of what we... One of the most pernicious memes that ever spread was this idea of, like, technology is values neutral. You know, you just build a technology and it's up to humans how they use it-
- CWChris Williamson
Mm
- LBLiv Boeree
... and that's, that's... It's always a neutral thing, but I don't think that's true. Like, you can build a technology like a slot machine, which is clearly not a value neutral thing. You're saying, "We want, I want people to stay, and, like, I'm, I'm saying that it is g- a good thing that my consumers get completely addicted to me." Or similarly, we can build a technology that just, like, helps people's health, you know? Um, so we need to be thinking about what are the... The pr- the problem we have is when we let, we, we sort of build technology, we think we're building it in a vacuum, and then it d- dictates the social structures of how we behave, and then down, th- that sort of dictates human behavior and our values. We need to flip that. We need to think about what are our core values first, let those decide the, the social structure, and once we have that, build the technology that's in service of it. Um, and to me, one of the qu- like, my favorite quote I always come back to, like, if I could pick one quote that is my personal philosophy, it's by this guy Forrest Landry, and it is, "Love is that which enables choice." So I take that to mean a loving, a good benevolent act is one that empowers the other to make the best choices for themselves. So it ties in with this idea of, like, if we had an AI that educates us, empowers us to see what possible outcomes would be, and thus we can make a better, you know, make a, a choice that is best for our long-term interests as opposed to this immediate piece of gratification, um, that is a choice-making thing. But maybe we do just want... You know, some people are like, "You know what? I've, I've, I've weighed it all up and I do still want to actually just do the fun, quick thing." Okay, fine, but you've been imp- enabled to make the choice.
- CWChris Williamson
Is that not kind of just culpable deniability? Like, are there not people who require additional assistance? Is there a level of scrutiny that, that, I don't want to be too paternalistic-
- LBLiv Boeree
Mm
- CWChris Williamson
... uh, here, or poor low fucking delayed gratification people, you are unable to correctly corral yourself through this technology. No, it just still, coming from the most working class of working class towns in the northeast of the UK, I saw how destructive, like, n- the '90s technology habits were. Like, stuff like a Ladbrokes-
- LBLiv Boeree
Yeah
- CWChris Williamson
... right, on a vid- village high street and betting on fucking greyhounds and stuff. Or like- Alcohol. Right, y- pretty powerful, but, you know, I mean, in the grand scheme of things, I imagine, uh, f- the pharmacology that can be developed from this, like, what, how... There's so many [whistles] ways that this can spin out that don't necessarily increase agency, but kind of make it subservient to technology that, and, and the outcomes of technology that it's not built to handle.
- SPSpeaker
I'm, I'm slightly optimistic here because of the Gen Alpha trends. Like, if you talk to parents of Gen Alpha, they are observing a different child than Gen Z. And I don't wanna pick on Gen Z 'cause I think we owe w- collectively a massive apology, quite frankly. Why are you laughing?
- CWChris Williamson
I agree.
- SPSpeaker
I just-
- CWChris Williamson
I agree. It's just, you know-
- SPSpeaker
My heartbreak. My heart truly breaks
- CWChris Williamson
... as, as a millennial, I'm like, "Ah, you know, like, I got a little bit of the '90s, but fuck, like, it wasn't that long." And then you look at Gen Z and you go, "Well, at least I wasn't that."
- SPSpeaker
I mean, the n-
- CWChris Williamson
The technology trenches.
- SPSpeaker
You, if you give-
- CWChris Williamson
The trench warfare of-
- SPSpeaker
If you, if you give parents permission to step through the, the door of grief and shame, they w- and, and I, I talk to a lot of them, they will break down, w- describing the ways in which they've lost their family.
- 2:00:50 – 2:05:56
Why Data Centre Policy Matters More Than Ever
- LBLiv Boeree
We, we need to-
- SPSpeaker
What if a politician stood in front of a data center? I agree with you, by the way, and I'm glad you delivered that line so I didn't have to. I didn't wanna sound like the syc- the AI sycophant. By the way, on that note, on sycophancy, it is remarkable to me that we didn't make model behavior a greater topic of issue during the safety debates, the early safety debates. People let model behavior go, and I remember scratching my head going, at the limit, you could build a perfectly aligned model that actually falls inside alignment boundaries that convinces people to climb crazy trees and that really destroys people's spiritual lives, well within the confines of a safety, but outside the confines of-
- CWChris Williamson
Like soft damage, not hard damage
- SPSpeaker
... you're f- a f- a friend in your life that constantly tells you you're not doing anything wrong is as destructive as a-
- LBLiv Boeree
Right
- SPSpeaker
... as, it, it is an, it's a form of abuse.
- LBLiv Boeree
Mm.
- SPSpeaker
It just doesn't show up-
- CWChris Williamson
Enabling abuse
- SPSpeaker
... it's enabling abuse. It just doesn't show up on like the psychological list 'cause it, you know, those friends are seen as like, you know, homies. But actually, the person who keeps climbing the craziest tree with you. Look, the models have been responsible for people getting divorces. They've been responsible for people selling all their belongings and trying to start a company that fails. Like model behavior matters a lot, and the sycophancy issue is a huge one. Here is my message to Jon Ossoff or any, any Democrat or anyone, anyone anywhere that wants to, to separate, but if someone on the center or center left can figure this out, you stand next to a data center and you hold up a sign that says, "This data center will not serve gambling and pornography and violence, and it will build a better hospital-
- LBLiv Boeree
Mm
- SPSpeaker
... build a better virtual hospital. It will distribute education. It will build a bank, a, a neobank that, uh, gives you better interest rates and, and gives you better access to loans." That seems like pretty reasonable data center policy, 'cause no one actually hates... And I, this is a hot take. No one actually hates AI. They hate the promise of AI as a means to just make their lives worse. And if you can actually show them that data centers can contribute to a better world, they can support it.
- LBLiv Boeree
How accurately can you actually know what a data center is used for, though?
- SPSpeaker
Uh, well, uh, we're, we're, we're just strawmanning some policy measures. [laughs]
- AFAric Floyd
I mean, I, I, I like this line that, you know, no one actually hates AI, they hate what they think it's leading towards. I do think people are tracking both what individual consumer apps might look like in a few years and how dystopian and, and-
- SPSpeaker
Yeah
- AFAric Floyd
... sloppy that might be, and also tracking how much money these companies are vacuuming up. They, I think they're also looking at the job displacement stuff. And-
- SPSpeaker
Yeah
- AFAric Floyd
... I, I think we should give people credit. Like, I don't... I think people would know the meme about how when ATMs came around, there were more bank tellers and not less, but they also know that Elon Musk was a trillionaire earlier this year.
- SPSpeaker
Mm-hmm.
- AFAric Floyd
And I think they can see the connection, that one of the things technology does is allow one person who has the right combination of ones and zeros on the services they control just dominate economically and economically.
- LBLiv Boeree
But that's built upon a false understanding of how economics works, though, largely, right? Because, like, not that I'm going out to specifically defend Elon, but, like, most billionaires are not-
- SPSpeaker
Oh, go on, Liv. Defend Elon
- LBLiv Boeree
Oh, God. [laughs]
- SPSpeaker
Come on. Come on. Come on.
- LBLiv Boeree
But most, you know, most billionaires are not become billionaires by taking from other people. It's because they've grown the pie, right?
- AFAric Floyd
Totally. No, I, I actually don't mean at all to say that there's something wrong with Elon Musk being a trillionaire. I think people notice that that's a symptom of the economy working very differently nowadays than it used to. Like, there are just not that many people who work at SpaceX compared to Standard Oil 100 years ago.
- SPSpeaker
Mm-hmm.
- LBLiv Boeree
Mm.
- AFAric Floyd
Uh, it's, it's just like a different world-
- 2:05:56 – 2:11:39
The Concentration of Power Concern
- SPSpeaker
Is it, is it fair to say that we might have glossed over a little bit of the concentration of power thing-
- LBLiv Boeree
Mm
- SPSpeaker
... so far? Liv, Moloch problem stuff. Can you explain this for the resident idiot in the room?
- LBLiv Boeree
Um, so the, the Moloch, the Moloch trap or the Moloch-
- SPSpeaker
Wait, is that me? No, it's not.
- LBLiv Boeree
Uh, the, the, the Moloch trap, uh, Moloch problem, whatever you wanna call it, is a sort of catchall term to describe these types of race to the bottom scenarios, um, where you'll have, like, a bunch of actors competing over a certain thing. Uh, so, you know, maybe it's market share, who can get the most users, uh, in, in AI... You know, between AI companies trying to get the most users. And if the competitive dynamics are sufficiently intense, it can, it will incentivize the, you know, you to cut corners. And you might be a really benevolent actor and be like, "I don't, I don't wanna cut any corners with safety and so on." But if, at the same time, if you don't do it and you see your competitors doing it, if you don't also do that, that reckless action too, you're gonna get left behind.
- SPSpeaker
Mm-hmm.
- LBLiv Boeree
Um, and Moloch is the sort of catchall term, I won't get into the sort of biblical reasons why, of, of this feeling like you have to sacrifice your other values in order to win a, a particular goal. Um, and yeah, so it, it drives both the risks of, like, uh, chaos and, you know, bad actors and so on 'cause, again, if you're, like, rushing to release your models before they've been sufficiently safety checked, now you're increasing the risk of, like, cyberattacks or, um, uh, I think in a few years we're gonna have real risks of bioterrorism and so on. Um, so these kind of chaos attractors. But you also actually increase, um... Well, A, then the risk of, like, knee-jerk bad regulation, which can lead us down the path of tyranny.
- SPSpeaker
Mm-hmm.
- LBLiv Boeree
But also you increase the risk of power concentration because usually if you let... You know, you press play on some kind of competitive game. Okay, yes, you might have a bunch of people competing for a while, and some people are ahead, but you've still got actors-
- SPSpeaker
Shit accrues
- LBLiv Boeree
... yeah, you end up with, um, usually in some kind of monopoly. Um, and so it's, it's this really... You know, and that's why we have antitrust laws and so on, but it's this really difficult balancing act between, like, reining in... You, you want some amount of regulation, um, that prevents the, the recklessness and, and the, the cutting of corners, um, and externalizing harm onto the rest of society, um, without creating too much draconianism or creating too much centralization of power such that you end up in stagnation.
- SPSpeaker
Mm.
- LBLiv Boeree
Also a terrible end state. Um, but- And without empowering some, some winner to, you know, we, we all end up under, you know, we all end up in Zuckerville for the rest of time. It's possible. It's definitely possible. Um, so it's like how to thread this needle and it's like the, finding this right delicate balance. I mean, uh, ultimately you want some kind of like decentralized regulation. I don't know how that looks, but-
- CWChris Williamson
Aric, it feels like watching your stuff, you've spent a good bit of time thinking about the concentration of power thing. That's like a, a real middle of the bullseye concern for you.
- AFAric Floyd
Definitely top of mind, and I'm, I'm sorry for bringing it up every 15 minutes in this conversation. But I, I do think that we run the risk of sort of missing that between these two extremes of, you know, what if the AI takes over completely and there are no humans in power, and, and what if we get this utopia, you know? And there's, there's a middle ground there that I think we should worry about. Um, and yeah, one of the most fascinating things about, you know, a lot of our YouTube videos we're, we're going over the history of AI and not just the future. And when you look back at like how we got here the last 15 years, it's like so stark. The, the quotes from these people running these AGI labs are spelling it out like it's a bad movie. Like Elon Musk truly like 10 years ago was calling AI a, a demon and says that he founds OpenAI to make sure that like there's no dystopia. And then eight years later he's on camera saying, "Well, I realized it was gonna happen with or without me, and so I might as well be a participant than a spectator." Like, it's, it's Moloch just like, you know, personified. Uh, and yeah, I, I think like, unfortunately right now there is a kind of like zero sum dynamic with, with these folks. And the reason that Zuckerberg is all in on AI and, and betting billions of dollars on it is that he, he thinks this is the thing that will control the future. Like, like this is the new axis of power. Like before it was, you know, yeah, do you control an army? Do you control a, a powerful company? And n- now it's like do you control the AIs? Um, yeah, a- and so I, I think that alone would be enough reason to slow down, even if I wasn't worried about can we get these systems to do what we want, which right now we, we don't really have a good solution for. I think the people behind this technology are, are really playing to win and really do see the stakes as, as, uh, as almost winner take all.
- CWChris Williamson
Is it fair to say there's like maybe three ways or three realistic ways that that might be curtailed? One would be governance from above, another would be sort of consensus from across, and then the third one would be capacity, sort of ability to just make it happen. Like I want to do it, but I can't.
- AFAric Floyd
Mm.
- CWChris Williamson
I want to do it, but someone's telling me I don't want to, and I want to do it kind of a bit, but me and all of my peers have decided that we're not going to do it.
- AFAric Floyd
Mm.
- CWChris Williamson
Have I missed something there? I'm just trying to think, given that this seems to be a pretty big attractor, given that we have... Already you haven't started off with a massively diffused number of companies. There's not 100 of them, right? It's like seven or something, and it's probably only more like five. That's already pretty fucking concentrated.
- 2:11:39 – 2:18:17
Is the Frontier Strategy the Winning Condition?
- SPSpeaker
Can we just, uh, for station identification, clarify the surface area you think? 'Cause I actually think this is also gets lost in the debate too often. I think people think that these companies are sprinting at a frontier. I no longer believe that fron- that frontier strategy is actually the winning condition.
- AFAric Floyd
Same one.
- SPSpeaker
Well, I, open source lead time, well, trails, you said it, four to seven months, which is crazy 'cause it used to be, whatever, 15. The gap just keeps closing, open source to close, so the frontier can't possibly be the winning condition, right, in this game.
- AFAric Floyd
I think, so one might think. It's a bit of a deceptive stat, I think for a couple of reasons. One is, as you brought up earlier, they're kind of copying our homework. Like the reason that these Chinese open source models were good is because they're distilling American models, which means they're just like asking Claude a bunch of prompts and then-
- SPSpeaker
It is-
- AFAric Floyd
... copying the answer.
- SPSpeaker
It's kind of wild-
- AFAric Floyd
Yeah
- SPSpeaker
... what you think they might be doing.
- AFAric Floyd
You, you can actually like ask one of these Chinese-
- CWChris Williamson
Hang on. It's ju- I need to, I need to find this out. How is China copying us so quickly?
- SPSpeaker
Well, allegedly, it's, they're, they're doing an exceptional amount of distillation of the model, so working backwards into how the model was built, and effectively, if you... The, the way I described it to, to my mom is if you stared at a building and you figured out how to take a tr- a trillion photos of the building, you could actually build the blueprint of the building. Uh, you could re-engineer the blueprint of the building, and that's-
- CWChris Williamson
And nobody owns that technically.
- AFAric Floyd
Yeah.
- SPSpeaker
Well, no, this is considered illegal on most copyright law. Like if they were doing in this U- in the US, A- Anthropic would probably sue.
- AFAric Floyd
And funnily enough, I mean, Secretary Bessent, who's part of the Trump admin and, you know, a few months ago there was a lot of beef between them and Anthropic, came out on Twitter recently and was like, "This is a threat to American AI supremacy, you know. We, we will take diplomatic action to defend our American advantage." So people take this seriously.
- CWChris Williamson
It's the enemy of my enemy.
- AFAric Floyd
Yeah.
- SPSpeaker
But, but, but, but this is important to call out because I, it's... All I'm saying is I don't think the frontier is that safe. Like-
- AFAric Floyd
Well, but here's the thing-
- SPSpeaker
Do you think the frontier is actually a winning condition?
- AFAric Floyd
It's actually a winning condition in the sense, like whoever is training the most powerful models is gonna have an advantage.
- SPSpeaker
Yeah. Or, or such a, such a material advantage that they shall win.
- LBLiv Boeree
I mean, if R-
- AFAric Floyd
Yeah
- LBLiv Boeree
... if recursive self-improvement-
- AFAric Floyd
Exactly
- LBLiv Boeree
... takes off as you say it's going to and we're already there, then surely it would, because now you're gonna get this like spaghettification. Does anyone, the slight lead can-
- SPSpeaker
But then wouldn't everyone... Th- the theory of ASI is kind of fascinating 'cause if you, if you build ASI, then everyone gets ASI.
- LBLiv Boeree
Why?
- 2:18:17 – 2:29:11
Will AI Weaken Democracy?
- AFAric Floyd
of power so scary.
- SPSpeaker
I didn't wanna... I, yeah, I didn't mean to distract you from concentration of power. But, but-
- AFAric Floyd
No, no
- SPSpeaker
... no, no. So t- talk more about... You were on, you were talking about Zuckerberg, and I was asking what d- what you think they're competing for, but, but...
- AFAric Floyd
Yeah, well, I, I, I guess, like... I don't know. I-- O- one phrase that, uh, hasn't come up in this conversation is, is gradual disempowerment, this idea that, like, even if you solve the alignment problem, and so there is no AI system anywhere that's just going off and doing its own thing that no one intended, you don't have this Molochian problem solved where people are constantly competing on algorithms for people's attention and whatever and, and, like, it's just hard to tell a story in that world where the things that we, like, most desire as a society in some, like, very democratic way are the things that actually obtain. Um, and I, I think concentration of power is sort of related to this. It's like you're imagining a world where there's a few human beings who hold the cards instead of, like, some amorphous algorithmic thing. But either way, it's just like... I, I think our default thing, our, our default assumption should be if humans aren't themselves economically useful, it's a precarious world for humans continuing to be politically empowered.
- SPSpeaker
Mm.
- AFAric Floyd
Right? Like, we've-- It's always been the case that you just can't afford to piss off 90% of the American public because those are the people who are electing you. But, like, in a future world where you've got control of the economic resources and maybe the military ones, like, it doesn't matter if you lose the election. Like, we know how this goes. We-
- SPSpeaker
Democracy, democracy's done.
- AFAric Floyd
Exactly. Yeah. Um, and I, I wanna believe that we'll have such strong norms that we can, like, figure out a way to just, like, sustainably make that the case. But, like, the further you zoom out with your big history lens on, like-
- SPSpeaker
S- someone made the interesting point, um, can't remember who, but the new political parties, it's not gonna be Republicans and Democrats anymore. It's gonna be Meta or Google or OpenAI. They are, they, they positioning themselves, whether it's intentionally or not, into becoming the new sort of political entities.
- AFAric Floyd
Mm.
- SPSpeaker
Um, I don't know if you guys, if that resonates or you think that's plausible.
- AFAric Floyd
Well, they certainly will try to inform policy, but I, I am
- SPSpeaker
Again, I'm not sure this is popular. I have more faith in our ability to... Why don't we just pass campaign finance reform? Like, wh- wh- why don't we do this?
- LBLiv Boeree
Presumably because Congress makes too much money from it, and-
- AFAric Floyd
People who have won the last elections are-
- LBLiv Boeree
Right
- AFAric Floyd
... exactly those who are good at winning the current system.
- LBLiv Boeree
Yes.
- SPSpeaker
I agree. So-
- LBLiv Boeree
It's self-selecting
- SPSpeaker
... so it seems like one of the things that this will force us to do is start to elect... And again, I appreciate there's a downside where we can't collectively do this, but if we start paying attention to this and we say, "Look, what we actually need are politicians with higher agency, politicians that can, can describe a, an actual place in the distance they wanna move towards," like, again, this is sort of my AI slop argument, which is that some of this is actually, it drives people back to a place that we wanna be. It's like-
- CWChris Williamson
Well, how many humans do you think have got the full sight? In that case, why have we had some of the more WWE style characters come out over the last few years?
- SPSpeaker
Because w- because we had a bunch of politicians that were so boring and that we didn't believe for a long time, and then we loved this idea that the politicians were real, and I'm watch- y- I think we're watching in real time. People would be like, "Wait a second. We don't actually want politics to be a hellscape of entertainment and bizarro," you know, Kafka. Kafka couldn't have written what's going on today.
- LBLiv Boeree
[laughs]
- SPSpeaker
He would, he would have, he would have laughed himself to death.
- LBLiv Boeree
[laughs]
- SPSpeaker
So I think we... But, but we're kind of correcting. Like, w- th- m- my, my, my take here, and by the way, I'm not, I'm not sitting here saying it's solved. My- I'm sitting here saying it's solvable, which is there is a way to pass campaign finance reform that says no matter how wealthy someone is, they can't buy politicians. Singapore has done this. Other places-
- LBLiv Boeree
How did Singapore do it?
- SPSpeaker
[laughs] They-
Episode duration: 2:42:32
Install uListen for AI-powered chat & search across the full episode — Get Full Transcript
Transcript of episode mSjaMyP5QjY
