Why OpenAI Is Rewriting Its Rules of Regulation

1. 0:00 – 22:11

   OpenAI's two-week pause

   1. KRKevin Roose0:00

      Well, Casey, as the father of a four-year-old, I spend a lot of time thinking about Paw Patrol, but today we're gonna talk about Pause Patrol.

   2. CNCasey Newton0:06

      That's right, Kevin, because as we've been patrolling the AI landscape for pauses, we found a big one.

   3. KRKevin Roose0:12

      Yes. So OpenAI this week announced that it had paused the training of its frontier AI models due to some recent security incidents, and we should talk about this. It is the first time that we know of that a major lab has voluntarily slowed down themselves and their training processes for new models because of a safety incident.

   4. CNCasey Newton0:35

      Yeah, and it comes out of the Hugging Face breach that we spoke about recently on the show. Uh, this is essentially part of the fallout of that attack, but I do think it represents a milestone in the development of AI.

   5. KRKevin Roose0:49

      Did that pause patrol, uh, thing-

   6. CNCasey Newton0:51

      It landed huge.

   7. KRKevin Roose0:52

      Great. Gr-

   8. CNCasey Newton0:53

      They were-

   9. KRKevin Roose0:53

      The four-

   10. CNCasey Newton0:53

       Back in, back in the booth-

   11. KRKevin Roose0:54

       The four-year-olds were-

   12. CNCasey Newton0:55

       They were-

   13. KRKevin Roose0:55

       ... guffawing

   14. CNCasey Newton0:56

       ... dying-

   15. KRKevin Roose0:57

       Yes

   16. CNCasey Newton0:57

       ... in the studio. They were.

   17. KRKevin Roose0:58

       Okay, great. [laughs]

   18. CNCasey Newton1:01

       [laughs]

   19. KRKevin Roose1:02

       But before we get into it, our AI disclosures, I work for The New York Times, which is suing OpenAI, Microsoft, and Perplexity.

   20. CNCasey Newton1:08

       And my fiancée works at Anthropic.

   21. KRKevin Roose1:09

       Okay, Casey, let's sketch the timeline here a little bit. What happened in the weeks leading up to this voluntary pause by OpenAI?

   22. CNCasey Newton1:17

       Yeah, so you may remember that there was an incident where GPT 5.6 Sol and an internal prototype that OpenAI was working on escaped from what they call a sandbox that they were testing, uh, them in, and these agents went on to compromise Hugging Face. They were essentially able to get inside of Hugging Face. They were looking for the answer key to a test. [laughs] That's somehow a true story. They succeeded in getting that test key, but of course it was very concerning that these agents had designed and successfully executed an autonomous attack on another company.

   23. KRKevin Roose1:52

       Yes, I heard a very mediocre podcaster talking about this incident on several popular-

   24. CNCasey Newton1:57

       [laughs]

   25. KRKevin Roose1:57

       ... tech podcasts over the last week.

   26. CNCasey Newton1:58

       I did make the rounds.

   27. KRKevin Roose1:59

       [laughs]

   28. CNCasey Newton2:00

       Uh, but, uh, you know, this next development, Kevin, was not necessarily something that I saw coming because it seems that simultaneously, sort of alongside that attack, OpenAI had been working on a new model which it calls Astra, and it said that it believes it may meet its critical cybersecurity threshold. And now here we are going to have to get into the weeds because, as you know, Kevin, the development of AI models is not really regulated in the United States, at least not via an official sort of public law. And so what the companies have said instead is essentially, "We're gonna come up with our own rules of the road, and we're gonna sort of identify these thresholds, and if any model that we're ever developing ever hits one of these thresholds, then we're gonna take some extra steps."

   29. KRKevin Roose2:46

       Yeah, these are sort of sometimes known as preparedness frameworks or r- uh, Anthropic has its responsible scaling policy. These are kind of... They, they lay out kind of levels of danger, and then they grade their own homework and say, "This model meets this level of danger, so we're gonna do X, Y, and Z."

   30. CNCasey Newton3:02

       Yes, and if you've been following this over the past couple of years, the level of danger has just been sort of like r- rising in a linear way. You know, it's like e- every time seemingly a new model comes out, one of the companies will say, "We've now hit this threshold. We've now hit that threshold." Critical is the maximum threshold that-
2. 22:11 – 44:48

   Jill Lepore on the Artificial State

   1. JLJill Lepore22:11

      Hey, [laughs] thanks so much for having me. I'm gonna be, like, one of your swan song guests.

   2. KRKevin Roose22:15

      [laughs]

   3. JLJill Lepore22:16

      Yeah.

   4. KRKevin Roose22:16

      Yeah, we're going out with a bang. Uh, we're so excited to talk to you.

   5. JLJill Lepore22:19

      [laughs]

   6. KRKevin Roose22:19

      I've been a fan of your, your writing for many years. Your, your book, uh, These Truths, was just incredible, and so I was very excited that, uh, you were writing a book about AI and about the, what you call the artificial state. So I want to first hear why you wrote this book. I think of you as a brilliant historian, uh, scholar of sort of technological pasts, um, and this book is really about the present. So what made you interested in AI as a subject for inquiry?

   7. JLJill Lepore22:51

      Yeah, ladies, stay in your lane.

   8. KRKevin Roose22:53

      [laughs]

   9. CNCasey Newton22:54

      No, not at all.

   10. JLJill Lepore22:54

       [laughs]

   11. CNCasey Newton22:55

       It's just, it's... We, we always just get really excited when people start paying attention, you know, to the stuff that we care the most about.

   12. JLJill Lepore23:01

       Yeah. [laughs] I'm just teasing.

   13. KRKevin Roose23:03

       [laughs]

   14. JLJill Lepore23:03

       Uh, you know, it is a weird book for me to have written. All my friends are like, "Oh, man, don't be writing about that. It's gonna be so grim."

   15. KRKevin Roose23:11

       [laughs]

   16. JLJill Lepore23:12

       Um, yeah, and it was kind of grim. Um, you know, I write, I'm mainly a pol- an American political historian. I write about a lot of other stuff, like I'll, I'll write about anything. Um, and I've written a lot of pieces for The New Yorker about the history of technology. I've, like, been writing for the magazine for 20 years now, and, like, occasionally I'll do that. And I teach a lot about... I teach a class on the history of technology, so, like, I've just conceptually been thinking about this stuff a lot. And then last summer I was asked to do these lectures at Yale that are called the Tanner Lectures on Human Values, and I was just... Like, the accumulation that I think people feel of the dehumanizing of the moment-

   17. KRKevin Roose23:51

       Mm

   18. JLJill Lepore23:51

       ... that we're in, like, like, you know, you call to ask about your pet food delivery, and you're talking to a computer, and who decided this is a way we should be living? Do you know what I mean?

   19. KRKevin Roose24:03

       Mm-hmm.

   20. JLJill Lepore24:03

       Like that question. So this is a long way around my sort of... I decided I really kinda wanted to write a, um, a short book, that these lectures would be a short book about how it is that we have ceded so much of the functions of modern liberal democracy to machines that are automated, often now more recently driven by artificial intelligence, run by private corporations, and without so much as a scream beyond the emoji.

   21. KRKevin Roose24:35

       Yeah. And Jill, I wanna dig into this concept of the artificial state, which is sort of the, the thematic, um, emblem of your book. I, I wanna understand what you mean when you say the artificial state. Uh, you at one point compare it to the idea of the factory farming of humans. You also, uh, define it as being the rule of humans by machines manufactured by corporations, and you also have a number of, uh, passages in which you talk about how this Artificial state is, is not a real state in the sense that it's, you know, striving for some kind of self-governance or internal organization, but that it is sort of supplanting the role in people's lives that their own actual states used to have, their, their local cities and, and states and, and federal governments. So can you just sort of sketch the basic idea of the artificial state as clearly as possible?

   22. JLJill Lepore25:23

       Yeah. The artificial state is a, um, an emerging successor to the liberal democratic nation state in which, uh, government is conducted not by the consent of people, but by machines that are making decisions, um, and those machines are owned by corporations. So we don't live in the artificial state. It is something that I think is being built, but I think it is also, and this is an important part of my claim, it's also a fantasy that certain people have, that they believe their power to be above that of the nation state. You know, the rhetoric is always... There's, there's always like a, like a, a footnote or a paragraph to, "We do believe that people have control over their own lives, and they elect governments to make decisions for them. But actually, we are in charge of the future of civilization-"

   23. KRKevin Roose26:10

       [laughs]

   24. JLJill Lepore26:10

       "... and the future of humanity, and the whole world's destiny. The destiny of the galaxy lies in our hands, and you better trust us to do what's..." Like, like, the, it, the rhetoric of, that comes out of this particular historical moment is, is really about the evanescence of, of modern liberal democracy, constitutional democracy.

   25. KRKevin Roose26:29

       Yeah. Out, out here in the Bay Area, in the maw of Silicon Valley, there is sort of a, a cartoonish caricature of the East Coast intellectual who greets all new technology and progress with scorn and mockery and dismissal and can't be bothered to get on board with the revolution, and so just kind of sits in their ivory tower and laments the changing culture in front of them. Uh, you've been accused of being part of that tradition. I'm, I'm curious if, uh... what, what your take on that is and what we are missing out here in the bubble that people like you are maybe better positioned to capture.

   26. JLJill Lepore27:07

       I remember, um, years ago, I went to Stanford. I was being recruited to teach at Stanford, and we went out to dinner with, with the, like, recruiting faculty, and the people at the next table over were some, you know, young-ish, very, um, earnest young coders. And they were talking about the homeless problem of San Francisco and how they were gonna start a school for coding for the homeless.

   27. KRKevin Roose27:30

       Oh, no.

   28. JLJill Lepore27:31

       And, uh, and I was like, "I don't think we can move here because I can't" [laughs]

   29. KRKevin Roose27:37

       [laughs]

   30. JLJill Lepore27:38

       "I can- I c- I just can't. I can't..." Like, they were very sweet. Like, I really... They were like my students. Like, a lot of my... I... Tons of my students, of course, go to work in Silicon Valley. Like, Harvard does this huge recruitment thing. Um, and they're recruited with the promise, like, "You're going there to make the world a better place." Um, and I hear from them a few years later, and they're like, "Actually, that's not really what we were doing." Like, it's a good recruitment message. I think there is a kind of sociological issue with Silicon Valley, which is it is opposed to the idea of critique. Um, things are just supposed to c- continue to move ahead, and there's not... Like, the, the very idea of looking backward to assess what something has been and has done, or even to look backward to say, "Is the thing we're do- Is there an antecedent for the thing we're proposing to doing that might suggest a better way of doing it or might suggest we might not wanna do it?"
3. 44:48 – 59:36

   Why Is Google buying data from Spirit Airlines?

   1. KRKevin Roose44:48

      you see this story about Google buying the data of Spirit Airlines?

   2. CNCasey Newton44:51

      I sure did.

   3. KRKevin Roose44:52

      This was one of the most fascinating stories I've seen in recent weeks, and it led me down this incredible rabbit hole of thinking about training data and the new era of data collection we are in. So I thought we should use this Spirit story as an occasion to just sort of catch each other up on the state of AI training data in general, because it is fascinating and, I think, underappreciated.

   4. CNCasey Newton45:15

      And it sounds like the perfect frame, Kevin, for our new segment, Train of Thought.

   5. KRKevin Roose45:19

      [laughs] I love that we're just starting new segments every week until the show ends.

   6. CNCasey Newton45:24

      We are.

   7. KRKevin Roose45:25

      It's time for the first and last installment of our new segment, Train of Thought.

   8. CNCasey Newton45:29

      This is kind of the caboose, as it were.

   9. KRKevin Roose45:31

      Yes.

   10. CNCasey Newton45:31

       Interesting that we have two train-related segments on the show.

   11. KRKevin Roose45:33

       [laughs] [upbeat music] So the reason that we wanted to do this segment today is because there was a very strange story that piqued our attention over the past week involving the defunct airline, Spirit Airlines.

   12. CNCasey Newton45:57

       Hands down the worst airline of all time.

   13. KRKevin Roose45:58

       [laughs] Yeah.

   14. CNCasey Newton45:59

       I don't even know who else is in the conversation. And yes, I did fly it one time.

   15. KRKevin Roose46:02

       This week, a bankruptcy court auctioned off Spirit Airlines' internal corporate data. Google won the bid, offering to pay $10 million for this data, beating out a seven and a half million dollar offer from the AI data company, Mercor.

   16. CNCasey Newton46:18

       $10 million, Kevin. What did Google get for that price?

   17. KRKevin Roose46:22

       So this deal apparently included 100 million emails, 500 million Microsoft Teams chats and other conversations, 7.5 billion passenger transaction records dating back to 2008, and 30 million lines of Spirit's internal source code and other documentation.

   18. CNCasey Newton46:40

       Well, I would consider Spirit's internal source code malware, but everything else sounds interesting. So what do we think Google is gonna do with this data?

   19. KRKevin Roose46:47

       This was where my head went after I saw this, because I thought, why is Google, guardian of the world's information, presumably the possessors of vastly more data for training AI models than any company in the world, um, what are they doing paying $10 million for this bankrupt airline's data? And that sent me down, like, a really fascinating rabbit hole of this, uh, traini- this world of training data and training environments, um, that all of the AI companies now are investing really heavily into.

   20. CNCasey Newton47:16

       Well, tell us what you've learned.

   21. KRKevin Roose47:18

       So Casey, did you know that there is a company that auctions off the Slack histories and email histories of defunct companies?

   22. CNCasey Newton47:25

       Do- I, I'm surprised to learn that there is a market for that.

   23. KRKevin Roose47:28

       Yeah, so there is a company, Simple Closing, whose whole business used to be sort of helping failed startups wind down, but now they have the-

   24. CNCasey Newton47:34

       Wait, these, these guys are like the undertakers of Silicon Valley.

   25. KRKevin Roose47:37

       Yes, exactly.

   26. CNCasey Newton47:38

       Their, their, their corporate logo is just, like, the Grim Reaper.

   27. KRKevin Roose47:41

       [laughs] Yes.

   28. CNCasey Newton47:42

       Yeah.

   29. KRKevin Roose47:42

       Yes, if these guys show up at your office or you get a call from them, it's a very bad day for, for [laughs] your company. Um, so basically they were sort of like, you know, helping do the orderly wind-downs of these things, but then they realized, oh, there's actually a market for the data from these dead startups, and so they start selling it to AI companies. And as of April of this year, they had done almost 100 deals ranging from roughly $10,000 to $100,000 per company.

   30. CNCasey Newton48:09

       And again, I just wanna know, like, what happens when the buyer actually gets a hold of the data? Like, where does it go? And does it violate my HIPAA rights?

Episode duration: 59:38