The Watchdogs of AGI — Rune Kvist of AI Underwriting Company

1. 0:00 – 1:29

   Hook

   1. RKRune Kvist0:00

      If, if you think fable concerns are bad, like see when Waymo hits a dog, NASA people lose their mind. Imagine when first robot knocks off a toddler off a kitchen table. You're gonna see some real strict liability. So physical AI, the level of stringency just goes up and up and up and up. Um, so that's kinda like the big picture, agents, models, robotics. As the technology progresses, as agents get longer horizons, new types of failure modes will emerge. That will also bring in just new kinds of ways to create value, but also more risk surface. You'll start to see true agent-to-agent interactions that are not mediated by humans. There's gonna be a bunch of interesting questions. You're basically gonna need a new legal system. How do they build trusts amongst each other?

   2. SWswyx0:42

      [on-hold music] Okay, we're in the studio with Rune from AIUC, AI Underwriting Company, uh, with our trusty co-host Vibhu. Welcome.

   3. RKRune Kvist0:53

      Thank you. Thanks for having me. Thank you.

   4. SWswyx0:54

      Uh, what are you announcing today?

   5. RKRune Kvist0:55

      We have raised $40M, uh, by Ribbit Capital and First Monarch.

   6. SWswyx0:59

      When you first came to my attention when, uh, Nat and Dan invested in you guys, is the story like pretty much th- the same? Like what, what, what are you today what you thought you were back then?

   7. RKRune Kvist1:09

      When we raised our seed round, we had a hypothesis that at some point risk was gonna hold down adoption. At that point in time, that felt kinda hy- hypothetical, and I think that ha- that is now over. Clearly the moment is now. With, uh, Mythos and with Fable, it's pretty obvious that literally the binding

2. 1:29 – 2:36

   AIUC’s $40M Round and the Risk Bottleneck for AI

   1. RKRune Kvist1:29

      constraint on adoption is risk. And so for us it feels like this is a natural continuation of the same hypothesis, but where previously it was speculation, now it feels like fact.

   2. SWswyx1:37

      And let's get the list of the customers that you, um, highlighting as part of your Series A.

   3. RKRune Kvist1:41

      Totally. Yeah, so we are now working with folks like Cursor, Harvey, Lovable, ElevenLabs.

   4. SWswyx1:48

      Yeah. Amazing. Congrats.

   5. RKRune Kvist1:49

      Thank you.

   6. SWswyx1:50

      So you were famously one of the first, like the first, uh, hire at An- Anthropic for GTM and, uh, product. I'm just kinda curious, like wh- what was your path into AI? Just-

   7. RKRune Kvist2:01

      Yeah

   8. SWswyx2:01

      ... recap.

   9. RKRune Kvist2:02

      Late 2021, I sold a company, my first company, an ed tech company. I had a bit of time to think about what was next. I came across the Scaling Laws paper and that just struck me like lightning. I was just like, "This is a big idea." In short, the Scaling Laws paper just says the bigger the model, the smarter the model [laughs] . And the-

   10. SWswyx2:21

       This is the Kaplan one, not the Chinchilla one.

   11. RKRune Kvist2:23

       Exactly, the Kaplan one.

   12. SWswyx2:25

       Yeah.

   13. RKRune Kvist2:25

       And the important thing that clicked for me there was, oh, now capital will understand this. If you put in more money, you get more money out, and so that

3. 2:36 – 9:27

   From Scaling Laws to Early Anthropic

   1. RKRune Kvist2:36

      will kick off a hype cycle. Uh, and so you'll actually kind of-- you'll get a sense of predictable returns, which is in fact what's played out. And so I just packed my bags. I'd never been to San Francisco. I'd never... I just packed my bags, flew out here to find the people who had written it. Uh, and at the time they had just started a small lab called Anthropic.

   2. SWswyx2:53

      [laughs]

   3. RKRune Kvist2:53

      Uh, there was like 40 people at the time or so. Drank a bunch of coffee until I eventually got introduced to Dario. And at the time they were wrestling with some of these questions of like, should we deploy our models? Should we make revenue? How should we engage with the rest of the world? They'd just broken off from OpenAI, uh, and it's been publicly reported that they were kind of concerned with how they were dealing with deployment, so they were wrestling with some of those questions. At that-- this point, this is like early fog of war, like early 2022. The sexiest product at the time was like Jasper [laughs] . Like there's, there's nothing out there. So where is value gonna accrue? Uh, what are gonna be the different parts of the stack, were all open questions.

   4. SWswyx3:31

      I wanna highlight to people, you ask these questions because you have a PPE background.

   5. RKRune Kvist3:35

      Yes.

   6. SWswyx3:35

      Uh, I actually was in, in Singapore in one of the sort of feeder programs for prepping people for PPE. So I had a tutor. We learned, uh, you know, philosophy and politics and economics. But like I think your kind-- Like machine learning people who read the neural, uh, Scaling Laws paper would not necessarily draw the same conclusions that you did, whereas any capitalist would read that and go, "Holy shit." [laughing]

   7. RKRune Kvist4:01

      [laughing] Correct.

   8. SWswyx4:03

      Right? Like-

   9. RKRune Kvist4:04

      Yes

   10. SWswyx4:04

       ... uh, who tipped you on to that paper? 'Cause it's not a paper that you normally read, right, like in your circles.

   11. RKRune Kvist4:09

       Yeah. I think I'd actually, uh, ever since AlphaGo had had some appreciation that AI was a big deal-

   12. SWswyx4:19

       Yeah

   13. RKRune Kvist4:19

       ... uh, but it kind of felt... It raised all these kind of interesting philos- philosophical questions, but it was kinda not clear from afar where exactly that would go. But it was obvious enough that it was like this is gonna be a big thing if we find the kind of right mechanism to kind of get the techno capital machine to work on this. But it was just not clear. And so I think there was some way in which like that became obvious and also it wasn't as obvious at the time than, than it is now, right? Like it was just like, "Wow, this is so interesting," but it still felt coming from kind of a philosophy and economics background, it felt like if this turns out to be true, you're gonna be wrestling with all of the big questions in society. Everything you've learned about politics gets thrown out of the window. Everything you've learned about economics at least gets challenged. And so what felt interesting was to be at that frontier that has just ramifications across everything. Um, so that's, that's why I th- I, I thought it sought it out.

   14. SWswyx5:15

       I mean, clearly really good insight. For people who, for who don't know, PP- the PPE program is like where prime ministers are born. So then you end up meeting Jared.

   15. RKRune Kvist5:24

       Yep. Uh, first Dario, yeah.

   16. SWswyx5:26

       Yeah. Uh, well, I mean, like so did you get extra insights from talking with them that you didn't get from your original hypothesis?

   17. RKRune Kvist5:33

       If you read the Scaling Laws paper you get this like very vague sketch of like, wow, this seems kind of important. There are some lines in a chart that seems kind of important. Um, and what I think the team at Anthropic had thought more about than anyone was like, "What are the implications of this if you really play this out?" And back then they had, uh, kind of vision documents for what the world would look like in 2026, and they were kind of in vivid detail playing out how much compute is gonna be needed. What is the CapEx gonna look like? What are gonna be some of the kind of societal concerns, but also what is the amount of economic value coming out here? And so it kind of felt like they held a crystal ball that in hindsight would turn out to just be dramatically correct. And they weren't holding it like they were obviously correct. They were just like, "Take this hypothesis-

   18. VIVibhu6:19

       Yeah

   19. RKRune Kvist6:19

       ... really, really seriously."

   20. VIVibhu6:21

       Think it through, yeah.

   21. RKRune Kvist6:21

       And think it through in the same way the kind of situational awareness that is now-

   22. VIVibhu6:26

       Across the street

   23. RKRune Kvist6:26

       ... across the street.

   24. VIVibhu6:27

       Your post, yeah.

   25. RKRune Kvist6:28

       Uh, uh-

   26. VIVibhu6:29

       Oh my God, we're all in the same-

   27. RKRune Kvist6:30

       All literally across the street

   28. VIVibhu6:31

       ... like same one square mile of like [laughs]

   29. RKRune Kvist6:32

       Correct.

   30. VIVibhu6:33

       [laughs]
4. 9:27 – 13:48

   Why Trust, Not Capability, Could Limit AI Adoption

   1. RKRune Kvist9:27

      or trust. That problem is, uh, general. The reason why right now Fable is not open for access is not because it's not a good model. It's because it's a very good model. It's just hard to make promises about what it will or will not do. And this problem gets worse as AI gets better. Basically, more intelligent AI can be more autonomous. That's more valuable, but also the risk surface grows. And so the... what Waymo illustrates is that unless you build the confidence infrastructure to make promises about AI, or at least bring light to the risks, you grind adoption to a halt. Governments, banks, hospitals, militaries need to have some sense of what AI will and will not do to be able to operate for them to incorporate it. And that's the problem that we're trying to solve. Now, why standards and insurance? Uh-

   2. VIVibhu10:21

      [laughs]

   3. RKRune Kvist10:22

      If you trace this problem back through history, every technology wave has had some version of this problem. So if you go back to, like, year 1900, electricity comes out-

   4. VIVibhu10:30

      Ben Franklin

   5. RKRune Kvist10:30

      ... cars burn down. Uh, sorry, houses burn down, lots of people die. 1930s, cars are a big deal, kill lots of people. '50s, private nuclear energy is a big deal, poses big risks. In each of those instances, the market runs ahead of regulation to create confidence infrastructure because that's required to make go/no-go decisions. That is required for adoption, and the market fundamentally wants adoption. And in all of those instances, common blueprint emerges between standards and insurance. The reason it's these two components is standards kind of provide the rules of the road, and they also specify, like, what are the tests that need to be run, so we can get a sense of how high the risk is. So taking the case of cars as like a car crash, great, everyone, they inform your insurance pricing today, they inform your purchasing decisions, et cetera. That's basically the risk framework. The insurers are important because they pick up the bill, so they are the private institution that is most on the side of... uh, that's best incentivized to quantify the risks truthfully and then figure out all the ways to reduce the risk, 'cause that increases their profit. So they're basically-- they help shape the incentives, and these two work really well in unison. Now, how does that show up as a company? Well, one of the things that was obvious even-- or starting to become obvious even a couple years ago was that frontier companies, some are our customers today, like Cursor, uh, Sierra, ElevenLabs, Harvey, were gonna have a very easy time selling a ... pilot to a bank. The, like, the, the demo just sells itself. It's magic. But bringing that through, if you wanna do a wall-to-wall rollout at a bank or a hospital, uh, you have to go through the risk process. These banks have no idea even which questions to ask, let alone which answers are sufficient, let alone, like, how do they go and test whether these agents actually work the way they're supposed to. And so they had, they had this problem of, like, what can we say to earn the trust? And we think there's, like, a golden sentence that goes something like, "Hey, I hear you're really worried about hallucinations or jailbreaks," or whatever it may be. "We've had an independent third party test us against the gold standard. We passed with flying colors. And as a vote of confidence, the world's most conservative insurers have looked at the data and are willing to take some of the risk onto their balance sheet."

   6. VIVibhu12:49

      Yeah.

   7. RKRune Kvist12:50

      So if something does go wrong-

   8. VIVibhu12:50

      There's money behind it, yeah.

   9. RKRune Kvist12:52

      Exactly. So that's kind of like the link between all of this. We can, we can get into some of the, the hard parts related to, to the technical testing, which is, I think, the crux of the matter. Uh, but I'll pause there.

   10. VIVibhu13:02

       How did you and Rajiv come together? Uh, this... There's always, like, you s- you come across very confident and, you know, you're, you're... We're announcing your Series A and all these things. But I wanna see, like, the early initial stages of, like, idea formation.

   11. RKRune Kvist13:14

       Yeah. Rajiv is actually my soon-to-be brother-in-law.

   12. VIVibhu13:18

       Oh. [laughs]

   13. RKRune Kvist13:19

       Right? So I'm actually, uh, in a week and a half getting married to Rajiv's sister. [laughs]

   14. VIVibhu13:24

       [laughs] Okay, now you're tight.

   15. RKRune Kvist13:27

       Exactly.

   16. VIVibhu13:27

       Now you know.

   17. RKRune Kvist13:28

       Uh, so, um, Rajiv and I have known each other for a decade. Uh, funny story, uh, I met both Rajiv and his sister, Henna, uh, at the same time when Henna and I were interns at McKinsey in London, and Rajiv was assigned as my mentor. Uh-

   18. VIVibhu13:42

       Wow

   19. RKRune Kvist13:42

       ... so met them at the same time. For the longest time, it was not obvious that we were necessarily gonna work together. Uh, I was in

5. 13:48 – 20:21

   Founding AIUC and Building AIUC-1

   1. RKRune Kvist13:48

      startups. He was, uh, an insurance partner at McKinsey. Three or four years ago, I think Henna convinced him that AI was gonna be a really big thing. And so he quit his job, cushy partner job at McKinsey in London, packed his bags, flew to San Francisco, and ended up joining Meter. You guys are probably online enough-

   2. VIVibhu14:07

      CEO.

   3. RKRune Kvist14:07

      Exactly.

   4. VIVibhu14:07

      Uh, we've, we've, uh, we've interviewed them

   5. RKRune Kvist14:08

      We've seen the tw- the, the chart of, uh, uh, the horizons of the tasks that agents can take on is, is doubling extremely fast. So he was CEO there, led their partnerships with Anthropic and OpenAI to test their models before release, but also working closely with the US and UK government, uh, to figure out, like, how do you know whether a model can be released? And in some ways, that's, like, the perfect background. He's spent a lot of time in insurance, knows that world, spent a lot of time with frontier testing of models. And so when I was bumbling around this idea space, starting with some of the ideas we talked about related to Waymo, as soon as we got into the content, we were both like, "Oh, this would be an amazing business to, to build together." This is like wrestling with the problem that we both think is the most important in the world from a market angle, which is kind of our intuitions is that the market can do a lot, and the faster AI moves, the harder it is for government to solve some of these problems. And then took a little bit of time to work through what is it like to work with family.

   6. VIVibhu15:10

      [laughs]

   7. RKRune Kvist15:10

      Uh, and, uh-

   8. VIVibhu15:12

      'Cause you were already dating at the time or...?

   9. RKRune Kvist15:14

      Uh, yeah, yeah. I mean-

   10. VIVibhu15:15

       Okay.

   11. RKRune Kvist15:15

       Yeah, ex-ex-exactly.

   12. VIVibhu15:16

       Yeah, yeah.

   13. RKRune Kvist15:17

       Already back then it was-

   14. VIVibhu15:18

       Yeah

   15. RKRune Kvist15:18

       ... we felt like we were a family.

   16. VIVibhu15:19

       Nice.

   17. RKRune Kvist15:19

       And so starting a business together felt like kind of a big step. And, uh, here we are with just immense amounts of trust.

   18. VIVibhu15:26

       Yeah. So now you're a company of how big? How big are you guys now?

   19. RKRune Kvist15:29

       There is just 20 of us now.

   20. VIVibhu15:30

       20 of you guys now have Series A, and you have your first certification out, the AIUC v1. Um, let's bring up the certification. So this is the agent certification, right? What goes into the process? I have, like, two questions here. One is, walk us through the certification, and two is, what is the process for a company to get certified, you know?

   21. RKRune Kvist15:51

       Great. As it says right at the top, uh, AI v1 is the standard for agent security, safety, and reliability. The fundamental design principle is take all of the concerns that slow down adoption, so all the questions, all the fears that keep, uh, security leaders in the Fortune 1000 up at night, and put them into one comprehensive framework. Uh, that's what you'll see there. You can see the six categories. Two, you wanna ground all of this in technical testing. So one of the concerns with security standards that often feel kind of like theater paperwork is that they're not actually ground out in does any of this work? Does any of this matter? And so we had a conviction from early on that that was gonna be the kind of crux, was to pass this you must get tested every quarter. Basically, run thousands of simulations to see, well, so can it actually be jailbroken? How hard is it to jailbreak? How often does it hallucinate? How often does it leak data, et cetera. And then the last, uh, core idea here, uh, if you scroll up to the top here, is to refresh it quarterly.

   22. VIVibhu16:50

       Mm.

   23. RKRune Kvist16:51

       So the core trait of AI is that it moves extremely fast. Whatever concerns we're discussing today were not the same ones three months ago, and this will keep changing. Typically, standards update on, like, a decade cycle.

   24. VIVibhu17:02

       [laughs]

   25. RKRune Kvist17:03

       Is obviously not gonna work. But the question is kind of how do you update it? And the core thing here was to basically get the risk leaders of the Fortune 1000 around the table. So if you go over to the left here-

   26. VIVibhu17:15

       Yeah

   27. RKRune Kvist17:15

       ... you'll see AI v1 Consortium. The consortium is a group of risk leaders who run real banks, real hospitals, real critical infrastructure, who are facing these challenges every day. And we meet with these folks twice a quarter and hear what's top of mind. What is keeping them up at night? There's tremendous amount of desire for that conversation, and then we operationalize that into a specific standard that gets into... And actually, we can go into and look at what is-

   28. VIVibhu17:38

       Yeah

   29. RKRune Kvist17:38

       ... what even is the standard. So if we go back to introduction, up there to the left, scroll up a little bit to the wheel, click into Reliability. So if we take something like hallucinations. Uh, hallucinations system reliability, there is a number of requirements here. If you go into the top one, prevent hallucinated requirements, uh, hallucinated outputs. This is one particular requirement. This is a technical control. Basically, we want some kind of groundedness filter. The first thing you see here is what's called a crosswalk. So- Everyone and their grandmother has put out a framework, very high-level framework for what are the AI risks

   30. VIVibhu18:10

       This is basically your competition, but-
6. 20:21 – 26:55

   How AI Agents Are Audited and Stress-Tested

   1. VIVibhu20:21

      like you must implement some guardrails. Mm-hmm.

   2. RKRune Kvist20:25

      Two, there are test controls. So you must have an independent third party go and run some tests against you. I'll show you one of those in a second. And then three, there are policy controls. For example, you must have a person whose name is on the line when you guys fuck up.

   3. VIVibhu20:39

      Right.

   4. RKRune Kvist20:39

      And you must have a plan for how you tell your customers and how you engage with them. They're kind of more traditional standard type stuff.

   5. VIVibhu20:46

      I see.

   6. RKRune Kvist20:46

      So in this particular instance, we just check whether they in fact have a groundedness filter. So we will partner with an auditor. So we partner with auditors like KPMG or like Schellman who go in and do the thing auditors do, which is to check the evidence. In this case, that might be a screenshot, it might be part of the code that they need to review to see that it actually... just that it exists.

   7. VIVibhu21:04

      Ah, okay.

   8. RKRune Kvist21:05

      And then the second thing is-

   9. VIVibhu21:05

      So you're not testing the effectiveness of it.

   10. RKRune Kvist21:07

       That's the second thing.

   11. VIVibhu21:08

       Yeah.

   12. RKRune Kvist21:08

       So if you go down to the third party testing for hallucinations out on the left, that's basically the next requirement. This is where we test how well does it actually work.

   13. VIVibhu21:15

       Okay. And is it you testing or the auditor?

   14. RKRune Kvist21:17

       We test them.

   15. VIVibhu21:18

       Ah.

   16. RKRune Kvist21:18

       We test them.

   17. VIVibhu21:19

       That's a lot of work. [laughs]

   18. RKRune Kvist21:20

       That's-

   19. VIVibhu21:20

       How long does testing take? So if I wanna get certified, just-

   20. RKRune Kvist21:23

       Yeah

   21. VIVibhu21:23

       ... how long does the end-to-end roughly take?

   22. RKRune Kvist21:25

       Yeah. The end-to-end, uh, almost always is dependent on the... our customers need to learn-

   23. VIVibhu21:30

       Yeah, yeah

   24. RKRune Kvist21:30

       ... something for us. It takes somewhere between like three to 10 weeks-

   25. VIVibhu21:35

       Ugh

   26. RKRune Kvist21:35

       ... depending on how up to snuff they already are. So some people show up to us with like extremely rigorous security programs. When we test them, it works extremely well. We can get that done very quick. Some people come to us and they're not that far along. We give them kind of the spec that they need to build towards, and then their security teams and engineers get to work and build to meet the standard. Uh, the testing itself typically takes a couple of weeks, including the time for them to remediate. Often we'll find something that we cannot pass, where, "Hey, this is actually just not up to the standard. You, you won't pass the standard." And then they will need to go and implement additional safeguards or additional remediation that makes them more robust so that they can actually kind of hand on heart look at their customers in the eyes and say like-

   27. VIVibhu22:15

       [laughs]

   28. RKRune Kvist22:16

       ... "Hey, we've done truly our very best."

   29. VIVibhu22:18

       And they're certified for a year and have quarterly updates?

   30. RKRune Kvist22:21

       Correct. Yeah.
7. 26:55 – 35:01

   Frontier Models, Government, and the AI Trust Gap

   1. RKRune Kvist26:55

      You have some people that are on the hook if something goes wrong. In the case of agents, it's often the security leaders in the enterprise. In this case, it's the government. They don't... haven't necessarily spent their entire lives thinking about what are the new risks that come here? What is the kind of data you might be looking for? How might you test that? But they do have to make sure that their concerns are addressed. You have some frontier AI companies that are deeply technical. They know a lot about the risks, but they fundamentally have an incentive to not always be truthful. So you have a trust gap between the government and the labs. And in every other industry, you end up with some kind of body sitting between, a neutral third party sitting between those people. There's no other industry where you allow people to audit themselves. So there's gonna be a need for a third party that can take the rigor of the labs to run frontier technical evals, but can also speak legible trust in the way that the government trusts PwC to go and run financial audits, and they know that they output audit reports in a way that's consistent, that's easy to read, that's factual, that's, uh, trustworthy. Those two things need to be brought together, and what we've learnt from our work with agents is that if you want those... that communication between those two parties to be smooth, there has to be one common standard that ev- that is public, that people can go and inspect. What are the risks that matter? Within each of these risks, what are the kind of threat models that you're really looking for? You need to specify for each of those risks what are the guardrails that need to be in place and what are the tests that you need to run to see whether those guardrails are effective. And then you need to go and run audits that are con-- technical audits that are consistent. So if you're trying to bring trust, it's extremely important that you methodically work your way through the risks. You can't send one researcher in and say like, "Come back with whatever you find." You need to be able to explain exactly what you did, exactly what you tried, exactly what you did not try, and therefore the kinds of promises you can and cannot make at the end of it. I think of Fable as a direct symptom of this problem, that the government was told that there's a risk. The government may struggle to assess just how big that risk is. They call Anthropic, and Anthropic is trying to tell them, "Hey, actually every model can be jailbroken."

   2. VIVibhu29:11

      [laughs] That's not what you wanna hear, right?

   3. RKRune Kvist29:15

      As the government, that might be hard to trust.

   4. VIVibhu29:17

      [laughs]

   5. RKRune Kvist29:19

      And we think that a broker is the k- most natural solution. In other markets you, you see something like, uh, in financial markets you see Moody's. Moody's goes in and they look at a bond, and they output a rating. They say like, "Here's the evidence we found. Here's the rating." We don't decide whether anyone should buy this bond or not buy this bond. Well, that depends on their risk appetite. But we do provide this common information layer that everyone can rely on. In the case of Moody's, the government, uh, points to them and say, "Hey, pension funds, you should probably really take care. You shouldn't risk your pensioners' money, so you can only invest in AAA-rated bonds." That means that now the government doesn't have to staff thousands of financial technical experts to rerun forecasts every week to see whether things are correctly rated. They get to point to some neutral third party. Uh, so my hypothesis is, my hunch is that you will see a third party that sits between the government and the labs, and it could either be the government builds it themselves- So something like CAISI was set up to, to do exactly this. And the question is-

   6. VIVibhu30:27

      Sorry, I'm not familiar with CAISI

   7. RKRune Kvist30:28

      ... CAISI is the Center for AI Standards and Innovation.

   8. VIVibhu30:32

      Okay.

   9. RKRune Kvist30:33

      I won't get into the details, but it's a subbody of NIST that typically sets standards. Uh, so it's basically a government body that has AI experts. Yeah, exactly. Exactly.

   10. VIVibhu30:42

       Very low, very low-key.

   11. RKRune Kvist30:42

       Ex- very low-key.

   12. VIVibhu30:43

       I, I think, you know, it's one of those things where when you just sit back and listen... uh, look at it, like, is there enough technical expertise in the government to measure, test these things right now? Probably not, right? And Fable is a result of, okay, we've had to scale back and pause things, but-

   13. RKRune Kvist31:00

       Yeah. And they have, they have excellent people, uh, but they have an extraordinarily small budget compared to the-

   14. VIVibhu31:07

       [laughs]

   15. RKRune Kvist31:07

       ... scale of the challenge that's ahead of us, and I think they have a role to play. The question is kind of like, who does what? Uh, we have now outlined the jobs to be done, and they're quite extensive. Every model release, there is an astounding... Given that they, they take in any input, their risk surface is astounding. And so the question is really, what can only the government do, and what can the market provide here that can keep up with the pace as AI risk changes? Our perspective is that also at the model layer, the risks that people care about today are not the same ones they cared about three months ago. So the pace of legislation is too slow to deal with pinpointing the risks here. And so we think there's a lot that the market can do to surface timely information. Ultimately, there is a bunch of policy decisions here. Is the national security risks of a model too high?

   16. VIVibhu31:55

       Yeah.

   17. RKRune Kvist31:55

       That's a political answer. Uh, but what we wanna make sure is that the process that produces this risk information is compatible with very fast innovation. So you don't wanna... This is not a question of like, can you slow things down? Can you keep, uh, the models locked up until... for months on end until everyone can make a guarantee? Uh, but it is just can you, in the time it... Given that the US is competing with China on releasing models, can you insert risk information that allows the government to, like, make rapid decisions on some of these questions? Balancing that trade-off between failing to adopt AI is gonna put us at risk, but also reckless adoption is gonna put us at risk, and that's a very kind of fine balance that they're gonna need, like, a lot of high-quality intelligence to, to make.

   18. VIVibhu32:38

       Just a side mention because you mentioned Chinese models. Any specific concerns that you're hearing from your CISOs about that? Uh, 'cause I guess it's free, but...

   19. RKRune Kvist32:48

       CISOs have a bunch of concerns around data flows in general that they're really concerned about. So, so a lot of questions are like, uh, if these models are Chinese, where does that, where does that data go? I think a lot of those can be addressed, but they, they come up often.

   20. VIVibhu33:01

       I mean, they understand they're running on American GPUs. [laughs]

   21. RKRune Kvist33:04

       Some of them, some of them understand that they're running on American GPUs.

   22. VIVibhu33:06

       They're not, like, phoning home every time you, like, call home. [laughs]

   23. RKRune Kvist33:09

       No. Uh, a year ago, there was not a lot of understanding of this. I actually think, uh, you're seeing the security leaders becoming kinda AI literate at a blistering pace, and you're actually also seeing my Twitter timeline that's very AI-pilled and my LinkedIn feed that used to not at all be AI-pilled kinda converge. They're both talking about Fable.

   24. VIVibhu33:28

       Right. Right. Yeah, that's true.

   25. RKRune Kvist33:29

       They are both talking about whether you can prevent models from being jailbroken, jailbroken these days.

   26. VIVibhu33:34

       Yeah.

   27. RKRune Kvist33:35

       Uh, s- like national security risks are kind of... That, that conversation is actually emerging. Other than that, I think you mostly see a kind of, uh... There's no concerns with any particular model or any particular model output, but there's a general nervousness of having critical infrastructure run on models that are not produced in America by Americans where the American government has control.

   28. VIVibhu33:57

       That doesn't necessarily show up in your framework d- directly, or it might. I don't know.

   29. RKRune Kvist34:01

       There's a bit of stuff in there actually on the, like, the provenance of the models and disclosing that. But I think there's a bunch of use cases where running a Chinese open source model is just the, the best solution.

   30. VIVibhu34:10

       Yeah.
8. 35:01 – 39:43

   Cyber, Child Safety, and AI-Enabled Biological Risk

   1. VIVibhu35:01

      Mm.

   2. RKRune Kvist35:01

      And specifically whether models will help adversaries produce biological weapons and making that extremely cheap, extremely accessible, producing... making the chance of another COVID or worse pandemic. COVID was not engineered to be bad-

   3. VIVibhu35:17

      Yeah

   4. RKRune Kvist35:17

      ... as if you were trying to do that. So I think those are some of the risks that are coming down the pipeline. Uh, I think one other thing to just note is that agents are kind of deliberately narrow. So, like, when a frontier agent company puts a chatbot that interacts with customers, they've really tried to narrow the topics it's interested in talking about, such that if you ask it, like, "What do you think of the president?" It will just decline. Which means that the kind of risk area is somewhat smaller. For models, it is infinite, and so there's not a single expert out there who can competently evaluate the risks of cyberattacks and 15-year-olds having month-long conversations with a chatbot and seeing whether it will in fact recommend suicide or something horrendous like that, and can evaluate the risks that terrorists can use AI to produce bioweapons. The risk surface is just too big, and so the central challenge actually becomes how do you get those subject matter experts to work t- within a c- one coherent framework that outputs one coherent report and rating that the world can go and inspect? 'Cause that global perspective is central, but there's not a single organization today that could produce that.

   5. SWswyx36:30

      And you would be the presumptive one when you put out your model standards

   6. RKRune Kvist36:34

      We think there can be one company that can, with a consortium of experts, build one coherent standard. I think we've shown that across all of the enterprise risks today. We think it could be one company that could, with a consortium, specify the audit rules, basically like the inputs and outputs that all these technical experts need. What access do they need? How should they treat infras-infrasecurity? They can look at whether the eval-evals are well-produced without necessarily being able to say, "Hey, this is a threat or not a threat," but overall evaluating whether the evals are good, well-constructed. That set of audit rules that basically becomes the interface for all these experts, we think one clearing house could put together. To be clear, when I say one company, I think of it as one company coordinating lots of this in the same way that when we saw our consortium, it's not like we say we have all the answers on agent security. What we say is we are taking on the role of eliciting all of the concerns and being the secretary that puts it together and runs a tight house such that the standard updates lockstep every quarter and that the audit reports that come out, in this case, hundred-page audit reports, uniform and crisp and clear all to the level of detail that is required for executives that need to make a g-clear go, no-go decision. So that's kinda the role that we think we might play.

   7. SWswyx37:54

      I think in many ways you're performing the role that OWASP used to do there, and you said, like, you know, competition and partners.

   8. RKRune Kvist38:01

      Yeah.

   9. SWswyx38:02

      Can you go more into, like, how they partner?

   10. RKRune Kvist38:03

       Yeah. So first of all, OWASP is basically an open source community of security practitioners that are coming together to build frameworks for addressing the latest security concerns. We think they are phenomenal at creating frameworks. We've in fact... We've, first of all, we're partners with them, so we have a joint article. Two, we've learnt a lot from them. We think they're a tremendous source of, of intelligence. What OWASP does not do is building the machine that runs third-party audits such that a company like Cursor or a company like JPMorgan could get a third party to go and review them against this and say, "Hey, you've passed the standard, and here is the report that you can use to build trust and preempt your partners' or customers' questions." So they fundamentally try to do something different. You can... They are part of the information gathering and intelligence gathering and creating clarity, but the operational layer of turning this into promises is, is not the business they tr-they try to be in.

   11. SWswyx38:57

       The standard is emerging and, and it's doing very well. Was it necessary to then also do underwriting? Uh, obviously it's in the name, so presumably you thought about it first. I feel like if you just have enough consensus, you, you don't actually need the money angle, but it does help.

   12. VIVibhu39:13

       I, I did wanna also know, you guys are a for-profit company too, right? It's not non-profit work. There's, there's a whole business side to it as well?

   13. SWswyx39:22

       Yeah, yeah. Yeah, so I'm basically-

   14. VIVibhu39:24

       Great

   15. SWswyx39:24

       ... asking about the money part.

   16. RKRune Kvist39:25

       Let's, let's get to this.

   17. VIVibhu39:25

       Yeah. [laughs]

   18. RKRune Kvist39:26

       Yeah, yeah. Yeah, let's get into the money part. Let's start from actually your question, for-profit versus non-profit. In the security space today, cybersecurity, most of the standards are produced by non-profits. I think that's an issue. [laughing]

9. 39:43 – 43:14

   Why Standards and Insurance Belong Together

   1. RKRune Kvist39:43

      The question you have to ask yourself is how do you create good incentives for these standards to be good and keep up? Non-profits tend to not have these adverse profit incentives where they, uh, hollow out their standard and create a race to the bottom, but they're also not at all responsive by default to the communities that they serve. There's, 'cause there's no pro... They don't have customers that they serve where they go and ask, "What do you want? What do you want? What do you want?" And when you look at the overall satisfaction with the security standards today, people tend to just not like them [chuckles] very much. You do see in other domains, uh, that for-profit standards can serve the world quite well. So there are examples, uh, like we talked about Moody's before. It's not without flaws, but, uh, it is absolutely critical societal infrastructure that gets run at astounding scale today. Your credit score, it's FICO. It's also a for-profit business. And when you go back even further in history, some of the crash testing standards came out of insurance companies. The insurance companies together funded the, uh, founded the Institute of, Insurance Institute of Highway Safety-

   2. SWswyx40:55

      Mm

   3. RKRune Kvist40:55

      ... 'cause they were very interested in, like how can we use standards to drive down mortality-

   4. SWswyx41:00

      Mm

   5. RKRune Kvist41:00

      ... and save money? Go back, uh, pri- our name actually pays homage to the Underwriters Laboratory, UL, which, uh, was started right around when electricity came out. Houses started burning down. Insurers again were paying the bill, and they were maybe also good people, but their profit incentive was, "Let's prevent houses from burning down. Let's test all the electrical products," the light bulbs. All the light bulbs in here are probably UL-tested, the toasters, et cetera. And they set up, uh, an entity to create those standards. Today, UL has a for-profit entity and a non-profit entity. Uh, what they've recognized... They spun out... They started non-profit. They spun out a for-profit because what they recognized was like, hey, actually to serve customers well, you need a for-profit entity. The lesson here is one of the ways that the market can align incentives, so you're both responsive to customers and not hollowing out your standard over time, is to align it with insurers because they fundamentally have good incentives. And so if you're a for-profit standard that works closely with insurers, you get the feedback loop in such that you're t- really clued into your customers but also have their interest at heart. So that's the model that we're... the kind of inspirational model that we've learned a lot from, and that's also where the name comes from. In some ways, the, the term underwriting can both be associated with insurance, but it's also a broad term for, like, making decisions.

   6. SWswyx42:22

      Mm.

   7. RKRune Kvist42:23

      If you underwrite a decision, uh, you're fundamentally kinda taking ownership for, for the consequences of it.

   8. SWswyx42:28

      Yeah. I mean, what, uh, does an insurance contract look like for AI? [laughs]

   9. RKRune Kvist42:32

      Yeah. Most of the demand comes today for insurance contracts is, uh, sitting between people who've built AI and people who are buying AI.

   10. SWswyx42:39

       Yes

   11. RKRune Kvist42:40

       And what you want is... The reason why people want insurers involved, both for the traditional reasons, hey, if something goes wrong, we wanna be compensated. But it's in particular because insurers can increase in, uh, can bring trust to the equation. Because insurers will take- pay for the damages. If they're willing to write an insurance policy, that is them saying, "Hey, we think there's risk here, but that is manageable." And that is kind of a-- Their incentive aligned with the enterprises adopting it, so that's a really a good signal to the market. In the same way, actually, uh, one of

10. 43:14 – 52:13

    What Does an AI Insurance Policy Actually Cover?

    1. RKRune Kvist43:14

       the things that Waymo tried w- to get their first permit to even operate in San Francisco was to get a lot of insurers to stack up a huge insurance policy in the case if something went wrong. Not because Google can't pay, but because it was very valuable to have a third party go and look at that data that are trusted by governments, trusted by enterprises as conservative people and say, "Hey, we've looked at it. We're actually willing to take some of this onto our balance sheet." So that's, that's kind of the reason why people are interested in it. What it looks like is, uh, in some ways like every other insurance contract. You specify what are the perils you wanna cover, how much do you wanna cover them, like up to what limits, and what does it cost to cover that. And in the case of, um, if we take a really concrete example, uh, Eleven Labs, uh, bought a first of its kind AI agent insurance policy. They work with some of the biggest, uh, enterprises. They work with governments. They're really interested in going above and beyond and making promises to their customers. So they wrote a policy that covers just some of the core concerns that their customers have been asking about. And, uh, the crucial thing was really g- to get Lloyd's of London, the world's oldest insurer, one of our partners, to look at this data and be that third party alongside us to say, "Hey, we think there's something here that's worth underwriting." Um, and that's actually what it looks like. And so they will show that cu- that contract to their customers, and they can see how much they're covered for, they can see what exactly it covers, uh, and that will also probably change next year. They will wanna write an insurance policy that might cover more.

    2. SWswyx44:47

       When you say Lloyd's, is it reinsurance, or are they sharing somehow at the same level, or b-

    3. RKRune Kvist44:54

       Yeah. So typically the way, uh, new companies get into insurance is that they partner with an insurers such that the insurers take the majority or all of the financial risks. Fundamentally, if, if, if insurance is useful, uh, because it brings trust, you have to be able to pay the bill. Lloyd's of London is four hundred years old. They've never not paid a claim. They're extremely trusted. Um, what Lloyd's of London struggle to do on their own is to figure out which of the risks are real, what should we be looking for, what are the kind of technical controls, and running the tests. So they use AIUC-1 as kind of the underwriting framework, and we produce a bunch of AI results that then directly feed in to inform the pricing. Uh, so this means that Eleven Labs customers know that that payment will be there. They don't have to look to our Series A and see, like, do we think they have enough cash on the balance sheet. They will look at Lloyd's.

    4. SWswyx45:48

       Yeah.

    5. RKRune Kvist45:48

       Um, yeah.

    6. SWswyx45:48

       And Lloyd's, like famously very creative. I, I think I remember some headline like they insured Jennifer Lopez's, uh, butt or something.

    7. RKRune Kvist45:56

       Correct.

    8. SWswyx45:56

       Right?

    9. RKRune Kvist45:56

       And I think, uh, was it, uh, David Beckham's right foot-

    10. SWswyx45:59

        So, yeah, right

    11. RKRune Kvist46:00

        ... and, and stuff like this.

    12. SWswyx46:01

        So, like, clearly n-not a large data set. [laughs]

    13. RKRune Kvist46:05

        Ex-exactly.

    14. SWswyx46:06

        [laughs]

    15. RKRune Kvist46:07

        It's actually a remarkable institution that's both kind of has some of the truly old school virtues of having been around for a long time. They, like, really-- they really operate like a trusted entity, and they have appetite to figure out the future. Uh, and I think there's a lot of recognition that both there is, like, tremendous amount of risk in AI that is poorly understood today, so getting into this business carries real risks. Uh, but also this is where lots of the risk disclosure will happen in the future. This is the one market where risk is truly growing. This is the one market that will also take out some of the existing markets. Take, like, auto insurance. When there are no human drivers, how's that market gonna look? Well, it's clearly gonna change. How are you gonna assess-

    16. SWswyx46:51

        You wanna insure Waymo? [laughs]

    17. RKRune Kvist46:53

        [laughs] I-- All I'll say is the principles for how you insure Waymo are very similar to how you insure other kinds of AI.

    18. SWswyx46:58

        Right. Right.

    19. RKRune Kvist46:58

        So again, crash testing. That's what we do for Cursor, Share, Lovable. That will also need to happen for Waymo, which is not how you do it for human drivers. So there's this growing awareness that the world is changing very fast, and the only way to learn how to underwrite AI is to write some policies. You may incur some losses and, and think of that as R&D expense really. But the question for them is, like, who are the trusted technical partners they can get into this business with that can help them navigate and make sure they don't make, uh, kind of foolish mistakes, but also who is willing to hear the wisdom that they have? They've done this before. They've seen it was-- They were there when cyber came out. So there are lots of ways in which AI feels completely new, but there's also lots of ways in which risks look the same. And so there's actually a tremendous amount of wisdom sitting in some folks that may have gray hair, uh, but really have, like, a, a keen sense of, uh, how to quantify risk.

    20. SWswyx47:51

        Yeah. And, and the number is... So it's basically like I want fifty million dollars worth of cov- of coverage against these perils, and Lloyd's will give you a quote on it, and then you, you have like a small markup or something, and then you, you turn it around and, and do that? Is that, is that as simple as it is?

    21. RKRune Kvist48:06

        You, you basically share some of that premium.

    22. SWswyx48:08

        Yeah.

    23. RKRune Kvist48:08

        So X percent goes to the people who do the, the pricing of it.

    24. SWswyx48:11

        Yeah. It's kind of like a, it's kind of like a merchant bank for insurance type of thing.

    25. RKRune Kvist48:16

        Exactly.

    26. SWswyx48:16

        Yeah.

    27. RKRune Kvist48:16

        You basically split the fee, and you can think of the insurance supply chain as like there's bringing the capital, there's doing the pricing, and there's doing the distribution. And typically you will pay out some ex-ex percent of premium here, Y percent of premium here, and the rest of it will go here.

    28. SWswyx48:29

        Does all the insurance world work like this, or d- is there some point at which like... So, so if I-- right now you have equity capital.

    29. RKRune Kvist48:34

        Yeah.

    30. SWswyx48:35

        At some point, maybe you start raising, uh, debt or whatever, and then you have enough of a bank account and enough history. Let's say you've been in operation for ten years-
11. 52:13 – 55:22

    The $20 Cursor Subscription and the $200M Plane Crash

    1. RKRune Kvist52:13

       works is that, uh, you start from a lot of uncertainty. This is not just in insurance, but also, like, can you use-- can Anthropic use books on the internet to train up? Well, they can go and look at precedent, they can see-

    2. SWswyx52:26

       Mm

    3. RKRune Kvist52:26

       ... but ultimately, this, these things get settled in court, and you hammer it out over time. So you start from this, like, place of ambiguity, which is both why insurance can be hard to do early on, but it's also why people want insurance, 'cause that ambiguity slows down adoption.

    4. SWswyx52:40

       Yeah.

    5. RKRune Kvist52:40

       That also sits at the heads of the, uh-

    6. SWswyx52:42

       Yeah

    7. RKRune Kvist52:42

       ... JP Morgan-

    8. SWswyx52:43

       In, in some ways, actually, the first incident will help to, uh, establish a lot of this.

    9. RKRune Kvist52:48

       E-exactly. And, and there have been a number of incidents out there that have just not been insurance covered. So-

    10. SWswyx52:52

        Yes

    11. RKRune Kvist52:52

        ... uh, take the now old, uh, example from Air Canada where-

    12. SWswyx52:57

        I was gonna bring that up

    13. RKRune Kvist52:58

        ... hallucinated a, a refund policy, and the question was... Air Canada in that case were like, "Hey, we have nothing to do with this. This chatbot messed up, but, like, sorry." [laughs] And the courts were like, "No. If you put your chatbots to interact with your customers, they make legally binding promises on your behalf." That is now precedent for everything in the future, where you will... If someone were to deploy a chatbot like that again, they-- you should not expect to be able to just pawn off and say, "Sorry, my chatbot lied. It's nothing to do with me. I bought it from OpenAI." No. If you're putting this in front of your customers, you are taking responsibility for it. And so every court case, whether insurance is involved or not, clarifies liability, and liability is kind of the foundation for insurance. There's another reason why standards and insurance come together. Liability... I'll go on a little tangent here-

    14. SWswyx53:49

        Please

    15. RKRune Kvist53:49

        ... get into the weeds of it.

    16. SWswyx53:49

        Please.

    17. RKRune Kvist53:50

        Liability, often one of the core concept is whether someone was negligent. Should they have seen this? Should they have prevented this? And the question you-- how do you judge that? Well, you basically judge whether they've met their duty of care. What does that mean in practice? Well, often they look to standards. So if there's a standard that is broadly adopted that says you must have a groundedness filter or you must have a jailbreak filter, it becomes way harder to claim ignorance that these things existed. And so setting standards help clarify liability. Coins-- Uh, courts will often point to standards and being like, "Well, this seems like best practice to do. It's there for everyone to see." So there's another way in which, like, standards are kind of civilization infrastructure that insurance can then build on, which promises can then build on.

    18. SWswyx54:36

        I totally get that. We don't have to get certified to, to write these, to, you know, make these, like, bots and all these.

    19. RKRune Kvist54:43

        Correct.

    20. SWswyx54:43

        Um, but, like, basically, whenever we get-- go for the audit, I think people, like, start to shape up and, and all this, all this stuff. I wonder if, like, that means that you don't also then become, like, the approving authority for me to ship to production. You know? Like, um, yes, you check once per quarter. I wanna ship once a day.

    21. RKRune Kvist55:02

        Yeah

    22. VIVibhu55:02

        ... and I don't know when one of my things breaks, like [chuckles] one of your certifications or not.

    23. RKRune Kvist55:07

        So there, there's a couple things, um, there's a couple of requirements in there that relate to how do you yourself, where you have to tell your customers-

    24. VIVibhu55:16

        It's like an ongoing monitoring

    25. RKRune Kvist55:17

        ... how are you yourself testing before you make at least major releases. We don't go and audit

12. 55:22 – 57:50

    AI Liability, Monitoring, and Earning Enterprise Trust

    1. RKRune Kvist55:22

       people every day, uh, but at least there is now a trail where if you do a major mess-up, then your customer-

    2. VIVibhu55:28

       Yeah

    3. RKRune Kvist55:28

       ... may come and ask you, "Hey, you promised me that you were gonna run these evals yourself." And for lots of them, most, most of the PRs that people merge will not fundamentally alter the product experience, but some of them will.

    4. VIVibhu55:39

       And sometimes you don't know.

    5. RKRune Kvist55:41

       And sometimes you don't know, and this is also true. And this is also there are some inherent risks that everyone kind of... Everyone knows that when they buy software, there can be bugs, and this, this is just part of it. But what they can, if you're selling to mom-and-pop shops, they may not care. They're just like, "Well, I, I wanna use your tool, so I'm just gonna willing, willing to take that risk on." If you're selling to a big bank, they might be like, "Sorry, we're making promises to our customers. If you can't make a promise to us that we can pass on, we don't wanna work with you." Then it's up to you to say, "Do I care for my agent to get used as critical infrastructure in this nation? If so, at least I can make promises about what process I run, and then we can go and test it every quarter to be like, well, does this seem like, uh, it's still of, uh, kind of it still meets the standard." So from our perspective, it's kind of a way to... Big companies by default kind of have some amount of trust when they ship AI. If you're a young company, if you're just starting out, by default you have no trust, and there are very few places where you can go and get trust. So one of the things that most of our customers did before they started working with us is that they would make their own security blog posts. That's great, but also, who's gonna trust you saying, "We're so secure"?

    6. VIVibhu56:47

       [chuckles]

    7. RKRune Kvist56:48

       Like, anyone can write that. But it's very hard where do you go and get that trust.

    8. VIVibhu56:51

       Yeah.

    9. RKRune Kvist56:51

       And so I think making the standards more legible makes it easier for smaller companies to prove that they're doing what they ought to be doing, uh, because the default assumption is that it's the Wild West.

    10. VIVibhu57:03

        Is there a roadmap you have of like, there's a lot of work to be done here, right?

    11. RKRune Kvist57:08

        Yep.

    12. VIVibhu57:08

        This is the first one.

    13. RKRune Kvist57:09

        Yeah.

    14. VIVibhu57:09

        Um, anything on the roadmap of what you see is next, what's coming, what's, what's missing?

    15. RKRune Kvist57:15

        I think when we, when we zoom out, AIUC-1 deals with agents. We will-- next up, we will deal with models. Next up from that, we will deal with robotics, of which in some ways Waymo is the first robot, but the exact same problem's gonna be someone's gonna develop a robot, someone's gonna need some promises, they're gonna struggle to make the promises. And you see this playing out when like, uh, if you think fable concerns are bad, like see when Waymo hits a dog in SF, people lose their mind. Imagine when first robot knocks off a toddler off a kitchen table.

    16. VIVibhu57:46

        Uh, yeah.

    17. RKRune Kvist57:46

        You're gonna see some real strict liability.

13. 57:50 – 59:58

    From AI Agents to Models to Robotics

    1. RKRune Kvist57:50

       So-

    2. VIVibhu57:50

       I mean, you could see it, right? Like Cruise got fully-

    3. RKRune Kvist57:53

       Destroyed

    4. VIVibhu57:53

       ... all permits are gone. Yeah.

    5. RKRune Kvist57:55

       Right. So physical AI, the level of stringency just goes up and up and up and up. Um, so that's kinda like the big picture. Agents, models, robotics. I think within agents, the current set of agents are well covered by this. But as the technology progresses, as agents get longer horizons, new types of failure modes will emerge. And so it's mostly of can you make sure the standard keeps up when they appear? And you also start to see new modalities like today, world models is mostly, uh, kind of a, a research question. There's no one who's really using it, but that will also bring in just new kinds of ways to create value, but also more risk surface that no one knows how to grapple with today. You'll start to see true agent-to-agent interactions that are not mediated by humans. There's gonna be a bunch of interesting questions. You're basically gonna need a new legal system. How do they build trust amongst each other? How... One of the core things when humans trade with each other is that you know that you have recourse, you can sue them. How do you make sure that there is a persistent balance sheet behind any agent such that if you trade with it and it screws you, you know you can get your money back? Those are some of the questions we're gonna have to deal with, and the technical testing of multi-agent systems is also gonna be interesting and complex. [chuckles]

    6. VIVibhu59:12

       Very fun. Uh, are there any perils that are uninsurable right now that people wish that you would?

    7. RKRune Kvist59:18

       Yeah. One of the places where there's a bunch of appetite for insurance and not a lot of, a lot of demand, but not a lot of supply is when it comes to copyright.

    8. VIVibhu59:29

       Oof.

    9. RKRune Kvist59:30

       In some ways, copyright is kind of mundane. It's always been an issue. Uh, there's a couple of reasons for this. The first is people who have trained on copyrighted materials almost always know that they've done that.

    10. VIVibhu59:41

        [laughs]

    11. RKRune Kvist59:42

        Uh, so if you wanna buy insurance for it, it probably signals that you might be a high-risk customer. The people who are most interested in getting insurance for copyright-

    12. VIVibhu59:51

        Okay

    13. RKRune Kvist59:51

        ... infringement-

    14. VIVibhu59:52

        Yeah, yeah

    15. RKRune Kvist59:52

        ... are the people who are most likely to have created copyright-

    16. VIVibhu59:54

        Yeah, yeah, yeah. It's like a, it's like a lemon problem.

    17. RKRune Kvist59:56

        Exactly. Um-

    18. VIVibhu59:56

        I, I actually think there's another side

14. 59:58 – 1:04:57

    Copyright, Adverse Selection, and AI Insurance

    1. VIVibhu59:58

       to it too, right? Like if you're building on something... So say I'm using an open model.

    2. RKRune Kvist1:00:02

       Yeah.

    3. VIVibhu1:00:02

       I don't know what it's trained on, right?

    4. RKRune Kvist1:00:04

       Yes.

    5. VIVibhu1:00:04

       And how far down that chain does copyright go?

    6. RKRune Kvist1:00:07

       Yes.

    7. VIVibhu1:00:07

       Am I liable to take down my product because company X trained on copyright?

    8. RKRune Kvist1:00:12

       Uh, but there's safety in numbers. If everyone's doing it, then you...

    9. VIVibhu1:00:16

       I mean, I would say until, you know, Fable has rolled back from everyone that used it, right?

    10. RKRune Kvist1:00:20

        Yeah. At least a hard question. I don't have the answer to it.

    11. VIVibhu1:00:22

        Yeah. [laughs]

    12. RKRune Kvist1:00:22

        But I think your intuition is, your intuition is right and kind of like, uh, what is the kind of duty of care? And people don't today think of it as customary that you go and you, like dissect your open models training data and you check everything. In fact, lots of people use them. It's seen as kind of generally acceptable to not check for this, and therefore we're not gonna hold you specifically-

    13. VIVibhu1:00:45

        I mean, we also really can't, right? We don't-

    14. RKRune Kvist1:00:47

        Exactly

    15. VIVibhu1:00:47

        ... we don't know the training data-

    16. RKRune Kvist1:00:48

        And so you can then ban it, but I think no court is gonna get a copyright question and be like-

    17. VIVibhu1:00:51

        Unless you-

    18. RKRune Kvist1:00:51

        ... "This actually needs to get banned."

    19. VIVibhu1:00:52

        ... product hire Nicholas Carlini and he can extract it for a week.

    20. RKRune Kvist1:00:55

        Exactly, though he's in short supply. [laughs]

    21. VIVibhu1:00:58

        Yeah, he only has so many Carlinis, but, uh-

    22. RKRune Kvist1:01:00

        Ex-exactly.

    23. VIVibhu1:01:01

        Yeah. Go ahead.

    24. RKRune Kvist1:01:02

        So I think this is, this is also fair that, uh, in the case of labs, there's a lot of interest for this, but the thing that makes lab want it is what makes ins-insurers suspicious of it, and so you have a lemons problem. Um-

    25. SWswyx1:01:13

        Yeah. Is there, like, a theory of insurance where adverse selection dominates the risk-sharing as- aspect of insurance? Like, where does this... Like, t- teach us insurance.

    26. RKRune Kvist1:01:23

        A lot of insurance does come back to, like, practical versions of microeconomics 101. Um-

    27. SWswyx1:01:28

        Yeah, it's, it's re- it's like, it's like this is why you do-

    28. VIVibhu1:01:30

        Harvest adversity

    29. SWswyx1:01:31

        ... you need to pool health insurance, uh, because if you make it too hyper-specific, then only people who are guaranteed to get the disease will sign up for your insurance.

    30. RKRune Kvist1:01:39

        Exactly.
15. 1:04:57 – 1:10:05

    Evals, Mechanistic Interpretability, and Eval Awareness

    1. RKRune Kvist1:04:58

       We're still a while, a little bit away at least, from this being, like, commercially available on demand such that there's, like, now a selection of vendors you can go to.

    2. SWswyx1:05:09

       Goodfire would say that it is commercially available.

    3. RKRune Kvist1:05:11

       Exactly.

    4. SWswyx1:05:12

       Uh, and just to just-

    5. RKRune Kvist1:05:13

       We, we, we would agree with them. We, we think that the work that they're doing is tremendous.

    6. SWswyx1:05:16

       Yeah.

    7. RKRune Kvist1:05:16

       We're not quite at a point where we could, like, literally require it, but it's the kind of thing where you can imagine relatively soon you could put in an optional control for if people use McINTURB as a way to reduce risk, you at least get credit for it. We can't require it because it would be hard to require everyone to become Goodfire customers.

    8. SWswyx1:05:32

       What good does credit do me? Uh, this is the-

    9. RKRune Kvist1:05:35

       Say again?

    10. SWswyx1:05:35

        This is a pass-fail, right? Do I, do I, do I care about credit?

    11. RKRune Kvist1:05:37

        Uh, it's a pass-fail, but it's also a 100-page audit report-

    12. SWswyx1:05:40

        Uh-huh

    13. RKRune Kvist1:05:40

        ... that you'd be surprised at how much Shakur and Liaos actually sit down and digest this stuff.

    14. SWswyx1:05:45

        Okay.

    15. RKRune Kvist1:05:45

        Uh, and I promise you that if someone is using McINTURB today-

    16. SWswyx1:05:50

        [laughs]

    17. RKRune Kvist1:05:51

        ... uh, they will have a slide on it 'cause they were trying to get credit for it.

    18. SWswyx1:05:54

        It is, it is cool. It's fancy, yeah.

    19. RKRune Kvist1:05:55

        But it's just easier if you have a third party saying, "Yep, they have McINTURB," and actually get someone for it.

    20. SWswyx1:05:59

        And, and just to, like, just to flesh... spell it out for people, people who have been following our, our McINTURB podcast.

    21. RKRune Kvist1:06:04

        Yeah.

    22. SWswyx1:06:04

        It is literally like, "Oh, you're using, uh, you know, GPT-oSS. It is activating these three dangerous things. We monitor for it, and we log it out in whatever tool of choice." Grace Swan has, like, Signal or whatever, and that's it. That, that, that's the McINTURB-based activation, uh, signal. Okay.

    23. RKRune Kvist1:06:21

        Yeah, yeah. So I think McINTURB is interesting, and I think if that promise truly comes to fruition, you can make stronger promises than you can with Evals, and so I think that's very compelling. Another thing that I think will become increasingly important is just kind of good old-school monitoring and slightly after the fact. Uh, one of the things you're seeing with Eval, some of the challenges that are emerging, is that the agents are starting to become aware that they're being Evaled.

    24. SWswyx1:06:47

        Yeah, Evo awareness.

    25. RKRune Kvist1:06:48

        Exactly, which is a problem. It means that they basically, if they know they're being watched, they won't do the thing that they think they get punished for. And by default, unless you know how to kind of reduce Eval awareness, you should trust Evals less. And one of the kind of truest things, monitoring, like, is the source of truth. Did you in fact give medical advice, and how quickly do you know? How often do... have you done that in the past? How fast do you respond? How often do you detect it? How fast do you detect this? Uh, so I think that is also a paradigm. They're slightly more intrusive You actually will look at some customer data, uh, but I think will become more prevalent over time.

    26. SWswyx1:07:28

        People talk about this like we should not write about e-eval awareness 'cause it's gonna leak into the data set and then be [laughs] like, like we should just... Like we should, like never talk about it, only meet in person and like talk offline unrecorded like [laughs]

    27. RKRune Kvist1:07:40

        Did you guys see the Anthropic research where, uh, I think this is literally Anthropic did that test.

    28. SWswyx1:07:47

        What?

    29. RKRune Kvist1:07:48

        They took... I can't remember the details here. But they, uh, ran some studies on misalignment, and then they took out the training data that related to LessWrong discussing misalignment, and they ran the same test again.

    30. SWswyx1:08:00

        [laughs]
16. 1:10:05 – 1:13:21

    The Impossible Enterprise AI Mandate

    1. RKRune Kvist1:10:05

       you'll see just like you do in every other enterprise market, if you really sell to the enterprise, you start to compete on some of these security features, and this is also happening AI unsurprisingly. And I think you're seeing some amount of enterprises wanting... Enterprises are really grappling with the thing that makes agents useful is they're stoc-stochastic, and the thing that makes them really hard to adopt is that they're stochastic, and these are just intentions.

    2. SWswyx1:10:28

       Mm.

    3. RKRune Kvist1:10:29

       Leaders come out on different sides of that table, in part depending on how much the CEO is trying to get the stock price to go up by saying they're AI native [laughs] and that we must be willing to take the risks. Uh, we see, we actually see phenomenal tension in the heads of the CISOs of the Fortune 1000, where on the one hand you have a CEO saying, "We must adopt, otherwise we're becoming irrelevant, and if we fuck up, you're fired."

    4. SWswyx1:10:51

       Oof.

    5. VIVibhu1:10:51

       Mm.

    6. RKRune Kvist1:10:51

       And that's kind of like the core emotional tension that we see showing up again and again and again and again. And one of the core problems that we solve for them is to take that abstract emotional concern and turn it into a framework in some ways just providing clarity to, to that concern.

    7. VIVibhu1:11:06

       So anything in here... So something I think we kind of skipped over. We talked a lot about agent language model, skipped over world models.

    8. RKRune Kvist1:11:14

       Yeah.

    9. VIVibhu1:11:14

       Um, you guys have voice, which is interesting with ElevenLabs.

    10. RKRune Kvist1:11:17

        Yeah.

    11. VIVibhu1:11:17

        How about generative media? So, you know, generating images, videos, that's a category that actually has a lot of usage. Is there anything in your current policy? Is it separate policy? How do you see that space?

    12. RKRune Kvist1:11:29

        Yeah.

    13. VIVibhu1:11:29

        It's like we did talk a bit about copyright, so-

    14. SWswyx1:11:32

        Music.

    15. VIVibhu1:11:33

        Yeah, music as well, like-

    16. RKRune Kvist1:11:34

        Yeah. I think a lot of the concerns that come up there either relate to, uh, copyright or there's a lot related to, let's call it broadly safety. So like this could be not safe for work or just very graphic materials, uh, are kinda some of the core things. Uh, we have done some work on this. There's a little bit in the standard as well that deals explicitly with that. Uh, video we've not done a lot in yet, and I think for proper production that has still... Especially proper production without a human in the loop, that's still got some ways to go. It's obvious that it's coming, but it's very rare that it's like one shot deploy a video to the internet, uh, but eventually that will also happen.

    17. VIVibhu1:12:16

        We see like, you know, uh, Luma has Luma agent where it's still pretty human in the loop.

    18. RKRune Kvist1:12:20

        Yeah.

    19. VIVibhu1:12:20

        It's not, it's not just-

    20. RKRune Kvist1:12:20

        And, and that just makes complete sense as the technology matures and over time it will become so good that people will not wanna slow things down by having a human in the loop, and then, uh, the need to make promises will drop.

    21. SWswyx1:12:35

        Why not just have prediction markets on everything?

    22. RKRune Kvist1:12:37

        Mm.

    23. SWswyx1:12:38

        Right? It's very EA adjacent.

    24. RKRune Kvist1:12:39

        Yes.

    25. SWswyx1:12:39

        [laughs]

    26. RKRune Kvist1:12:40

        Uh, the core thing is that the people prediction markets rely on public information. There is not a lot of public information. There's just insiders trading on each side.

    27. SWswyx1:12:49

        [laughs]

    28. RKRune Kvist1:12:51

        Uh, that's illegal. [laughs]

    29. VIVibhu1:12:53

        There's leaked information.

    30. RKRune Kvist1:12:55

        There's leaked information. The core challenge is that often you have private sensitive information, and you need to convey confidence and trust around that, and you can of course for some claims, like can any model be jailbroken? You could rely on public evidence 'cause there'll be lots of people being like, "Well, there's tons of studies, and actually they all can," so that resolves fine. I think that's good. For,

17. 1:13:21 – 1:16:12

    Prediction Markets vs. AI Audits

    1. RKRune Kvist1:13:21

       "Hey, this new unreleased- Meet this model, how capable is it actually?

    2. SWswyx1:13:26

       Hmm.

    3. RKRune Kvist1:13:26

       Prediction markets have not a lot to say because actually just no one knows. And so I think that's the core place where some of this breaks down, is that actually lots of the world's information that guides some of these high-level decision is private and often also just not known.

    4. VIVibhu1:13:39

       I think the thing with prediction markets that people like is it's not, it's not answering the broad question, it's a specific right. So will a model do this by this date, or is a model capable to do this by then, right?

    5. RKRune Kvist1:13:50

       Yes.

    6. VIVibhu1:13:50

       So that's a little distinction there.

    7. RKRune Kvist1:13:53

       Yeah. And often the most interesting question, if you're, say, the head of security at a bank, the question you're really trying to answer is, will this product, this agent, do this bad thing that maybe primarily I care about, specifically in the setting that I care about? And the question is like, what is the closest... That information may not exist anywhere, so prediction markets aggregate existing information. This information may not exist, and you want something very specific, and you're willing to pay for it. That's kind of where a third-party audit comes in. We also don't really use prediction markets to figure out whether, uh, public companies have committed fraud on their books. You use audits.

    8. VIVibhu1:14:30

       [laughs]

    9. RKRune Kvist1:14:31

       You probably could, but the information is just not that available.

    10. VIVibhu1:14:33

        [laughs]

    11. RKRune Kvist1:14:33

        And i-if so, it would be like just trading on vibes. Uh, uh, actually, it would have been really interesting to see whether prediction markets 2001 would have predicted Enron going bankrupt, and the kind of pr- uh-

    12. SWswyx1:14:45

        Yeah

    13. RKRune Kvist1:14:45

        ... could you have told-- could you have sensed from the like the, the craziness of the CEO or some other trait that they were more likely to cook their books than others?

    14. SWswyx1:14:53

        Or enough insiders leak it then that you-

    15. RKRune Kvist1:14:55

        That could also be a way

    16. SWswyx1:14:56

        ... right, right. Which, which is like, I mean, this, that, that's the sort of the ideal dream of prediction markets is you have liquid markets and everything-

    17. RKRune Kvist1:15:03

        Yeah

    18. SWswyx1:15:03

        ... and then you can compose your exact set of risks to offset.

    19. RKRune Kvist1:15:07

        Yes.

    20. SWswyx1:15:08

        Right?

    21. RKRune Kvist1:15:08

        Yes. Yes, yes, yes.

    22. SWswyx1:15:09

        [laughs]

    23. RKRune Kvist1:15:09

        Yeah, and, and I think like prediction markets will bring lots of new information to it, as the thing is mostly not like which one is it and more like what are the types of questions that prediction markets are really good at-

    24. SWswyx1:15:20

        Yeah

    25. RKRune Kvist1:15:20

        ... and what are the ones where the information doesn't even exist for insiders such that no one can in fact trade on it and it needs to get generated.

    26. SWswyx1:15:26

        Okay, one self-serving question and then one open-ended one, uh, on like the, the future of AI you see. Uh, self-serving question would be so you have your standard, right?

    27. RKRune Kvist1:15:35

        Yes.

    28. SWswyx1:15:35

        I run, you know, a large AI engineer conference. Like, there's been a lot of talk about us certifying AI engineers.

    29. RKRune Kvist1:15:41

        Yep.

    30. SWswyx1:15:42

        Training programs, level one, level two, level three. I was a CFA myself, so I know what... that-that's what the finance industry does.
18. 1:16:12 – 1:20:39

    Should AI Engineers Be Certified?

    1. RKRune Kvist1:16:12

       security considerations that they had not thought of and now is not built to spec. Now you have to go and re-eng-- like add these filters, et cetera. So if you shifted that left, like if everyone knew what the spec that they were building to, if everyone knew the grading scheme-

    2. SWswyx1:16:24

       Yeah

    3. RKRune Kvist1:16:25

       ... that'd be awesome if they were already trained. So by default-

    4. SWswyx1:16:27

       But you're the grading scheme, right?

    5. RKRune Kvist1:16:28

       Say again.

    6. SWswyx1:16:28

       I, I don't get to set the grading. You guys, you set the grading scheme.

    7. RKRune Kvist1:16:30

       We set the grading scheme, and I think what's, uh, valuable is like if you could turn this into-

    8. SWswyx1:16:35

       Training programs

    9. RKRune Kvist1:16:36

       ... training programs-

    10. SWswyx1:16:37

        Yeah

    11. RKRune Kvist1:16:37

        ... such that people-

    12. SWswyx1:16:37

        Which you're, you're not doing

    13. RKRune Kvist1:16:38

        ... we're not doing that.

    14. SWswyx1:16:39

        Yeah.

    15. RKRune Kvist1:16:39

        I think there's value in doing it.

    16. VIVibhu1:16:40

        There, there are others doing, I mean, not to interrupt, interrupt, but you know-

    17. SWswyx1:16:43

        Yeah, yeah

    18. VIVibhu1:16:43

        ... the last week's OpenAI has their-

    19. SWswyx1:16:45

        Anthropic also has like a CCPA thing.

    20. VIVibhu1:16:47

        Yeah, yeah.

    21. SWswyx1:16:47

        Uh-

    22. VIVibhu1:16:47

        You know, they want, they want hundred thousand deployed certified consultants, right? Which is-

    23. RKRune Kvist1:16:53

        I basically think it's good for-- we will accelerate adoption if we have more people who know how to build secure agents, and we are not working on the side of training people at the moment. I think it's like very aligned with our mission. We only have so much, uh, attention.

    24. SWswyx1:17:07

        I tell you why I haven't done it.

    25. RKRune Kvist1:17:09

        Yeah.

    26. SWswyx1:17:09

        It's not like I, I haven't thought about it before.

    27. RKRune Kvist1:17:11

        Yes.

    28. SWswyx1:17:11

        It's just being prescriptive-

    29. RKRune Kvist1:17:13

        Right

    30. SWswyx1:17:14

        ... about like, well, this is what you should know, therefore, like the stuff that I didn't include is what you don't need to know.
19. 1:20:39 – 1:26:57

    AIUC’s Roadmap, AGI, and Who Watches the Watchdogs?

    1. RKRune Kvist1:20:39

       that are most likely to slow down adoption of AI. And then, yeah, we think robotics comes after that. Um-

    2. SWswyx1:20:46

       What are you hiring for? What's hard to hire for?

    3. RKRune Kvist1:20:48

       We are hiring across the board, across go-to-market and members of technical staff. The people who do really well on our technical team are folks who are really excited about kind of being truly full stack. So let's say when we started working with Cursor, uh, we'd never done coding, uh, tools before. So taking the standard and extending it and fleshing out what does frontier evals look like for long-horizon coding agents, and taking that problem all the way from like working with Cursor and other folks in the space down to like fleshing out and shipping a new version of the standard. Uh, so that's like a truly a full stack entrepre- entrepreneur-shaped technical people do extremely well at AIUC. The hard part is building one universal red teamer that works across from Harvey to Cursor and everywhere in between that both has one consistent methodology, one consistent taxonomy of what are the risks and the attacks, uh, and making... We think that's fundamentally the best way to make consistent promises. JP Morgan is buying both. They wanna have one framework, one consistent way that this comes out, and the mechanics of making that happen, you get to deal with a lot of the complexity of the real world. I think we have good answers in a bunch of that, but there are some pretty hard engineering problems in, in executing that.

    4. SWswyx1:22:01

       Can I push a little bit? Like must you have one? Why not just be like, "Okay, look, 40% of our use cases are coding agents, so we will specialize in coding agents, and that's the, that's the one of them."

    5. RKRune Kvist1:22:11

       Yes. And that was the-

    6. SWswyx1:22:12

       And then, okay, 30% is like RAG.

    7. RKRune Kvist1:22:14

       Yes.

    8. SWswyx1:22:14

       Just do RAG.

    9. RKRune Kvist1:22:15

       Yes.

    10. SWswyx1:22:15

        [laughs]

    11. RKRune Kvist1:22:16

        Uh, I think there's some wisdom in that question.

    12. SWswyx1:22:19

        Yeah.

    13. RKRune Kvist1:22:20

        Um, it depends on what we found that there's a lot of value on is being able to... If the decision maker on the buying side, let's say you're the head of risk at a bank, and your biggest risk is not in coding or in customer support or whatever the top two biggest use cases, but is somewhere else, you wanna still make sure that that framework has something to say about it to the burning question you have. Otherwise, you'll not earn their trust. Now, it's true that a lot of the burning questions follow where there's a lot of adoption, and so great, so do we. So we today do not cover every single edge, but we have a framework that we can add all of these within. We have one global taxonomy of risks and attacks. That keeps adapting. As like every time a new incident occurs that has never been seen before, great, let's go and update the taxonomy so we bake that in. So I think we have one coherent universal approach. It doesn't mean that we spend equal amounts of time on code and insert niche use case. Uh, we, we do spend time where people, where people care. We think it's very valuable to have one language.

    14. SWswyx1:23:26

        Yeah. Yeah. That makes sense. Um, that's, that's a, that's an important choice. Uh, we were gonna end actually, but I, I thought of one final ending closing question, which is, uh, take this however you want, right? Um, let's say one and a half years from now, OpenAI's secret panel of five experts declares that we have reached AGI.

    15. RKRune Kvist1:23:42

        Mm-hmm.

    16. SWswyx1:23:43

        Do you expect your business to change?

    17. RKRune Kvist1:23:46

        No. I think there is some important way... I think the, the last businesses to exist beyond the labs-

    18. SWswyx1:23:53

        Will be underwriting. [laughs]

    19. RKRune Kvist1:23:55

        [laughs] Well, there's one, there's one job that the labs can never do for themselves, which is to be their own watchdog.

    20. SWswyx1:24:02

        There you go.

    21. RKRune Kvist1:24:04

        So I, I think kind of to the extent that you believe this frame of like you'll see hyper concentration-

    22. SWswyx1:24:09

        Mm.

    23. RKRune Kvist1:24:09

        Like the labs will kill all the startups-

    24. SWswyx1:24:11

        Yeah, yeah

    25. RKRune Kvist1:24:12

        ... which, uh, we can go into with the pros and cons.

    26. SWswyx1:24:15

        I feel like the labs actually care a lot about this, right? There was a whole superposition, what do we do when we have models smarter than us-

    27. RKRune Kvist1:24:21

        Yes

    28. SWswyx1:24:21

        ... and a tier above, right? Models smarter than them training them. So the-

    29. RKRune Kvist1:24:24

        Yes

    30. SWswyx1:24:24

        ... the labs actually think about this a lot.

Episode duration: 1:26:57