The Next Chapter of AI Is Inside Our Software

Aaron Levie 0:00 If you regulate AI too early, you actually don't solve anything. You still just kinda have the same risk ultimately.

Martin Casado 0:06 You willed the thing into being-

Aaron Levie 0:07 Yeah

Martin Casado 0:07 ... but you haven't figured out how to control it.

Steven Sinofsky 0:09 The problem we have now is this rift between the labs and the security community that keeps coming to two conclusions: sloppy, and you're not complete in what you're telling us happened.

Martin Casado 0:18 An employee is like 10% chance of species extinction.

Aaron Levie 0:22 Agent swarms completely flip that. These are just roaming drones. But like times 10,000-

Martin Casado 0:27 Right? It's just more

Aaron Levie 0:27 ... and they will easily mistake a good task for a bad one.

Steven Sinofsky 0:30 So now we need a whole layer internally that just is tracking way more about what authentications are being done, what APIs are being done.

Aaron Levie 0:37 Yeah, yeah.

Steven Sinofsky 0:37 The US, about 15 years ago, stopped leading in tech antitrust. The problem is that Europe is gonna lead with that because they have nothing to lose. This could change the nature of software fundamentally. The center of innovation has just moved. This is the signal

Steven Sinofsky 0:50 that-

Erik Torenberg 0:52 Guys, welcome back to the podcast.

Aaron Levie 0:54 Happy to be here.

Martin Casado 0:55 Thank you.

Steven Sinofsky 0:55 Great to be here.

Erik Torenberg 0:56 We've lo-

Aaron Levie 0:56 Didn't think we'd ever do this again. This is ama- I can't believe it. This is great.

Martin Casado 0:59 [laughing]

Steven Sinofsky 1:00 I mean, Martin's just building these-

Erik Torenberg 1:02 I know [laughing]

Aaron Levie 1:02 ... like $100 billion companies, too busy for this podcast, so.

Martin Casado 1:05 Or at least taking credit for them as V- as VCs do.

Erik Torenberg 1:07 Exactly. Um, we have a lot to discuss today, but fir- first, look, A- Aaron, why don't we start with you? Pacing the frontier.

Aaron Levie 1:13 Yeah.

Erik Torenberg 1:13 How have you, uh, reacted and reflected on, on what's happened there and just the discourse that, that's followed? How do you make sense of it?

Aaron Levie 1:18 Oh, boy. I think we should start with Martin on this one. The, uh, he, he-- you were fighting lots of good ground wars. Um, I, I... Maybe I'll say one thing that we probably all agree with, and then we can figure out where we maybe, maybe, uh, fr- kind of fracture off. Um, I think we would agree that, uh, any AI lab right now at the frontier should be building, you know, in the safest way possible with, with, you know, the, the highest degree of governance and, and security and, you know, kind of whatever your definition of alignment is. Like, this is an incredibly important area of research. It's an incredibly important area for the diffusion of AI. Like, you're not gonna have AI diffusion without extremely high-quality products that can be, you know, trusted by enterprises and that aren't kind of constantly hacking systems. So when, when at least I read the Dario post, um, I actually didn't agree with m- didn't disagree with almost anything, um, because it was all about how do you have better, you know, uh, security of these systems, sandboxing, better testing. There's gonna be some debates around the embedded nature of e- of, of the testers and, like, do you agree with who those are, and does the industry all align on that? But I think actually all of the major points were, were probably salient and ap- and appropriate. Then the only question is, does this get sort of used or leveraged to do things that maybe we don't agree with? Which would be, like, a slowdown of, of AI dramatically because of regulatory controls that would, would sort of, you know, not make it eas- easy to compete, you know, with, with the frontier labs. Um, or do politicians kind of end up sort of taking the, the message and run with it and maybe even worse outcomes happen? Like, you, you... You know, it's used to ban data centers, you know, far faster and, and whatnot. And so, so I think the, the actual, like, substance of the topic is actually incredibly important and, and I, I think very important for, for AI advancement in general. And then the question is what do you do about it? Especially what do you do about it from a regulatory standpoint? And that's probably where the industry's gonna land on very different points in the continuum. But Martin was putting up a good fight on, on, like, let's make sure that we don't use this for regulatory kinda capture, um, which I th- I, I also agree with. But, but I think the, the ideas in the pacing conversation are important. Um, you know, again, like, it's a little bit of a funny concept 'cause maybe it's not even pacing as much as just, like, good hygiene and, and then good w-

Martin Casado 3:30 Engineering.

Aaron Levie 3:31 And, and engineering. And, and so with, with good engineering, obviously there is a slight slowdown, but it's a slowdown that obviously allows acceleration of your diffusion because you, you wouldn't be able to have any of the AI be diffused if, if nobody would, you know, trust using it. So-

Martin Casado 3:46 The post is, is very reasonable, but the atmospherics

Martin Casado 3:50 are not, right?

Aaron Levie 3:50 Right.

Martin Casado 3:51 Like, so an employee is like, "This is gonna kill, whatever, 10% chance of species extinction." And you know what Dario says? "I agree with him more than I disagree with him," right?

Aaron Levie 4:03 Disagree with him.

Martin Casado 4:03 Like-

Aaron Levie 4:03 On, on TV in an interview

Martin Casado 4:04 ... on, on, on TV the same day that he landed these things.

Aaron Levie 4:08 Yeah.

Martin Casado 4:08 And so in some way you can't have these conversations in isolation.

Aaron Levie 4:11 Mm-hmm.

Martin Casado 4:11 Which is, of course, if he's gonna agree in species extinction-

Aaron Levie 4:15 Mm-hmm

Martin Casado 4:16 ... this post that he has looks like this milk toast capitulation that's totally not adequate for the task at hand. And so I think the atmospherics are totally broken. And a lot of my comments were on the atmospherics.

Aaron Levie 4:28 Mm-hmm.

Martin Casado 4:28 And then I have this quibble, but it really bothers me because I'm a pedant, which is I think pacing is the wrong way to describe this.

Aaron Levie 4:36 Right.

Martin Casado 4:36 For, for one-

Aaron Levie 4:37 Yes

Martin Casado 4:37 ... it is orthogonal to security.

Aaron Levie 4:39 Mm-hmm.

Martin Casado 4:40 Right? So, like, you can very slowly build a nuclear weapon-

Aaron Levie 4:44 Right

Martin Casado 4:44 ... and that doesn't make anybody feel better that it's slow-

Aaron Levie 4:46 Right

Martin Casado 4:46 ... versus fast. So that's one. The second one, it just feels like a capitulation to the pause folks without actually, you know, addressing it. So you're saying, "Well, we're not gonna pause, we're gonna pace to make them happy, but we'll also somehow make the regulators happy." And I think it makes them both unhappy.

Aaron Levie 5:02 Mm-hmm.

Martin Casado 5:03 'Cause the pause people are like, "Well, that's not a pause, that's just pacing." And then the regulators are, "You're still doing the thing."

Aaron Levie 5:09 Right.

Martin Casado 5:09 And so I just feel like they're trying to... My, my sense of what's happening is the labs are actually trying to do the right thing.

Aaron Levie 5:15 Yes.

Martin Casado 5:15 And I applaud them for that. I think this is a pragmatic proposal, and I applaud them for that. I think the messaging is wrong because they're trying to, like, split the difference between an internal kind of fringe faction, which are doomers-

Aaron Levie 5:28 Yes

Martin Casado 5:28 ... who say pause, and then the regulators on the other side. And the problem is they're making both of them unhappy. And I think what they have to do is they need to come out, they need to address the X risk question directly.

Aaron Levie 5:38 Mm-hmm.

Martin Casado 5:39 Like, they need to say, "No, we don't think this stuff we're gonna do is gonna cause extinction." And then I think this becomes just very sensible.

Aaron Levie 5:45 And what would you do, just to, just to play the other side for one second. What do you do? Uh, do you, do you, uh, m- make room for the, the one possible Venn diagram, which is the, the lab researcher that is both simultaneously super scared ... but also works on advancing the state of AI because they believe that, that it's so important to get right that, that they wanna pursue that. And then obviously the language is, is right now very, you know, problematic. But like, what... But that, that person does exist, and, and that, that is a real kind of person in our industry, which is like, like, we have to be at the, at the forefront of AI. I'm also very scared of it.

Martin Casado 6:25 Yeah.

Aaron Levie 6:25 And, and, and so that's why I'm working on this.

Martin Casado 6:27 So let, let me address this directly. I used to work for Lawrence Livermore National Labs on a weapons program, nuclear weapons. I know what it's like to work on things that have existential risk.

Aaron Levie 6:35 You were the first pacer. [laughs]

Martin Casado 6:37 [laughs] We were part of the first pacers.

Aaron Levie 6:39 [laughs] Yes.

Martin Casado 6:41 So if a constituency within the labs that are the most knowledgeable people believe this stuff has existential risk-

Aaron Levie 6:48 Yes

Martin Casado 6:49 ... the answer is to nationalize it and actually put controls that we know that work.

Aaron Levie 6:54 Mm.

Martin Casado 6:54 Right? Now, if they don't actually believe that, and in the private conversations I have, the most sensible people don't, it's a small fraction that do, this is an HR problem.

Aaron Levie 7:03 Mm.

Martin Casado 7:04 Right? So to me, an HR problem is a company problem. Like, if they are worried that they can't recruit people-

Aaron Levie 7:10 Right

Martin Casado 7:10 ... you know, or they can't retain people, which it seems to me a lot of this is just that concern. It's almost more of this kind of researcher currency. If that's the case-

Aaron Levie 7:18 Mm-hmm

Martin Casado 7:18 ... I think that is the wrong reason to cause a national level lockdown on a very promising technology.

Aaron Levie 7:25 Yeah.

Martin Casado 7:25 And so listen, I think the post, again, I think the post is actually very sensible. I think there are real concerns around security. We've had many compute epochs that have real security concerns. I don't think you can ex- you can reconcile discussions on X risk with the proposal that was put out.

Aaron Levie 7:41 Right.

Martin Casado 7:41 You just can't reconcile those two things, and that has always been my primary point.

Aaron Levie 7:44 Right. Right. Okay, unleash. [laughs]

Martin Casado 7:47 [laughs] I've just been holding it back.

Aaron Levie 7:48 [laughs]

Steven Sinofsky 7:49 I'm holding it for you.

Aaron Levie 7:50 So, okay. The, the first thing is you can't have... You... Is there a schedule that they've published that says when all this stuff, whatever bad stuff is gonna happen, is gonna... They haven't. So you can't pace it because nobody knows when it was supposed to finish in the first place.

Martin Casado 8:05 Right. It also, it also just seems disingenuous too.

Steven Sinofsky 8:06 It, it's complete nonsense.

Aaron Levie 8:07 100%, yeah. Totally.

Steven Sinofsky 8:08 Like, you can't claim... It, this is like when the press reports on Apple's latest iPhone is late.

Aaron Levie 8:13 [laughs] From what?

Steven Sinofsky 8:14 The phone that nobody knows exists, that they haven't told anybody about.

Aaron Levie 8:17 Yeah, my Apple Car was very late. [laughs]

Steven Sinofsky 8:19 Yeah. Like, I, I don't understand. Like, in order for something to be slower, you need to know the rate at which it was moving in the first place. So it's all just utter nonsense.

Aaron Levie 8:28 [laughs]

Steven Sinofsky 8:28 And, and you can't escape that.

Aaron Levie 8:30 Yeah.

Steven Sinofsky 8:31 And then the-

Martin Casado 8:31 By the way, by the way, this is another problem that like, again, from the, like, my, my little quibble on PR is nobody believes it anyway, right? And so if that's the thing that-

Aaron Levie 8:41 Wait, which part?

Martin Casado 8:42 The, the pacing, right?

Aaron Levie 8:43 No, but which part they don't believe?

Steven Sinofsky 8:44 That, that they're going to pace?

Martin Casado 8:45 That they're going to pace.

Aaron Levie 8:46 I see. Okay. Okay.

Martin Casado 8:47 They, they've been at a dead run. They've raised more money than ever before.

Aaron Levie 8:49 Uh-huh.

Martin Casado 8:49 They've run faster than ever before.

Aaron Levie 8:50 Uh-huh.

Martin Casado 8:50 There's no indication that they're pacing. So if this is what you're gonna hang your-

Aaron Levie 8:53 I see, I see. Yeah. Well, well, the internal, the issue obviously is that the models that, that everybody has internally far exceed what anybody else has access to. So-

Martin Casado 9:00 Well, that's part of the problem

Aaron Levie 9:01 ... it's, it's really just the pacing of external releases. But again, back to your point, pacing versus what?

Steven Sinofsky 9:06 Right. Like, we don't know if the one that scares everybody also doesn't, you know, work for a legal brief.

Aaron Levie 9:10 Mm.

Steven Sinofsky 9:11 Like, it might actually screw all that stuff up because-

Aaron Levie 9:13 Right

Steven Sinofsky 9:13 ... it's so... Who knows?

Aaron Levie 9:13 Totally. Right.

Steven Sinofsky 9:14 So there's that. And it's just, it's just disingenuous to claim that you're pacing. Second, why do they have to announce all of this and ask the government to tell them to pa- Like, that's the part that starts to go, "Well, this is really spooky." If you are the most afraid of how everything is gonna go because of your product, just stop.

Martin Casado 9:31 Don't do it. Yeah.

Steven Sinofsky 9:32 Like, like I, like this is their... Like, I worked at, at, at a missile factory in college, and we had nuclear missiles, and I walked the floor.

Aaron Levie 9:39 What's with you guys and missiles? I don't know. I'm just here with software.

Martin Casado 9:43 It's a phase. It's a phase.

Aaron Levie 9:43 Okay. I get it. I get it. [laughs]

Steven Sinofsky 9:46 You, you were one of two people in college-

Aaron Levie 9:47 Yeah

Steven Sinofsky 9:47 ... and we were, which was either you were protesting to keep them off campus-

Aaron Levie 9:51 Okay. [laughs]

Martin Casado 9:51 Or building them.

Steven Sinofsky 9:52 Or building them.

Aaron Levie 9:53 [laughs] Yeah.

Steven Sinofsky 9:53 So that was your choice.

Aaron Levie 9:55 Yeah. [laughs]

Steven Sinofsky 9:56 And, and like, look, man, like, like nuns from the Catholic Church show up and pour blood all over our missiles and-

Aaron Levie 10:03 Oh, totally

Steven Sinofsky 10:03 ... you know, and I'm busy just wheeling PCs around on carts saying, "Here's your secure PC."

Aaron Levie 10:07 This is-

Steven Sinofsky 10:08 And I'm like scared to death. I have no idea what's going on, and then I'm like, "That's a nuclear missile." And then you find out that that's what stopped the Cold War.

Aaron Levie 10:14 Mm-hmm.

Steven Sinofsky 10:14 Like, that literally, it was a Pershing missile.

Aaron Levie 10:16 Right.

Steven Sinofsky 10:16 And that was what did it. And so, but the, the... You said something that I think is super interesting, which is pacing is this sort of fuzzy non-word between like going and not going.

Martin Casado 10:28 Yeah.

Steven Sinofsky 10:29 And, and the, the problem is it, y- you're exactly right. Like, the, it... There's, no one is gonna be happy with the middle road.

Aaron Levie 10:36 Mm.

Martin Casado 10:36 Right.

Steven Sinofsky 10:36 And so one of the things that, that people, I think it's almost fun for me to watch as a sport, they think that they're, that all these people saying to the government, "Do this, don't do this," that they think they're gonna get what they want. And it's a complete 100% misunderstanding of how government works.

Aaron Levie 10:51 Mm.

Steven Sinofsky 10:51 Which is you, when you talk to the government, they actually know how these things work, and they know they're just listening to you, and they're listening to everyone, and no one is gonna get what they want. Because they know in order to do anything in our system, it's a compromise. And so everything that, that all the inputs can make it into the government-

Aaron Levie 11:12 Yep

Steven Sinofsky 11:12 ... the output never makes everyone happy.

Aaron Levie 11:15 Right.

Steven Sinofsky 11:15 100% of the time. It either doesn't go far enough or it goes way, way too far.

Aaron Levie 11:21 Yeah.

Steven Sinofsky 11:21 And, and so y- you can't take the view that you're gonna talk to the government and talk your way into the solution you want. So the bottom line is if they're asking for pacing, they're gonna get the wrong velocity.

Martin Casado 11:32 Well, my favorite thing is like, is, was it David Sacks was like... I don't, I don't... I think it was David Sacks, but w- someone from the government said, "Uh, you're asking us to regulate you. No." [laughs] So like basically the answer was no.

Steven Sinofsky 11:46 Well, but the thing that-

Aaron Levie 11:47 That was probably just Trump, I think. But, uh-

Steven Sinofsky 11:49 But the thing that they know now-

Aaron Levie 11:50 Yeah

Steven Sinofsky 11:50 ... that, that you just know from experience is once the wheels start on regulating-

Aaron Levie 11:55 You can't slow that one down.

Martin Casado 11:56 No.

Steven Sinofsky 11:56 You, you... And now it's become an election issue for every party in every jurisdiction up and down the whole government stack.

Aaron Levie 12:01 Yeah, yeah.

Steven Sinofsky 12:02 So there is, there's now this- This whole basket goods, basket of, of regulatory approaches

Aaron Levie 12:07 Well, the, the

Aaron Levie 12:08 next election will 100% be a referendum on, on AI. So, like it has to happen that, that 2028 is like the AI election, and you could basically run on... The problem is, is like it's not obvious what- who would run on the pro-AI story because it would be, it's gonna be too nebulous-

Steven Sinofsky 12:25 No

Aaron Levie 12:25 ... to tell that story. So then it's just, it's just basically varying degrees of how much do you regulate it or at least try and like avoid the topic.

Steven Sinofsky 12:33 Yeah.

Aaron Levie 12:33 But it is, it is too bad that we've, we as a country are in a spot where like the pro-AI case is just like, it sounds too-- It's like, it's just like takes too many words, you know, it's way too nuanced.

Steven Sinofsky 12:45 Well, no, it's defensive.

Aaron Levie 12:46 Uh, yeah, it's defensive.

Steven Sinofsky 12:46 It's defensive-

Aaron Levie 12:47 Yes

Steven Sinofsky 12:47 ... and we don't, and we, we own none of the vocabulary.

Aaron Levie 12:49 Right. Right.

Steven Sinofsky 12:50 So like the whole debate is, is, is pause, it's, it's swarms, it's rogue. Every word-

Aaron Levie 12:57 Mm-hmm

Steven Sinofsky 12:57 ... has been chosen by the people who don't wanna do AI.

Aaron Levie 13:00 Yeah.

Steven Sinofsky 13:00 And so it means the first thing you have to do is invent new words and say that their words are wrong, which takes so many words that-

Aaron Levie 13:06 Yeah, so we gotta be like union jobs and, and, you know, cancer and like, you know, there, we need, you know, there needs to be another word cloud that emerges.

Martin Casado 13:15 I mean, what I, what I don't, what I don't understand is why the labs have not taken a position on X risk. Like short of that-

Aaron Levie 13:21 Yeah

Martin Casado 13:21 ... I don't think this goes in any direction other than heavy-handed regulation.

Aaron Levie 13:25 Yeah. Well, the-

Martin Casado 13:25 It would, it would just be negligent of the government to be like, "There's a 10% chance of species extinction." The CEO of the top company says he agrees with it. Like h-how can a government not do heavy-handed regulation?

Aaron Levie 13:37 Well, but what, who, but what would anybody real- knowing the, knowing this, this ecosystem though, I don't know that you would be able to pin anybody down on that other than something-

Martin Casado 13:46 No, I would say Dario said it on-

Aaron Levie 13:47 No, but I'm saying you're not gonna pin anybody down on a lower number.

Steven Sinofsky 13:49 Well, well, in fact, he, he, he does the worst thing about it, which is he agrees with people who claim that they believe that there's an X percentage of extinction happening.

Aaron Levie 13:59 Yeah.

Steven Sinofsky 13:59 But he w- he specifically goes out of his way to say, "I'm not gonna put a percentage on it."

Aaron Levie 14:03 Mm-hmm.

Steven Sinofsky 14:04 Which I just think is the weirdest-

Aaron Levie 14:05 No, but that, that, to, to be fair, to be fair-

Steven Sinofsky 14:08 No.

Aaron Levie 14:08 Okay, no? [laughing] Okay. Nobody can be fair. Uh, to be fair, the, that's, like that's, uh, it's not 100% like, um, uh, like disingenuous or whatever. Like, like he probably doesn't specifically agree that it's 10%, or maybe he does and it's just too scary if he were to say it.

Steven Sinofsky 14:25 Well, let's just play binary search.

Aaron Levie 14:26 But-

Steven Sinofsky 14:26 I mean, is it more or is it less? [laughing] At least give us-

Aaron Levie 14:30 But, but the whole thing is a made up... You do know it's a made up concept that we just created. But like nobody can... You can't quantify any of this.

Steven Sinofsky 14:36 Well, but that's sort of Martine's point.

Aaron Levie 14:38 Well, no, but, but you just had an official position though. So the only official position would be, like I think the only official position you could possibly get would be the PR version of this that would be, the only thing that would be intellectually honest would be, "There are real risks with AI. There's incredibly positive, uh, benefits as well."

Martin Casado 14:53 I, I don't-

Aaron Levie 14:53 "We are working to mitigate the risks so they are as reduced as humanly possible."

Martin Casado 14:57 I don't think that's-

Steven Sinofsky 14:58 No.

Martin Casado 14:58 I don't, I don't think that's true. Listen, so we've been through-

Aaron Levie 15:00 Wait, what do you think the-

Martin Casado 15:01 We, we've been through multiple epochs of technology.

Aaron Levie 15:03 Yeah.

Martin Casado 15:03 We've been through compute, we've been through the internet, we've been through the web, we've been through social networking. We had a discussion about the risks without talking about X risk, right? So for example, one thing you can say is, "We do not think the marginal risk for species extinction is different-

Aaron Levie 15:17 Is high. Right

Martin Casado 15:17 ... than it is-"

Aaron Levie 15:18 But what if they do think it is higher?

Martin Casado 15:19 Well, then we should [laughing] nationalize risk.

Aaron Levie 15:21 Okay, okay, okay. Yeah, but like I think the answer is they do think it's more than just the internet.

Martin Casado 15:26 Well, we, we have one of two options.

Aaron Levie 15:27 Yeah. A-

Martin Casado 15:27 You believe that we're gonna go extinct-

Aaron Levie 15:29 Right. Yeah

Martin Casado 15:29 ... and we shut it all fucking down, like shut it down, or-

Aaron Levie 15:31 No, they believe that there's just a chance that we could. [laughing]

Steven Sinofsky 15:34 Speaking as-

Martin Casado 15:34 10%.

Steven Sinofsky 15:35 Speaking as cold-

Aaron Levie 15:36 Dario thinks less. [laughing]

Steven Sinofsky 15:38 Like, like speaking as cold warriors, no one-- Like the Pentagon, you know, they ran a lot of simulations on the chances for nuclear war, great movie "War Games" about the whole thing-

Aaron Levie 15:47 Yeah. Yeah

Steven Sinofsky 15:47 ... all of that. But the thing was, it was, it was non-zero.

Aaron Levie 15:52 Yeah.

Steven Sinofsky 15:52 And so once you said it was non-zero-

Aaron Levie 15:54 Can they say non, can they say non-zero then? Is that allowed?

Martin Casado 15:56 I think as soon as you think it's non-zero-

Steven Sinofsky 15:58 But, but, but-

Aaron Levie 15:58 The problem is, is if you say non-zero, that could be like, you know, shit, man. The guy thinks 93%.

Steven Sinofsky 16:02 Wait, wait, wait. Let's, let, wait, wait, so sorry. Just so we're all having a think conversation.

Aaron Levie 16:05 Yes.

Steven Sinofsky 16:05 Let's talk about marginal risk here.

Aaron Levie 16:06 Yes. Yeah.

Steven Sinofsky 16:07 We're not talking about absolute risk.

Aaron Levie 16:09 Right.

Martin Casado 16:09 Yes.

Steven Sinofsky 16:09 Right? Okay. It's just that I think if, it, once you say it's non-zero, the only ans- of, of like catastrophic-

Aaron Levie 16:17 For marginal, for, for marginal. Yeah

Steven Sinofsky 16:17 ... the only answer is you have to nationalize it.

Aaron Levie 16:20 Yeah.

Martin Casado 16:20 Yes.

Steven Sinofsky 16:21 And so-

Aaron Levie 16:21 Right

Steven Sinofsky 16:21 ... what he's trying to, what the, the labs in general that have that view are trying to thread is they want it to be non-zero as a license to do a certain set of things without the burden of just becoming a nationalized-

Aaron Levie 16:34 Well, well, well, do, do you have-- I, I actually don't know all of the precedents. You hopefully do, but, but like there must be some things that are non-zero, uh, let's say X risk, X risks that are not particularly nationalized, but the, but the regulatory environment around them is so heavy that it might as well be nationalized because of the KYC requirements on-

Steven Sinofsky 16:53 Well-

Aaron Levie 16:53 ... like, like, like I'm sure like to develop anthrax you have to go to a particular kind of lab that-

Steven Sinofsky 16:58 Well, BSL4 labs-

Aaron Levie 16:59 Okay

Steven Sinofsky 16:59 ... are, but they're nationalized.

Martin Casado 17:00 They're nationalized.

Aaron Levie 17:01 They are nationalized. Okay. Uh, yeah.

Martin Casado 17:02 S- S risks tends to be nationalized.

Aaron Levie 17:04 Yeah. Yes.

Martin Casado 17:05 Like normal human safety, not so much, right?

Aaron Levie 17:09 Mm-hmm.

Martin Casado 17:09 I feel like industry oversight that over the time becomes federal regulated.

Steven Sinofsky 17:12 Right. And the progression, which I think is just super important to this discussion, has been since the, uh, post-World War II era, most industries that are critical to the infrastructure, power and banking and healthcare, the trend has been to basically be nationalized.

Aaron Levie 17:31 Yeah.

Steven Sinofsky 17:31 By, by just like what you, like your examples of KYC and-

Aaron Levie 17:34 Yeah

Steven Sinofsky 17:34 ... all of the other stuff. W- the banks are, for all practical purposes, nationalized, and the financial crisis-

Aaron Levie 17:39 But, but okay, but, but you said all practical purposes.

Steven Sinofsky 17:42 Right.

Aaron Levie 17:42 They're literally not nationalized.

Steven Sinofsky 17:43 Right. But-

Aaron Levie 17:44 So, so maybe that, but this might be the intent of the labs is to look like JPMorgan or look like Verizon, and it's just like we're critical infrastructure. We get re- heavily regulated. It's not good for open source or at least frontier open source, but it's a, it is like, it's st- it's like a plausible outcome for this industry within the limits.

Steven Sinofsky 18:01 Right, but it's not necessarily good for innovation is the problem.

Aaron Levie 18:03 No, 100%. 100%.

Martin Casado 18:03 Well, I, I think even that's fine. Just don't use species extinction as-

Aaron Levie 18:06 Sure, as your REA. Yeah.

Martin Casado 18:07 This literally is like the difference between the things that are like stuck in the lab and... Let me just say something. I actually think the labs are moving in the right direction.

Aaron Levie 18:15 Yeah.

Martin Casado 18:15 I actually think- The actual statement was really good. You know, in my discussions with, you know, executives and leaders, like they understand that they have like this tension and they're gonna reconcile it. So I actually am quite optimistic that the labs are both doing the right things and trying to do the right thing. I just think that grew so fa- grew so fast-

Aaron Levie 18:34 Yeah

Martin Casado 18:34 ... and they're just trying to figure out how this machinery works, and I, I think Sinofsky really hit the nail on the head.

Aaron Levie 18:39 Me too.

Martin Casado 18:39 The political process is its own, is its own thing.

Aaron Levie 18:41 It is.

Martin Casado 18:41 And I don't think, I don't know if it's naivety or hubris, but I just don't think that they kind of know how to navigate it.

Aaron Levie 18:46 Well-

Steven Sinofsky 18:46 Well, I think the tech industry has literally over 100 years consistently relearned the lesson at each technology wave it, that, that we don't understand the regulatory climate-

Aaron Levie 18:58 Yeah

Steven Sinofsky 18:58 ... and we can't navigate it. And even the companies like AT&T and IBM that were born out of basically being government monopolies from the start never figured it out.

Aaron Levie 19:07 Yeah.

Steven Sinofsky 19:07 Both got sued for antitrust, both got substantially and structurally changed-

Aaron Levie 19:11 Yeah

Steven Sinofsky 19:11 ... as a result. And they had hundreds of lawyers in the 1960s navigating them, and that, you know, Microsoft came along. Like, we were just like, "What?"

Aaron Levie 19:20 Yeah.

Steven Sinofsky 19:20 Like, we had no idea what was happening to us. And there's, you know, there's Bill Gates playing golf with Bill Clinton, and then we get slapped with an antitrust lawsuit from his administration.

Aaron Levie 19:29 [laughs]

Steven Sinofsky 19:29 And Bill was like, "I was playing golf. Here's the picture."

Aaron Levie 19:32 Right.

Steven Sinofsky 19:32 And that doesn't help. And like, isn't that what I was supposed to do was go and play golf? That's a shortened version of the whole story.

Aaron Levie 19:37 Yeah.

Steven Sinofsky 19:38 But I, I thi- and I think it's just, it's, it, I always use this example, which is the Hollywood got together during the, the Red Scare and all, and, and censorship when they were worried about the government censoring movies for, for, for, uh, sexual content, for adult themes. And they all got together and, and of course they would never be able to win in court if they tried to, uh, but they were gonna, threatening to do it. And they got together and they formed the Motion Picture Association.

Aaron Levie 20:03 Right. Yeah. Okay.

Steven Sinofsky 20:04 And movie ratings and all of that, and they policed themselves. And-

Aaron Levie 20:07 So how do you, do you like guys like the-

Steven Sinofsky 20:08 So we haven't done that

Aaron Levie 20:08 ... do you like the FINRA, uh, proposal? Okay.

Steven Sinofsky 20:11 No, because FINRA is-

Aaron Levie 20:12 But no, that's the, that's as close to MPAA as you can get with more teeth.

Steven Sinofsky 20:14 No, it's not because-

Aaron Levie 20:15 Okay

Steven Sinofsky 20:15 ... FINRA is gonna, it, FINRA becomes legislation which comes with direct oversight.

Aaron Levie 20:20 Yeah, I think MPAA might not have as much consequence in like how society functions.

Steven Sinofsky 20:25 Well, it was only the First Amendment.

Aaron Levie 20:26 Yeah. [laughs] Um [laughs]

Steven Sinofsky 20:31 Did I win that? Is that the winner?

Aaron Levie 20:32 I, no, I, I, I, I, first of all, fantastic. Um, but, but I, I still don't know if you won it. Like-

Steven Sinofsky 20:38 No. Okay

Aaron Levie 20:38 ... like I agree free speech is really important, but like, but like, you know, the, like the, uh, I just think-

Steven Sinofsky 20:44 There was no societal risk at the same time

Aaron Levie 20:46 ... I just think what, what's in our movies will be like, we'll survive, like on like a different continuum of-

Steven Sinofsky 20:50 It, it always-

Aaron Levie 20:51 Yeah.

Steven Sinofsky 20:51 Every history is always relative. And, and at the time, being a communist was a really bad thing, and 30% of Hollywood got fired for, for, you know, it was all this stuff.

Aaron Levie 21:01 Yeah.

Steven Sinofsky 21:02 So I, it's always a risk to bring up something in that kinda way 'cause it sounds so dumb, like nobody thinks about movie ratings. And that's because-

Aaron Levie 21:08 Right

Steven Sinofsky 21:08 ... like actually they were ruled basically you can't constitutionally mandate them. So they couldn't regulate them on cable TV, and so we got to grow up with HBO and all this other stuff.

Aaron Levie 21:17 Yeah.

Steven Sinofsky 21:17 But the problem with FINRA is that is a perfect example of essentially nationalizing risk.

Aaron Levie 21:23 Yes.

Steven Sinofsky 21:24 Because even though the banks all pay money into it and that's how it's run and stuff, it's all mandated.

Aaron Levie 21:29 Okay, wait, sorry. You think we're gonna end up with a situation that is better than FINRA?

Steven Sinofsky 21:33 Well, not anymore.

Aaron Levie 21:34 FINRA appears to be the best case scenario, but like-

Steven Sinofsky 21:36 It is, it is the best case scenario

Aaron Levie 21:37 ... okay. But not even anymore. Like, like I do think this, this technology is in, in the limit, again, so powerful, uh, relative to what it c- I mean, com, uh, for what it can deliver that it would be impossible for eventually Congress not caring about that.

Steven Sinofsky 21:51 Well, that's, that's-

Aaron Levie 21:51 Like it's just like not possible

Steven Sinofsky 21:52 ... that's just like the-

Martin Casado 21:53 My question to you guys is at what, at what point-

Aaron Levie 21:55 If it's eventually making every recommendation in your healthcare process and, and it's inside of your medical device as an open weights model, and it's, and it's on, and every trading system for high-frequency trading, there's just, and it's on an airplane.

Steven Sinofsky 22:08 Well-

Aaron Levie 22:08 Like there's no chance that the government doesn't say we need something where FINRA actually is like the probably the best case scenario of what that looks like.

Steven Sinofsky 22:15 Right.

Aaron Levie 22:16 Yeah.

Steven Sinofsky 22:16 And so now that's a, that's absolutely the most crucial point because if you, the, all the people in the Senate now were in the Senate when, when the Communications Act was passed and the lia- and liability was not passed through to ISPs-

Aaron Levie 22:29 Right

Steven Sinofsky 22:29 ... and to social networks. And they sat around. The debate at the time was they would literally say, say to us, "The internet is so big, how did we not have anything to do with it?" And that's why Al Gore gets a bunch of abuse for saying he created it.

Aaron Levie 22:42 Right.

Steven Sinofsky 22:42 Because he-

Aaron Levie 22:43 He-

Steven Sinofsky 22:43 ... he was actually trying to get out in front of that-

Aaron Levie 22:45 Yeah

Steven Sinofsky 22:45 ... and say, "No, the government was instrumental in cr-" And it all backfired because it made it look like he was a crazy person. But, but that, it is absolutely the case that they felt like they did, they missed their chance to be on top of the internet.

Aaron Levie 22:58 With Al Gore being on the positive side to innovation.

Steven Sinofsky 23:01 Like, well, that was their whole point.

Aaron Levie 23:02 Yeah.

Steven Sinofsky 23:02 They supported-

Aaron Levie 23:03 So who is the Al Gore of-

Steven Sinofsky 23:04 There's no one.

Aaron Levie 23:05 Yeah, yeah, yeah. Uh-huh. Uh-huh.

Steven Sinofsky 23:05 There's, there, there's a couple. Well, you, well, Bessem seems to be like that.

Aaron Levie 23:08 Right. Right.

Steven Sinofsky 23:08 The, you know, the defense people sort of want it private but not, which is exactly where they were on the internet.

Aaron Levie 23:13 Right. Right.

Steven Sinofsky 23:14 And so it's very interesting that a lot of this is just this, like Elizabeth Warren's, Senator Warren's tweets were all like, "We missed this for social networking."

Aaron Levie 23:22 Yeah, yeah, yeah.

Steven Sinofsky 23:22 And it's like you-

Aaron Levie 23:23 There's a lot of pent-up energy against tech-

Steven Sinofsky 23:25 Yeah

Aaron Levie 23:25 ... that, that could end up just all siphoning into, into AI regulation.

Steven Sinofsky 23:29 No, that is exactly what it is.

Aaron Levie 23:31 Yeah, yeah.

Steven Sinofsky 23:31 That's what's happening.

Aaron Levie 23:32 Yeah.

Martin Casado 23:32 I think there's another problem, which is we're all trying to predict what's bad, which is not how we've normally done things. Like, by this time on the internet, we've taken out like tens of billions of dollars [chuckles] of like economic... Well, we've caused tens of billions of dollars of economic damage. We've, we had worms that took out 10% of the infrastructure.

Aaron Levie 23:50 Yeah.

Martin Casado 23:50 The, the internet infrastructure-

Aaron Levie 23:51 Yeah

Martin Casado 23:51 ... which was running critical infrastructure. We had hospitals go down.

Aaron Levie 23:53 We really should have blocked the internet in like '97. [laughs]

Steven Sinofsky 23:56 God, this is-

Aaron Levie 23:57 How do we... Yeah, okay.

Steven Sinofsky 23:58 Like this is-

Aaron Levie 23:58 Okay, okay, okay, okay.

Martin Casado 24:00 So we, we had, I mean, I, I remember a time you, you, like I, I remember when you would, you would, you would buy your CD of Windows 95, and by the time it was done installing You would have a, a, a worm potentially. I mean-

Aaron Levie 24:14 Way to go, Steven.

Steven Sinofsky 24:15 No, no. This was-

Martin Casado 24:16 No.

Steven Sinofsky 24:16 It, it, it was, this was the reality of the-

Aaron Levie 24:18 No, totally. 100%.

Steven Sinofsky 24:18 The reality of the PC until 2001-

Aaron Levie 24:21 Yeah

Steven Sinofsky 24:21 ... was you could not install a PC connected to the network without getting infected by the virus.

Aaron Levie 24:26 Oh, viruses were everywhere.

Steven Sinofsky 24:27 For the ti-

Aaron Levie 24:27 Yes.

Steven Sinofsky 24:27 And, and there were, there were two-

Aaron Levie 24:29 Every

Steven Sinofsky 24:29 ... two level, two, two rounds of congressional hearings.

Aaron Levie 24:32 Yeah. Yeah.

Steven Sinofsky 24:33 You had all of the same-

Aaron Levie 24:33 That is actually... Okay, okay. I'll grant you, that's a very interesting point. Uh, this type of zeitgeist in '94 would've, we would probably not have the internet.

Martin Casado 24:42 Listen, listen, listen, listen.

Aaron Levie 24:43 Yeah.

Martin Casado 24:43 I mean, the automotive, the airline industry, and you always have these periods of kind of like teeth cutting-

Aaron Levie 24:47 Yeah

Martin Casado 24:47 ... where like you learn about the dangers and you learn about the technology and the internet. I mean, we were on the ground floor of this.

Aaron Levie 24:52 Yeah.

Martin Casado 24:52 I mean, things were down all the time. There's economic damage all the time. Like, there was like novel really... There was new viruses, there were new worms. They were all over the place.

Aaron Levie 25:00 Yeah. Oh, Y2K, Y2K was gonna destroy the world.

Steven Sinofsky 25:01 But here's the, here's the interesting thing about this.

Martin Casado 25:03 But what, it's just, I just, I just wanna say like, like, so when we created policy, and we did create a lot of policy, it was kind of like with a bunch of very specific data points on what you're trying to do, and so you know that the policy actually fits the, the fact pattern. And in this case, I mean, even when you were talking, you're like, "Well, what if," you know, yada, yada, yada.

Aaron Levie 25:21 Yeah.

Martin Casado 25:21 And like it's very hard to-

Aaron Levie 25:22 Yeah, yeah

Martin Casado 25:22 ... predictive policy when we don't know.

Aaron Levie 25:23 With that.

Martin Casado 25:24 Now, one area we can talk about is there seems to be novel cybersecurity risk. Great. And what is nice about the discourse, and it's actually starting to evolve around that, like you actually hear people from the lab saying, "Novel cybersecurity risk. This is an engineering problem. We're talking about that." So the more this becomes concrete around real identified risks, I think we can all fall in line, but that's not been the discussion to date.

Aaron Levie 25:45 Yeah.

Martin Casado 25:45 That's a very recent thing.

Steven Sinofsky 25:46 Perfect, perfect point because if you look in, in, in 1986, the Computer Crime and Fraud Act got signed. It was out of a very, very specific scenario, which was, uh, two groups of hackers broke into GTE Telemail.

Aaron Levie 26:00 Mm.

Steven Sinofsky 26:01 One, one of them lived in my dorm. Go figure. And-

Aaron Levie 26:04 Sure.

Steven Sinofsky 26:05 No.

Aaron Levie 26:05 It wasn't you.

Steven Sinofsky 26:06 It was the other guy.

Aaron Levie 26:06 Okay. [laughs]

Steven Sinofsky 26:07 He worked, he worked, he worked at Cisco later.

Aaron Levie 26:09 Yeah. [laughs]

Steven Sinofsky 26:11 And, um, and, and the problem was that was 1983, and there was no crime. And it-

Aaron Levie 26:16 Wow

Steven Sinofsky 26:16 ... it took two and a half years for the law-

Aaron Levie 26:19 To figure out, yeah

Steven Sinofsky 26:19 ... to make it through. That made it very, very specific, and that's the law that says you can't use-

Aaron Levie 26:24 So-

Steven Sinofsky 26:24 You can't access unauthorized computer systems.

Aaron Levie 26:27 So I think we'd all agree that, that you... We probably, you know, we, we probably have like 90% of laws already in the applied layer, like you can't hack systems, et cetera.

Steven Sinofsky 26:35 Right, right.

Aaron Levie 26:35 Do you think there's anything that should be from a liability standpoint in the model layer? Which of course then creates-

Steven Sinofsky 26:40 Wait, and now are you making a technical or legal talk?

Martin Casado 26:41 Well, that's a very-

Steven Sinofsky 26:42 Legal. No, the, the legal... Everything, my read of, of Hugging Face, OpenAI-

Aaron Levie 26:47 Yeah

Steven Sinofsky 26:48 ... four, they're all absolutely blatant computer crime acts, except for the fact that there's, they carved out in amendments later that if you're a white hat, it's not illegal anymore. And that was because people kept making mistakes.

Aaron Levie 27:02 Mm. Yeah, yeah, yeah.

Steven Sinofsky 27:02 And, and they didn't wanna go around arresting everybody who was actually trying to make the system better because they messed something up. And so the Justice Department wrote a memo that said, "We're not gonna prosecute for, for this, and we're also not gonna prosecute if you just, like, violate the terms of use-

Aaron Levie 27:17 Right

Steven Sinofsky 27:17 ... of a system versus actually try to breach it."

Aaron Levie 27:19 Uh-huh.

Steven Sinofsky 27:20 And, and so I think the law is ample for this, for the scenario, and, and it was all written, this GTE Telemail was used by NASA and Livermore and all the labs.

Aaron Levie 27:28 Yeah.

Steven Sinofsky 27:29 And so that's why it was, it caught the attention of DC, 'cause it was federal systems that were being broken into. And the problem we have now is, is this, is this rift between the labs and the security community that keeps looking at all their postmortems and coming to two conclusions: sloppy-

Aaron Levie 27:46 Yeah

Steven Sinofsky 27:46 ... and, and you're not complete in what you're telling us happened.

Aaron Levie 27:49 Mm.

Steven Sinofsky 27:50 And so of course the CVE process came about in the 1980s-

Aaron Levie 27:54 Yeah

Steven Sinofsky 27:54 ... the f- the computer virus and vulnerability reporting stuff out of CMU. And, and for years they worked on very structured reporting with obligations and how to... And they're, for some reason they're not using any of that to do this reporting.

Aaron Levie 28:06 Mm.

Steven Sinofsky 28:07 And so there is this, this very basic stuff that I look at and say, "Well, until they're doing that, they really should stop talking."

Aaron Levie 28:14 Mm.

Steven Sinofsky 28:14 Like they shouldn't do a postmortem on a breach that looks like an intern wrote it and it's not a postmortem. It's this selective memory. It looks like exact-

Aaron Levie 28:22 Narrative

Steven Sinofsky 28:22 ... kind of postmortem you do when you hire outside lawyers to investigate some random thing, and you only give them certain stuff.

Aaron Levie 28:29 Mm-hmm.

Steven Sinofsky 28:30 'Cause you don't have to give the lawyers you hire all the information about what happened.

Aaron Levie 28:33 Yeah, yeah.

Steven Sinofsky 28:34 Like where's all the Slack messages? Where's the actual details of, of what happened?

Martin Casado 28:39 Can, can I, can I just interject an annoying aside? So, um, which is, which is-

Steven Sinofsky 28:43 Which is more annoying than what I just did. [laughs]

Martin Casado 28:44 Which is... No, no, this is good. Which is very in line with this, which is, um, I've been in the security com- like the actual cybersecurity community for a long time, and it is, you know, it's always been one of these things where like they just don't like practical solutions, so even if you build like, you know, a secure system, like well what if somebody, you know, shows up-

Steven Sinofsky 28:59 Oh, yeah, yeah

Martin Casado 29:00 ... and like, you know, like, you know, can Russell Crowe can like break the encryption or something like that. And you-

Steven Sinofsky 29:04 No, it's, it's "Mission Impossible." Tom Cruise just crawling through the ceiling.

Martin Casado 29:06 Yeah, no, yeah. There's, there's always like these kinda like whatever. So my favorite thing that happened recently was like Noam Brown-

Aaron Levie 29:11 Yeah, yeah.

Martin Casado 29:11 On some podcast. We've all said stupid stuff on podcasts. I've already said stupid stuff.

Aaron Levie 29:13 Oh, you didn't like this one?

Martin Casado 29:15 No, no, it was great. No.

Aaron Levie 29:16 I thought it was fun.

Martin Casado 29:17 No, it was fantastic.

Aaron Levie 29:17 Okay. [laughs]

Martin Casado 29:18 No, no, no, no.

Aaron Levie 29:18 Okay, okay.

Martin Casado 29:19 I'm setting, I'm setting it up.

Aaron Levie 29:19 Okay, okay, okay.

Martin Casado 29:20 So, so Noam Brown was like, "Listen, you know, uh, uh, you don't know what a superintelligence could do. It could maybe use like the heat of a, uh, CPU to exfiltrate itself to another computer." Which this brought me back to my-

Aaron Levie 29:32 Okay, okay, okay.

Martin Casado 29:32 And I'm like, "I'm very, now I'm very comfortable." [laughs]

Aaron Levie 29:34 Okay, okay, okay.

Martin Casado 29:35 Like great, now I can have endless pointless discussions on this. But what is interesting is you basically have the X-risk people saying something they thought was plausible.

Aaron Levie 29:42 Mm.

Martin Casado 29:42 And then you have like the security people having this kind of endless discussion. And so I think at some level now these communities are being bridged.

Aaron Levie 29:50 Yeah.

Martin Casado 29:50 Which is like, you know, I actually think that was-

Aaron Levie 29:52 Huh

Martin Casado 29:52 ... a very reasonable thing for Noam Brown to say. I think you can pick holes in it, but we all say weird stuff on podcasts. I actually think exfil risk is real. I mean, I worked in secure, um, computing environments that were highly classified that you would... The covert channels were unbelievable.

Steven Sinofsky 30:06 Oh, yeah, yeah.

Martin Casado 30:06 Like you'd actually... I mean, the... So these are very real comments, but like we actually can have a real discourse, and it was the first time I saw really hardcore systems people-

Aaron Levie 30:13 Huh

Martin Casado 30:13 ... having pretty con- like I would say constructive discussion.

Steven Sinofsky 30:16 Well, they were calculating the bit rate and all this stuff.

Martin Casado 30:18 Yeah, yeah. It was great. No, it was a constructive discussion.

Aaron Levie 30:20 Uh-huh.

Martin Casado 30:20 And like, you know, you had like the kinda the, the, the typical X-risk people were engaging. Of course it was, it was Twitter, so it was like a lot of name-calling, this and that, but it was actually, I felt like a real discussion for the first time. So-

Aaron Levie 30:29 Hmm

Martin Casado 30:29 ... I hope we see more of this.

Aaron Levie 30:30 Wow, yeah.

Martin Casado 30:30 I hope we see more You know, you know, cyber-related things, I think the lab should talk more about it.

Steven Sinofsky 30:36 Yeah.

Martin Casado 30:36 I think it'll engage the community, and I think once that happens, we can actually start having a real-

Steven Sinofsky 30:39 Well, Greg, yeah, Greg's been out there a lot more on this topic-

Martin Casado 30:42 For sure, yeah

Steven Sinofsky 30:42 ... um, which has been, which has been good. But I think the takeaway is Noam Brown should be doing more brainstorms on podcasts of, of-

Martin Casado 30:48 I thought it was gr- I thought it, I, I-

Steven Sinofsky 30:49 Yeah

Martin Casado 30:49 ... it was just, it opened people's eyes to the fact that actually there's a lot of risks-

Steven Sinofsky 30:54 Yeah

Martin Casado 30:54 ... in security that most people don't understand.

Steven Sinofsky 30:56 Yeah.

Martin Casado 30:57 And so that means that there's more stuff that, that... Like, you can't make baseline risk, you know, like how does your authentication layer work? You can't just wave your hand and say, "That should go away," and then bring up, "Oh, but, you know, space aliens can invade."

Steven Sinofsky 31:08 Yeah. Mm-hmm.

Martin Casado 31:09 And that's a little bit of what was going on that I felt uncomfortable with. But like-

Steven Sinofsky 31:13 What do, what do you mean?

Martin Casado 31:14 Like, it was sort of like, "But you know, there's also this risk." And, and it's how I viewed that-

Steven Sinofsky 31:18 Of, of the heat thing.

Martin Casado 31:19 But, but at least we're now, we're now in a-

Steven Sinofsky 31:20 Right

Martin Casado 31:21 ... a, a, in a domain we're comfortable. Like, I can talk about entropy and-

Steven Sinofsky 31:24 Oh, yeah, yeah.

Martin Casado 31:24 Like, we can actually have a concrete discussion that's not, "Oh, well, it's super powerful."

Steven Sinofsky 31:28 Right.

Martin Casado 31:28 Like, at least we've reduced it to, like, the laws of physics and the laws of systems.

Steven Sinofsky 31:30 Although-

Martin Casado 31:31 And, and most people did... All, I would say most people had no idea that that kind of risk was real. I mean, like, when I'm walking around the Pershing missiles, I actually had to test the graphics cards in PCs because a DoD requirement is that the screen memory not be sustained when you pull the power.

Steven Sinofsky 31:46 Oh.

Martin Casado 31:46 But not for, for zero time. Like, it, the minute that the power went off-

Steven Sinofsky 31:50 Uh-huh

Martin Casado 31:50 ... the memory image had to go. And if there was, like, a three-second delay-

Steven Sinofsky 31:53 Yeah

Martin Casado 31:53 ... you, you, you, you had... That could be read. Like-

Steven Sinofsky 31:56 Oh, Steve, see, we, we had people that came into our offices and would literally measure the distance of the monitors to each other-

Martin Casado 32:02 Yeah

Steven Sinofsky 32:02 ... because of tempest attacks, which are-

Martin Casado 32:03 Oh, yeah, yeah

Steven Sinofsky 32:04 ... which is 100% a way to use electromagnetic radiation to leak information. I've seen the same thing-

Martin Casado 32:09 Wow

Steven Sinofsky 32:09 ... with spread spectrum from the BIOS. I've seen it from-

Martin Casado 32:11 It's, it's-

Steven Sinofsky 32:11 I've seen it from, from audio speakers. Like, we had to remove the speakers out. We had some very high bandwidth channels.

Martin Casado 32:15 We, we, my, our building, our building, which was security-

Steven Sinofsky 32:16 These are all very real things

Martin Casado 32:17 ... our building had, had just Muzak speakers aimed at the windows-

Steven Sinofsky 32:21 Yeah. Right

Martin Casado 32:21 ... just to produce interference. Like, it was-

Steven Sinofsky 32:23 You need to go run safety at one of these labs.

Martin Casado 32:25 [laughs] What, what, what-

Steven Sinofsky 32:25 This is like, I've never seen you at more excited than heat, you know, based communication.

Martin Casado 32:30 Can I tell you the craziest covert channel I've ever seen? So it turns... Remember the, remember the OCR? I know this is, like, one of my-

Steven Sinofsky 32:34 Yeah.

Martin Casado 32:35 So I-

Steven Sinofsky 32:35 I'm glad that someone's older than me. Not really, but acting it. [laughs]

Martin Casado 32:38 So remember the old CRT? So it turns out if, like, let's say it's night and you're, you're using a, a CRT terminal i- in your room. The lightest thing in the room is actually the pixel that the raster beam is on.

Steven Sinofsky 32:48 Yeah.

Martin Casado 32:48 So most people think it's, like, the glow of the monitor, but it's actually that given pixel.

Steven Sinofsky 32:51 Mm-hmm.

Martin Casado 32:52 So somebody figured out that, like, let's say you're in, like, you know, a hotel room and you're on your computer. If you have something that can sample the color of the window, you can reconstruct the screen.

Steven Sinofsky 33:02 Oh, that's crazy.

Martin Casado 33:03 Right? You just do it at the same hertz that the raster beam is moving. Then somebody else figured out if you can subvert three pixels, you can use those, you know, 'cause it just looks like bad pixels, you can use those to basically send a message. So, like, you could literally, like, sit out in-

Steven Sinofsky 33:15 Right

Martin Casado 33:15 ... whatever, like a parking lot.

Steven Sinofsky 33:16 Do your own SOS.

Martin Casado 33:17 You sample the message and you can... It is a relatively high bandwidth one-way communication.

Steven Sinofsky 33:21 Right.

Martin Casado 33:22 And so, like, what Noam Brown was saying, like, maybe heat is not the way to do it, but that level of sophistication-

Steven Sinofsky 33:26 Right

Martin Casado 33:26 ... is actually real.

Steven Sinofsky 33:27 I, my-

Martin Casado 33:27 Like, that's a, that's a thing.

Steven Sinofsky 33:29 My, I... When I was at the missile factory, like, I had to lock my keyboard up. I, that's really an impolite word, but that's what we call, call some. I had to lock my keyboard up at night because they didn't want the custodians who didn't have clearance walking by and, and just noticing which keys were dirtier or cleaner.

Martin Casado 33:44 Oh, yeah.

Steven Sinofsky 33:44 And I once left it out, and there's, like, a, a note from security, you know, telling me to report to security and pick up my keyboard. Like, the, the guard-

Martin Casado 33:53 Right

Steven Sinofsky 33:53 ... who walked the hallways just took the keyboard that night off my machine, and that's, like, baseline. I was not cleared. I was just that sensitive, you know? I was, like, nothing. I was an intern.

Martin Casado 34:01 The big problem with this conversation is now this is all in the training data of every AI model-

Steven Sinofsky 34:06 Yeah, yeah

Martin Casado 34:06 ... in the future, so w- w-

Steven Sinofsky 34:07 But that's also-

Martin Casado 34:07 Yeah

Steven Sinofsky 34:07 ... but that's also the threat model. This is the thing we should cover.

Martin Casado 34:09 Yeah.

Steven Sinofsky 34:09 Well, that, but that's the, the, the, the threat, the threat model for these things is always you've got a trusted side and an untrusted side.

Martin Casado 34:15 NIST has-

Steven Sinofsky 34:15 Yeah

Martin Casado 34:15 ... NIST has 500-page manuals on every one of these topics.

Steven Sinofsky 34:17 Okay, okay, okay. All right. This is already out there. Okay, good. You gotta get-

Martin Casado 34:19 And the untrusted side, you assume basically an oracle that can do and know everything.

Steven Sinofsky 34:23 Yeah.

Martin Casado 34:23 Uh, and then the, the question is can you get information off the trusted side? Which, by the way, is what Noam was saying, which again, this is actually quite fascinating, so.

Steven Sinofsky 34:28 Which I do think brings up a super interesting point, which I'm gonna bridge to-

Martin Casado 34:31 Please

Steven Sinofsky 34:31 ... which is just that I think the thing that people really aren't wrapping their heads around and are using the language that's really confusing is just that what AI can do is it can try all of those things-

Martin Casado 34:42 Yeah

Steven Sinofsky 34:42 ... in a very short time.

Martin Casado 34:44 Yeah.

Steven Sinofsky 34:44 Yep. And, and it doesn't get tired. It doesn't get bored. And you know, and so but the interesting thing about it is there's a whole layer of security that you now have to go look at every single API, every single service you're running internally on your network as, like... You know, like, nobody thinks that their internal GitHub or their internal Slack or internal finance expense tool is, is vulnerable to a denial of service attack.

Martin Casado 35:06 Mm-hmm.

Steven Sinofsky 35:06 But swarms-

Martin Casado 35:07 Will do it

Steven Sinofsky 35:07 ... look, literally look like a denial of service attack.

Martin Casado 35:10 Yeah.

Steven Sinofsky 35:11 And so now we need a whole layer internally that just is tracking way more about what authentications are being done, what APIs are being done.

Martin Casado 35:18 Yep, yeah, yeah.

Steven Sinofsky 35:18 And, and but that's just, like, now it's just gonna be basic. And, and all the offsec people that are old are, like, mailing me, like, "Why are we explaining to this to everybody?"

Martin Casado 35:26 [laughs]

Steven Sinofsky 35:27 Like, "This is so basic," because nobody did it internally.

Martin Casado 35:30 Yeah.

Steven Sinofsky 35:31 And-

Martin Casado 35:31 Well, you didn't have to worry about it for your people, and that's the, like-

Steven Sinofsky 35:34 But now your person is just-

Martin Casado 35:35 Yeah, exactly

Steven Sinofsky 35:35 ... a piece of software.

Martin Casado 35:36 Yeah, or, and, like, unlimited. And, and the-

Steven Sinofsky 35:38 Has a credit card.

Martin Casado 35:39 Yeah, I mean, we, we, you kind of got by with, with, um, information security, like, to some extent, uh, on the fact that most people will do the right thing 95 to 99% of the time.

Steven Sinofsky 35:51 Oh, wait. Is that-

Martin Casado 35:52 Fine

Steven Sinofsky 35:52 ... the malicious employee is, like, one in 10,000.

Martin Casado 35:54 Yeah, yeah, yeah. So, so, like, so all these systems are basically open to whoever wants to access them or, like-

Steven Sinofsky 36:02 Yeah

Martin Casado 36:02 ... one tap on the shoulder and, and then you have access, and agent swarms completely flip that because they will j- these are just roaming, you know-

Steven Sinofsky 36:10 They're roaming as drones

Martin Casado 36:11 ... drones. Yes. And, and but, like, yeah. Times-

Steven Sinofsky 36:12 They don't get tired. They don't get bored

Martin Casado 36:12 ... times 10,000 and, and they will easily mistake, uh, like a good task for a bad one and, and vice versa. So, so the, the data security in our systems is a, it, this is gonna be a huge upgrade moment. I, I actually

Steven Sinofsky 36:25 Martin, like I think that actually we're gonna need a different access and security model-

Martin Casado 36:29 Yeah

Steven Sinofsky 36:30 ... going forward.

Martin Casado 36:30 Yeah, so-

Steven Sinofsky 36:31 Like where are we gonna go on that? Because the, the model we have is not granular enough-

Martin Casado 36:34 Yeah

Steven Sinofsky 36:34 ... and it's not performant enough-

Martin Casado 36:35 Yeah

Steven Sinofsky 36:35 ... to handle this stuff, so.

Martin Casado 36:37 So I wanna step back and I wanna say like a, a meta con- like a meta point, which is I think this is how these conversations should go. We've identified a novel risk-

Steven Sinofsky 36:43 Mm

Martin Casado 36:43 ... which is like cybersecurity, which we actually have proof points, and now we're talking about solutions. I think the entire discourse around AI can be of that form, and that's-

Steven Sinofsky 36:51 Yes

Martin Casado 36:51 ... the biggest mistake is that's not what it's been.

Steven Sinofsky 36:53 Uh-huh.

Martin Casado 36:53 Like, I think the entire industry and community is very happy to engage exactly like this.

Steven Sinofsky 36:58 Yes.

Martin Casado 36:58 And I have something to say about exactly what you're asking, but I'm saying like, like I think this is where the conversation should be.

Steven Sinofsky 37:02 Yeah.

Martin Casado 37:03 So here's-

Steven Sinofsky 37:03 We're, we're known for having healthy conversations-

Martin Casado 37:05 [laughs] That's right

Steven Sinofsky 37:06 ... that everyone should learn from, so.

Martin Casado 37:07 [laughs] So-

Steven Sinofsky 37:07 That's, that's what we do. [laughs]

Martin Casado 37:08 So, so, so here's the thing. Um, I don't think there's a, a, a technical limitation here. Like these threat models are very well understood and have been in the literature for a long time. I mean, like the operating systems research, the, um, MLS, the multilayer security research has, has, has considered these sorts of things from an academic lens. The reason it hasn't been adopted has just tended to be a usability issue, is I guess it's really hard to, to maintain and you didn't have to. So you could argue that AI solves the usability issue because it's AI is using it. So maybe now is gonna be a renaissance in operating systems and network-

Steven Sinofsky 37:43 Mm-hmm

Martin Casado 37:43 ... and computer languages, and we should like go back to the old research and we should kind of start rebuilding cis- systems that are secure by design. And oh, by the way, if we don't think that, you know, these things are safe to put out, we don't put 'em out until we have these systems built. And, and, and oh by the way, the AI is very smart, so they can help us build it. And so I think again, like computer always evol- Like you know how much of the stack we had to change for the internet?

Steven Sinofsky 38:05 Mm-hmm.

Martin Casado 38:06 Everything. [laughs]

Steven Sinofsky 38:07 Right. Tell me about it. [laughs]

Martin Casado 38:08 And you know, you know how, and you know how not vulnerab- how vulnerable everything was? Like, like we could be in one of those moments like, "Oh shit, we gotta rethink everything."

Steven Sinofsky 38:15 Right.

Martin Casado 38:15 And that's fine. We've done that before. But, but I think that's a conversation we should have. So I agree, it's time to think about evolving these things.

Steven Sinofsky 38:21 Well, like look at like here, here you mentioned earlier like the booting a PC and getting a virus in, in 30 seconds or whatever. So if you look at how like a, you take an iPhone out of the box, which a lot of people are doing this week, you know, the, what happens is it, the whole network is basically shut down except for getting the latest version of the operating system.

Martin Casado 38:39 Mm.

Steven Sinofsky 38:39 Because it does- even though it's was pressed, you know, six weeks ago, you know-

Martin Casado 38:43 There's more, yeah

Steven Sinofsky 38:44 ... there, there some zero day thing has been discovered since. And so actually the whole out-of-box process now involves first step updating, doing an update where the, the device can't do anything else and it can never do anything else until it's updated. That's the k- like there are all these benign-

Martin Casado 38:58 Right. You have to learn

Steven Sinofsky 38:59 ... things that are completely benign-

Martin Casado 39:00 Yeah

Steven Sinofsky 39:00 ... that we turned off in all of this desktop software-

Martin Casado 39:03 Yeah

Steven Sinofsky 39:04 ... of the era that used to be good things. Like it was having a macro for Word so you could build automatic citations or something was like this super cool thing until it became a vir- We had a thing where you could put a CD in and it would just arbitrarily run a program. And so then what somebody did was like, "Oh, well I'm gonna burn a clone of that CD and replace the program with my virus, but it's gonna look like the thing that's supposed to run and it's just collecting all this stuff and being evil."

Martin Casado 39:28 Yeah, I, I-

Steven Sinofsky 39:28 Then we disabled that. And so what, one of the things that's happening right now is we, there's a whole bunch of stuff that happens on-

Martin Casado 39:35 Yeah

Steven Sinofsky 39:35 ... like you, on, on your own Box Corp network that actually-

Martin Casado 39:40 Well-

Steven Sinofsky 39:40 ... is gonna have to just change as the, the standard procedure. Two-factor auth-

Martin Casado 39:43 Yeah

Steven Sinofsky 39:43 ... five years ago was not standard-

Martin Casado 39:45 Yeah

Steven Sinofsky 39:45 ... in most places. The, this whole, your whole s- like your whole SaaS world, like I remember in like 2015 or so when, when you would talk to a new company about their, "Oh, we're gonna do enterprise pricing," or whatever, and then you, they realized the first thing they had to do was go do Okta integration and/or Google Auth because they could not have their own directory of, of how to manage it. And then that just became a thing.

Martin Casado 40:07 Right.

Steven Sinofsky 40:07 And now there's not a SaaS program anywhere that doesn't just launch-

Martin Casado 40:10 Right

Steven Sinofsky 40:11 ... with managed authentication.

Martin Casado 40:12 Right.

Steven Sinofsky 40:13 And, and so there's just so many things that need to happen before you're, you're even software now. And-

Martin Casado 40:19 Yeah. Well, yeah, we're, we're, uh... I mean, the- there's, uh, probably every layer of the stack has to evolve a, a bit on this. Like even the, like lack of granular nature of like, you know, you, you have these modes of like the agent will either ask you every single time, you know, if it, if you wanna, you know, give it permission to do something or the exact opposite of like it can just like delete your entire computer.

Steven Sinofsky 40:40 Right, right.

Martin Casado 40:40 And like our OS probably wasn't built for the right level of, of granular set of, of tools you wanna give the agent. Um, we, we've kinda done a lot of work, work in this space because obviously like, you know, do you wanna give an agent exa- like your entire file system? Probably not.

Steven Sinofsky 40:55 Right.

Martin Casado 40:55 Maybe in some cases you do, but oftentimes you wanna have granular controls of like in this folder you can do read, write, and in that folder-

Steven Sinofsky 41:01 Right

Martin Casado 41:01 ... you can only do read. And, and so how do you kinda make this all intuitive for the user? So it's, it's very difficult.

Steven Sinofsky 41:07 Yeah.

Martin Casado 41:07 And so, so there's gonna like... [laughs] Yeah, exactly. Yeah.

Steven Sinofsky 41:10 What you just said is a very deep comment-

Martin Casado 41:12 I know. Exactly, yeah

Steven Sinofsky 41:13 ... which is basically the conclusion of 40 years of, of cybersecurity.

Martin Casado 41:16 Right. No, 100%.

Steven Sinofsky 41:17 Is like you actually can't make it... But maybe with AI you actually can. Like maybe there is like kind of UA, UI. Well, I, I actually think this is, if you ask me like walking in, like what I wrote down on my note, like was my biggest fear-

Martin Casado 41:26 Yeah

Steven Sinofsky 41:26 ... is that Europe decides that GDPR was the best thing ever.

Martin Casado 41:31 Uh-huh. Right, right.

Steven Sinofsky 41:31 And they're gonna just GDPR AI.

Martin Casado 41:33 Yeah.

Steven Sinofsky 41:34 And, and the AI will be fine. It'll be, have one, it'll have one prompt for when text gets emitted that just says, "This vendor is emitting text, and it's probably wrong, yes or no?" That, that's gonna be... 'Cause you can't really-

Martin Casado 41:47 Right

Steven Sinofsky 41:47 ... at least in North America, you're not gonna censor speech. In Europe, they still will. And, and they'll have filters and keywords and block lists. But then on any verb, any time that an agent or, or, you know, a background agent or a frontline agent touches a third-party product, I, I'm really worried that they're just gonna say, "We need a GDPR prompt on that."

Martin Casado 42:08 Uh-huh.

Steven Sinofsky 42:08 And it's gonna be-

Martin Casado 42:09 Oh, back to the user through the agent

Steven Sinofsky 42:11 ... each single, every single write or every single non-lookup-

Martin Casado 42:15 Yeah

Steven Sinofsky 42:15 ... becomes like a, a, a safety warning, like the airbag thing in your car.

Martin Casado 42:19 Mm-hmm.

Steven Sinofsky 42:19 And, and the regulators love that-

Martin Casado 42:22 Yeah

Steven Sinofsky 42:22 ... because it, it's a liability assignment, and so it has this sort of legal precedent.

Martin Casado 42:26 Yeah.

Steven Sinofsky 42:26 And, and I really worry that that's, that's actually the midgr- middle ground where we're gonna end up. And, and, and unfortunately because the US about 15 years ago stopped leading in tech antitrust, the, the problem is that Europe is gonna lead with that because they have nothing to lose.

Aaron Levie 42:44 Yeah.

Steven Sinofsky 42:44 Like they-- And, and so I don't wanna be sad and down about it, but I, I, I just can't get out of my head that they love prompts.

Aaron Levie 42:52 [laughing]

Steven Sinofsky 42:52 They... I mean, look, I had to put in that browser ballot-

Aaron Levie 42:55 Europeans love prompts

Steven Sinofsky 42:56 ... choice thing.

Aaron Levie 42:57 Yeah.

Steven Sinofsky 42:57 No, they, they, it, it, because it, it, it's the same-- Look, get into a new car-

Aaron Levie 43:01 Yeah

Steven Sinofsky 43:01 ... which I haven't done in years, but it, you know, like, uh, you're-

Aaron Levie 43:04 A lot of prompts [chuckles]

Steven Sinofsky 43:04 ... pulling stickers off. You got, you, you know, you have all of these things.

Aaron Levie 43:07 Yeah.

Steven Sinofsky 43:08 And, and someone thinks that that was success.

Aaron Levie 43:10 Yeah.

Steven Sinofsky 43:10 And it's like, has anybody ever read, like, what is this if you're, if there's a baby in this seat? And it's like, well, that's relevant for some people some of the time, but it's this sticker.

Aaron Levie 43:19 It's the entire, yeah, fabric. [chuckles]

Steven Sinofsky 43:20 It's, like, attached to the, it's attached to the seat.

Aaron Levie 43:22 Yeah.

Steven Sinofsky 43:22 And, and they love... Th-that is a very particular-

Aaron Levie 43:26 Yeah

Steven Sinofsky 43:26 ... thing that they just love. And so I would, I would say if I were an AI now, the one thing I would be trying to avoid i- And look, we added it-

Aaron Levie 43:33 Okay. F-FINRA for AI. We'll, we'll, we'll create that standard.

Steven Sinofsky 43:36 I, I, I mean, look, w-we had to put this, we, uh, on the, on the... When the internet was new, the big thing was to download a program and run it. And of course, if your machine is running in administrator mode, it was download a virus and take all your files forever.

Aaron Levie 43:48 [chuckles]

Steven Sinofsky 43:48 And so with, with Windows, um, XP, which was in 2000, we added this thing that prompted you and stopped.

Aaron Levie 43:55 Yeah.

Steven Sinofsky 43:55 Like, literally your machine stopped, user account control, and like, it was absolute assault.

Aaron Levie 44:00 Yeah.

Steven Sinofsky 44:00 And we also did it in Word. The stupid little macro to help you write your thesis also came with a warning every time you opened your thesis saying-

Aaron Levie 44:07 Yeah

Steven Sinofsky 44:07 ... "This has my macros in."

Aaron Levie 44:08 Everybody would just click through that.

Steven Sinofsky 44:09 And everybody-- And so you, you end up in this world where just like with GDPR, everybody is numb.

Aaron Levie 44:13 Yeah.

Steven Sinofsky 44:13 And so then they're like, "Well, it needs to be bigger."

Aaron Levie 44:15 Although M-Mac, I mean, M-Mac kind of has found a-

Steven Sinofsky 44:17 Nobody downloads software. That's the thing.

Aaron Levie 44:19 Uh-huh, uh-huh.

Steven Sinofsky 44:19 It's a very different usage pattern on Mac. So-

Aaron Levie 44:21 I, I wish-

Steven Sinofsky 44:22 Sorry, I, I was the first one to-

Aaron Levie 44:23 I don't have, like, I don't have, like, 10 applications on my Mac, but [chuckles]

Steven Sinofsky 44:26 Well, that's, but 10 and then you're done.

Aaron Levie 44:28 Yeah, yeah, yeah.

Steven Sinofsky 44:28 And, and it's, but it's, it, a lot of people still do all this stuff. Imagine if instead it was every time you go to a new website-

Aaron Levie 44:35 Yes

Steven Sinofsky 44:35 ... which you do now because-

Aaron Levie 44:36 No, totally. That would be bad. Yeah, yeah.

Steven Sinofsky 44:37 Yeah. To-

Martin Casado 44:38 I mean, to, to, to, to start from here, but to bring it kind of back to the, the macro point, like I wish this was the discussion we were having.

Steven Sinofsky 44:45 Yeah, yeah.

Martin Casado 44:45 Which is like, I feel like we've dealt with a lot of these problems. I feel like when we talk about philosophical X risk, we're not solving these very pragmatic problems. I actually think this is actually a, a, a constructive conversation to have. Maybe prompts would help, I don't know. And I, I think part of the problem is, is like people don't remember like, you know, how, how unfettered access was and how bad it was and just how relatively benign that ended up being. Like, I even remember like, this is at Stanford during our PhD. I remember the oscilloscope was kind of janky, and I'm like, "What's going on with this oscilloscope?" Like it's, you know, it's a little slow. And I was measuring the network traffic and it was like more network traffic than you would expect. Um, and I'm like, "Why is there network..." I didn't even know the thing had a TCP stack. And somebody like broke in 'cause there was an old version of Windows CE and was running a porn server, you know? And so like-

Steven Sinofsky 45:28 I had no idea that.

Martin Casado 45:29 [laughs] It used to be-

Steven Sinofsky 45:30 Just for the record, nothing.

Martin Casado 45:31 It, it, it used to be the case that like anytime you turned over a stone-

Steven Sinofsky 45:36 Yeah, yeah

Martin Casado 45:36 ... somebody had broke into something.

Steven Sinofsky 45:38 Ah.

Martin Casado 45:38 And like, was it this like worst case malicious whatever? And it was actually very rarely, even though the capability was there. And so if, if we could just some-somehow-

Steven Sinofsky 45:48 Mm-hmm

Martin Casado 45:48 ... tone down the rhetoric-

Steven Sinofsky 45:49 Yeah, yeah

Martin Casado 45:49 ... and put it in context, and these still are computer systems, and yes, there's very serious stuff, and people have definitely died because networks have gone down.

Steven Sinofsky 45:57 Yeah.

Martin Casado 45:57 There's been real issues, but like... And then can we just quibble about GDPR? That would be amazing. But the problem is like that's not the discussion. It's not about GDPR and prompts. It's about species extinction and-

Steven Sinofsky 46:08 And philosophy and-

Martin Casado 46:09 And philosophy and, and irrefutable things, and it's just, it's also very-

Steven Sinofsky 46:13 Not constructive. Like I, I, I think that that's such a great point, and I, I think you, you hear people now talk about we regulate airplanes and we regulate cars, and you forget like, well, the first cars were at the turn of the century.

Aaron Levie 46:25 Yeah, yeah.

Steven Sinofsky 46:26 And unsafe at any speed was in the mid 1960s.

Aaron Levie 46:28 Yeah, totally. Mm.

Steven Sinofsky 46:29 And, and, you know, people had been doing, selling pharmaceuticals in the, during the Gold Rush, and thalidomide happened 50, or, you know, 50 or 75 years later, and not even in the US, and then there was the FDA. And, you know, people, the first pilots were flying obviously at the beginning of the 20th century, and it wasn't until the 1920s that you had to get a license to be a pilot, and you literally showed up with your own plane and you got a certificate.

Aaron Levie 46:53 [laughs]

Steven Sinofsky 46:54 If you had... It was like any, it was literally no different than driver's ed is today. And then it was 20 more years until they had anything to do with airworthiness and looking at your plane, but it was, it was minimal. And then it wasn't until way after World War II that they got involved in like what you think of as the modern FAA. And so you're looking at 40 years-

Aaron Levie 47:12 Yeah

Steven Sinofsky 47:13 ... of, of innovation, and they were not moving slow. I mean, have you ever seen that video in black and white of like all the different planes that crashed and everything?

Aaron Levie 47:20 Yeah.

Steven Sinofsky 47:20 That was 20 years after the Wright brothers.

Aaron Levie 47:22 Yeah.

Martin Casado 47:22 Mm.

Aaron Levie 47:22 And the FAA is incredibly effective.

Steven Sinofsky 47:24 And, and, and right. It's incredibly effective.

Aaron Levie 47:25 It's, it's, it's the safest form of transportation, like, you know. And so I, I do think that this process has merit.

Steven Sinofsky 47:30 It's also the slowest, most difficult form of innovation. And so if you start the F- if you had started the FAA in 1910-

Martin Casado 47:36 You would never have, you would never have commercial jets. You would never have-

Aaron Levie 47:39 Well, I, I was, I was, uh, listening to a Nick Bostrom, uh, podcast, uh, just a couple days ago. No, no, I, I, it was, I, I-

Martin Casado 47:46 Oh, he's turned. He sees.

Aaron Levie 47:47 It was interesting. It was interesting. And, and, but no, but, but, but he actually makes this point. If you regulate AI too early, you actually basically don't solve anything and you, you, uh, you, you still just kinda have the same risk ultimately, but you don't understand the systems well enough.

Martin Casado 48:00 Yeah, you, you've invented the, you willed the thing into, to, to being-

Aaron Levie 48:03 Yeah

Martin Casado 48:03 ... but you haven't figured out how to control it.

Aaron Levie 48:04 Yeah.

Steven Sinofsky 48:04 Well, we don't even figure out what it actually is yet. Like, like there are new things coming out all the time and like casting AI completely differently-

Aaron Levie 48:12 Right

Steven Sinofsky 48:12 ... as, uh, uh, than we thought of just six or nine months ago. And I think that that, like all the innovation that's gonna happen at the application layer is gonna cause things to move in and out of the models-

Aaron Levie 48:22 Yeah

Steven Sinofsky 48:22 ... in different ways. And like we're, we, you know, we, we thought up until last week, I think, that, you know, text prompts and text coming back was gonna be the best way to interact with-

Martin Casado 48:32 I know, then Jeff shows up.

Steven Sinofsky 48:33 And then Jeff shows up. [chuckles]

Martin Casado 48:34 He's so good too. I mean, it's just like, and then-

Steven Sinofsky 48:36 So talk about that-

Aaron Levie 48:37 Yes

Steven Sinofsky 48:37 ... 'cause I think it's super important.

Aaron Levie 48:37 Why you find

Aaron Levie 48:38 it so remarkable, yeah.

Steven Sinofsky 48:39 Well, okay. So, so the way I think about it is, um-

Martin Casado 48:42 So, okay, so LLMs were kind of text in, text out, right? They generate text. And, and, and they came from chat, right? It was to communicate with a human, and we've spent the last few years ti- trying to take this thing that spits out text and cram it into a, a traditional program.

Steven Sinofsky 48:54 Mm.

Martin Casado 48:55 Right? And but traditional programs don't really speak text, right? And so then you end up doing this janky thing-

Steven Sinofsky 49:00 [chuckles]

Martin Casado 49:00 ... where you're like, in the prompt you're like-

Steven Sinofsky 49:02 Here's the schema and-

Martin Casado 49:02 Here's the schema.

Steven Sinofsky 49:03 Yeah, yeah.

Martin Casado 49:04 But like the thing is generating text-

Steven Sinofsky 49:05 Yeah

Martin Casado 49:05 ... and it kind of ignores it, and it's just been super janky. And so what, what Jeff basically said is that, "Listen, um, you know, generating the text on the out- is, is a very expensive thing, but it's also kind of, you know, it's, it's more complicated than you need. So why don't we, we, we'll, we'll, we'll read text and we'll have all of that kind of knowledge to read the text, but then rather than generating text, which is very expensive, we will just, if you give us a set of options, we'll choose the best option." We can do that incredibly, um, we can do it incredibly fast, incredibly cheaply, uh, but also we can do it with much more accuracy 'cause we can train just for this. And so for all of the use cases that are not talking to, to like a, a chatbot, but are actually trying to put it in traditional software, this is a great fit. And so this has probably been the fastest adoption of an AI model since ChatGPT. It's just been remarkable because we're all primed for this.

Steven Sinofsky 49:51 I just wanna pile on, on this one 'cause I, I can't tell you how much I love seeing this exact form of, of, of innovation and because what it's does is it does the thing that's bugged me from the very beginning, which is there's been no user study ever that shows like interacting with a computer using, um, full natural language is efficient.

Martin Casado 50:11 Hmm.

Steven Sinofsky 50:12 It's, it's like literally always the least efficient way, and it's very simple, and it, it's just like ask yourself, how many people are asked, are really, really good at asking questions? And immediately that's like less than half the people can ask a good question in a meeting.

Martin Casado 50:25 Yeah.

Steven Sinofsky 50:25 And then how often do you look at the answer and get really frustrated before it's finished, but you have to pay all this money to watch the seven paragraphs come out and then apologize-

Martin Casado 50:33 [chuckles]

Steven Sinofsky 50:33 ... that it's only a little? And so that's one, like having a different model. And then the other, of course, is my favorite, which is the output of it is designed for probabilistic programming.

Martin Casado 50:43 Yeah. Yeah.

Steven Sinofsky 50:43 And so instead of saying like, "Is this a customer service question?" Then route to customer service, otherwise route to general help desk or whatever. It's like, "Well, this is 80% customer service."

Martin Casado 50:54 Mm-hmm.

Steven Sinofsky 50:54 And that's exactly simulation.

Martin Casado 50:57 Mm.

Steven Sinofsky 50:57 And it turns out there's like 50 years of computer science research in literally like probabilistic if statements.

Martin Casado 51:02 Mm.

Steven Sinofsky 51:03 And so suddenly the coolest place to be in computer science is gonna be in probabilistic programming-

Martin Casado 51:08 Mm

Steven Sinofsky 51:08 ... which was like all of computer science in the 1960s and '70s. So it was basically how do-- Because all of computers started with doing math, and it was all simulation.

Martin Casado 51:18 Mm.

Steven Sinofsky 51:18 So it was like, let's launch the missile and hit that target, but it's windy, but wind isn't constant, so like let's model the wind and decide where to put the thrusters in order to do the arc. And so most programming through like say 1970 before it got to accounting was probabilistic.

Martin Casado 51:37 And then we ruined everything.

Steven Sinofsky 51:38 No, it-

Martin Casado 51:38 Yeah

Steven Sinofsky 51:38 ... in accounting, there's no probability-

Martin Casado 51:40 Right

Steven Sinofsky 51:40 ... in accounting.

Martin Casado 51:40 Right.

Steven Sinofsky 51:41 Uh, but, but most programming was basically this modeling kind of thing, and so most programming language design was trying to figure out how to put probability into if statements or into-

Martin Casado 51:52 Mm

Steven Sinofsky 51:52 ... while loops. Like do this-

Martin Casado 51:53 Yeah

Steven Sinofsky 51:53 ... until something happens maybe most of the time.

Martin Casado 51:56 [chuckles]

Steven Sinofsky 51:56 And, and like so my first CS class, like the f- very second assignment or so was a simulation about like waiting in line at a store. And I didn't know it at the time, I actually looked all this up when I was reading about Jeb, which was like the, the whole thing was my, my professor was like wrote the book called "The Theory of Simulation" in like 1960. And I, I just didn't know that-

Martin Casado 52:15 Huh

Steven Sinofsky 52:15 ... because it was kinda died by the '80s-

Martin Casado 52:18 Right

Steven Sinofsky 52:18 ... 'cause it was all replaced by hyper... And so this, this notion of probabilistic and, and, uh, that slide deck you shared, um, about the future, what's different about language models and stuff that you said was super good.

Martin Casado 52:29 Oh, Hal Perflex one.

Steven Sinofsky 52:30 Hal, yeah.

Martin Casado 52:30 That was phenomenal. Thomas Del-

Steven Sinofsky 52:31 It was a great deck.

Martin Casado 52:32 Yeah, yeah, yeah.

Steven Sinofsky 52:32 But the part that I felt was missing was that like-

Martin Casado 52:34 Yeah

Steven Sinofsky 52:34 ... oh wait, this is not all new.

Martin Casado 52:36 Yeah, yeah, yeah. For sure.

Steven Sinofsky 52:36 Like, like all of computer science was this probabilistic stuff.

Martin Casado 52:39 Yeah. Yeah, yeah.

Steven Sinofsky 52:39 And so it's gonna be very interesting to dust off all of that work because it's exactly what's going on. Like it's not-

Martin Casado 52:45 Yeah

Steven Sinofsky 52:45 ... an if statement now is if X percent-

Martin Casado 52:48 Uh-huh

Steven Sinofsky 52:48 ... not if always. And so the way that, that Jeb worked is just to like you, you basically it, it's a custom programming language almost, which is here's the-

Martin Casado 52:57 Right

Steven Sinofsky 52:57 ... prompt, come back with a percentage.

Martin Casado 52:59 Yeah. That's right.

Steven Sinofsky 53:00 And, and then you put that

Steven Sinofsky 53:01 in the if statement.

Martin Casado 53:01 But the consequential thing is like finally we have a way to integrate these language models into a traditional-

Steven Sinofsky 53:07 New software. Right

Martin Casado 53:08 ... software. And, and I think it's kind of funny 'cause it, like you asked the question like why haven't the labs done this, right?

Steven Sinofsky 53:12 Right.

Martin Casado 53:13 And it, [chuckles] it's kind of like a r- like not an indictment, but a reflection on how they think. Like they're trying to create beings, and beings speak.

Steven Sinofsky 53:19 Mm.

Martin Casado 53:19 If you're trying to create God, God speaks in natural languages or whatever, where this is really about something that's for traditional software, which is kinda not the direction that they've been taking. But one of the reasons the uptick has been so dramatic is because a lot of us software people have been trying to integrate these models into software-

Steven Sinofsky 53:36 Mm

Martin Casado 53:36 ... it just hasn't. So even before you get to the probabilistic, like if I want a language model to drive an if statement, like it's really hard today-

Steven Sinofsky 53:43 Yeah, yeah

Martin Casado 53:43 ... with this model, it makes it much, much, much easier. And then of course this could change the nature of software fundamentally to make it more stochastic over time.

Steven Sinofsky 53:50 Well, I think, but I, I, absolutely, and I think that what's so cool is that, that it is happening outside the models because that's what I think is just gonna happen, which is the center of innovation has just moved.

Martin Casado 54:02 Yeah.

Steven Sinofsky 54:02 And it's just, and now people need to... Like it turns out that the plat-

Martin Casado 54:05 Oh, you mean outside of the lab, the big-

Steven Sinofsky 54:07 Right

Martin Casado 54:07 ... the front ends, yeah.

Steven Sinofsky 54:07 The platform providers-

Martin Casado 54:08 Yeah, yeah, 100%

Steven Sinofsky 54:09 ... you know, basically reach a critical, a point of critical mass where the innovation stops happening at the platform layer.

Martin Casado 54:15 Mm.

Steven Sinofsky 54:15 And, and then, you know, Apple, there's this famous expression in the Apple community called Sherlocking, where Apple looks around and the, the things from the outside world become features and people complain and it's a real... But that's sort of how the innovation works because once you're a platform, you're overwhelmed, no matter how many people you add, you're overwhelmed with just keeping the thing running and compatibility and stuff like that. And so I think that this is the signal that now people have figured out that there's innovation to be done-

Martin Casado 54:43 That's awesome

Steven Sinofsky 54:44 ... to the model-

Martin Casado 54:44 Yeah

Steven Sinofsky 54:44 ... but outside the model.

Martin Casado 54:46 Yep. Yep.

Steven Sinofsky 54:46 Cool.

Erik Torenberg 54:47 Guys, thanks for coming. This was a great episode.

Steven Sinofsky 54:48 Okay. Sounds great. Thank you.

Martin Casado 54:48 Okay, great.