Aaron Levie and Steven Sinofsky on the AI-Worker Future

Erik Torenberg 0:00 We thought that we were looking at the form factor of AI, which is you're talking back and forth to something. The real ultimate end state of AI, and thus AI agents, is these are autonomous things that run in the background on your behalf and executing real work for you. The more work that it's doing without you having to intervene, the more agentic it's becoming.

Martin Casado 0:17 Somehow it produces output that it feeds back into itself.

Steven Sinofsky 0:20 It's literally just the ampersand in Linux, which is-

Martin Casado 0:24 [laughs]

Steven Sinofsky 0:24 ... it's a background task.

Erik Torenberg 0:25 [laughs]

Martin Casado 0:25 Okay.

Steven Sinofsky 0:26 And it's like the worst assistant in the world. And agentification is just hiring a lot of these really bad interns.

Erik Torenberg 0:33 [laughs]

Erik Torenberg 0:36 I thought we'd- I'd start this wide-ranging podcast by asking the very simple but very provocative question, what is an agent?

Martin Casado 0:42 Oh, boy. To who?

Erik Torenberg 0:44 Steven.

Martin Casado 0:44 Okay, Steven. That's yeah-

Steven Sinofsky 0:45 Oh, to me?

Martin Casado 0:45 Yeah, exactly. Steven.

Steven Sinofsky 0:46 Oh, oh, I go first. So, so I, I actually have a very old person view of what an agent is, which is it's literally just the ampersand in Linux.

Martin Casado 0:55 [laughs]

Steven Sinofsky 0:55 Which is, it's a background task.

Erik Torenberg 0:57 [laughs]

Martin Casado 0:57 Okay.

Steven Sinofsky 0:57 Because, like, you type something into o3, and then it's like, "Hey, t-t-t-t-t-t- I'm trying this out. Oh wait, I need a password. Can't do that." And it's, like, the worst assistant in the world, and really it's just 'cause they need to entertain you while it's taking a long time-

Erik Torenberg 1:11 [laughs]

Steven Sinofsky 1:11 ... to answer your prompt. And so that's my old person view of what an a- And agentification is just hiring a lot of these really bad interns.

Erik Torenberg 1:20 [laughs]

Martin Casado 1:21 [laughs]

Erik Torenberg 1:22 The intern, they're getting better. The-

Steven Sinofsky 1:23 They are getting better.

Erik Torenberg 1:24 The intern, they're getting better.

Steven Sinofsky 1:24 But they still don't remember if I have a password to nature, you know. Like, it's just-

Erik Torenberg 1:28 Is it possible you guys just had bad interns in, like, the '80s and '90s?

Steven Sinofsky 1:31 It, we, we had-

Erik Torenberg 1:32 Okay

Steven Sinofsky 1:32 ... we had terrible interns.

Erik Torenberg 1:33 Okay, okay. I have, like, a very high esteem for interns, so.

Steven Sinofsky 1:35 [laughs]

Erik Torenberg 1:36 Like-

Steven Sinofsky 1:37 But now a real answer. [laughs]

Erik Torenberg 1:38 No, no,

Erik Torenberg 1:39 I mean, uh, I, I think, I, I think collectively we're seeing what, what these are becoming. So if you think about two years ago, the, you know, post-ChatGPT moment, we, we thought that we were looking at the form factor of AI, which is you're talking back and forth to something. And I think to Steven's point, the real, you know, ultimate end state of AI, and thus AI agents, is these are, these are autonomous, you know, uh, uh, things that run in the background on your behalf and executing real work for you. And you're ideally in a, an ideal world interacting with them actually relatively little, m- uh, relative to the amount of value that they're creating. And so, so there's some kind of, you know, metric where the more work that it's doing without you having to intervene, the more agentic it's becoming and, and I think that's, that's sort of the paradigm that we're seeing.

Martin Casado 2:22 Yeah, the only addition I'd have in addition to long-running, which I agree, is that somehow it produces output that it feeds back into itself-

Erik Torenberg 2:30 Mm

Martin Casado 2:30 ... as input, which you can actually do long-running inference. Like, you can make a video that's really long-running-

Steven Sinofsky 2:36 Right

Martin Casado 2:36 ... but it's just basically a single shot video, and you just throw more compute at it. I, I think there's, like, technical limitations, um, you know, if you start feeding the input back in, 'cause we're not quite sure how to contain that, too. And so, you know, I think you can do a, I think you can measure things based on how long they run, and you could also measure it by how many times it's actually taken its own guidance-

Steven Sinofsky 2:55 Mm

Martin Casado 2:55 ... which would be kind of more of an agency.

Steven Sinofsky 2:56 Yeah, 'cause I think-- I do think it's important that in this transition... Look, we are, what Aaron described is where we're gonna be. It, it-

Erik Torenberg 3:03 Yeah

Steven Sinofsky 3:03 ... j- it's just that what are the interesting steps that happen along the way? 'Cause we are gonna need it, for the time being, it to stop and say, "Am I heading in the right direction or not?"

Martin Casado 3:13 Exactly right. Like it-

Steven Sinofsky 3:13 'Cause you, you really, putting aside all the horror stories about, you know, taking, um, action without consent and using accounts and data, whatever, there is this thing where, like, you just don't wanna waste your time-

Erik Torenberg 3:26 Right

Steven Sinofsky 3:26 ... on the clock-

Erik Torenberg 3:27 Right. Yeah

Steven Sinofsky 3:27 ... while it's churning away way off in the wrong direction.

Martin Casado 3:30 Yeah, so the question is to what extent do they have their own agency, which to me means they've spit something out, and they've kind of consumed it back up again, and it's still a sensible thing. Which, by the way, as you start thinking of these things in distribution, it's actually a very difficult thing to do because it doesn't know if it's gonna be spitting something out that's still in distribution when it brings it back in. Like-

Steven Sinofsky 3:46 Mm

Martin Casado 3:47 ... they don't have that self-reflection. So I, I think there's actually a very kind of technical question here of to what extent we can make these things have independent agency. But we can make them long run pretty easily.

Steven Sinofsky 3:55 Yeah, yeah. We're good at the long run.

Martin Casado 3:57 The long-running thing.

Steven Sinofsky 3:58 What, what you get back is, yeah.

Martin Casado 3:59 Yeah. [laughs]

Erik Torenberg 4:00 Yeah, I mean, I think the, um, uh, the, the interesting thing is how the ecosystem is sort of solving, um, or, or mitigating then, you know, the, the issues. Like, you're, you're seeing sort of this logical division of the agents. So they might be long-running, but they're not actually trying to do everything.

Martin Casado 4:16 Yeah.

Erik Torenberg 4:17 And so-

Martin Casado 4:17 They're very smart

Erik Torenberg 4:18 ... so the more that you subdivide the tasks out, then actually the more that they, they can go pretty far on, on a single task without, without getting kinda totally lost on what they're, what they're working

Erik Torenberg 4:27 on.

Steven Sinofsky 4:27 Well, Unix is gonna prove to be right.

Erik Torenberg 4:29 [laughs]

Steven Sinofsky 4:29 Which is, like, you're gonna ha- you're gonna wanna break things up into much smaller granularity and tools. And I think to other points that you've made o- on X, like, you're gonna wanna divide things up so that it's, like, an expert in this thing.

Erik Torenberg 4:42 Yeah.

Steven Sinofsky 4:43 And, and then it might be a different, let's just say body of code, where you go and asks, like, ask, you know, "Are you good at this thing?"

Erik Torenberg 4:50 Yep.

Steven Sinofsky 4:50 "Let me get your answer on, on this part of the problem."

Erik Torenberg 4:53 Yeah. Um, it's, it's kind of interesting. I, I don't know, um, how much you've plotted this, but, like, the conversation on AGI ha- has sort of evolved, you know, very clearly-

Martin Casado 5:02 Oh, yeah

Erik Torenberg 5:02 ... in the past, like, six months. And, and I think that the consensus was, may- maybe not even consensus, what, what some of the view was, let's say two years ago, was is this sort of monolithic system that's just super intelligent, and it solves a- you know, all things. And now if you kinda fast-forward to today, and let's say whatever s- we, we agree kinda state-of-the-art is, it's sort of looking like that's probably not gonna work, and, um, for, for a variety of reasons, at least in, in today's architecture. So then what do you have is maybe a system of many agents, and those agents have to become very, very deep experts in a particular set of tasks, and then somehow you're orchestrating those agents together, and then, you know, now you have two different types of problems. One has to go deep, the other has to be really good at orchestration. Um, and, and that maybe is, is how you end up solving, you know, some of these, some of these issues over the long run.

Martin Casado 5:49 I, I just think it's very difficult to think cleanly about this. Like, I've still yet to see a system where you, you, they perform very well, and you don't draw a circle that doesn't have a human being in it somewhere.

Steven Sinofsky 5:58 Mm.

Erik Torenberg 5:58 Oh, yeah. [laughs]

Martin Casado 5:58 And so in a sense, like, the G, like, often seems to be coming from... Like, the general seems to be-

Erik Torenberg 6:02 Yeah.

Martin Casado 6:03 So, like, I just...

Martin Casado 6:04 Listen, these things are tremendously good at in-increasing productivity of humans. At some point, maybe they'll increase productivity without humans, but until then, it's just very hard-

Steven Sinofsky 6:12 Yeah

Martin Casado 6:12 ... for me to actually just talk cleanly.

Steven Sinofsky 6:13 Well, and it's just, it's, it, it's so important for peopleTo get past sort of the anthropomorphization-

Martin Casado 6:19 Yeah, totally

Steven Sinofsky 6:20 ... of AI, because that's what's holding everybody back. Like AGI is about, about robot-

Martin Casado 6:24 Right

Steven Sinofsky 6:25 ... fantasy land, and it, and that leads to all the nonsense about destroying jobs-

Martin Casado 6:29 Yeah, totally

Steven Sinofsky 6:29 ... and blah, blah, blah. And none of that is helpful because it, it, you have to then you dig yourself out of that hole to just explain, wow-

Martin Casado 6:36 Yeah

Steven Sinofsky 6:36 ... you know, it's really, really good at writing a case study.

Martin Casado 6:38 Right. Right, right.

Steven Sinofsky 6:38 Like, which, like it writes a better case study than all the people that work for it, but it doesn't know who to write it about.

Martin Casado 6:44 Right.

Steven Sinofsky 6:44 It doesn't know what necessarily you wanna emphasize. It doesn't know what's, what the budget is, what's needed-

Martin Casado 6:49 Well, yeah

Steven Sinofsky 6:49 ... how many words.

Martin Casado 6:50 Right. But and it also turns out like AGI just does an awful lot of work [laughs] in other words.

Steven Sinofsky 6:54 Yeah, yeah.

Martin Casado 6:54 So for example, someone asked me recently, they say, "Well, um, you know, are you worried that like if we have, uh, AGI, then you'll no longer be investing in software companies?" I, I'm like, "Well, I mean, you're AGI." [laughing] Like, right, you are. I'm still investing in software companies, right?

Steven Sinofsky 7:09 Right, right.

Martin Casado 7:09 And so like just because you're AGI says nothing about economic equilibrium-

Steven Sinofsky 7:13 Right

Martin Casado 7:13 ... or economic feasibility, et cetera. So like just the term AGI does basically infinite work for every kind of fear we have and maybe every hope that we have. And the moment we tie it down to like not only it solves a class of problems, but the economics pencil out yes or no, we can actually have a more sensible discussion, which I actually I think is finally entering the discourse.

Steven Sinofsky 7:31 Yeah.

Martin Casado 7:31 I think we're actually talking-

Steven Sinofsky 7:32 Yeah

Martin Casado 7:32 ... a lot more sensibly now than we were a year ago.

Erik Torenberg 7:34 And so when you hear, when people say things or the, the AI 2027 paper, when they talk about sort of automated research or recursive self-improvement, does that feel like fiction or fa-fantasy, or does it feel like... Or is it thinking that even with those things we're, you know, sort of nowhere near, um, you know, peak software and there would just be unli- unlimited, uh, sort of demand?

Steven Sinofsky 7:54 I think you gotta go first for each question because [laughing] I don't wanna-

Martin Casado 7:57 Well, when I don't wanna solve it too [laughing]

Steven Sinofsky 7:58 ... I need you to anchor us in reality and then-

Martin Casado 8:00 That's right. [laughs]

Steven Sinofsky 8:00 ... and then we can deviate.

Martin Casado 8:02 [laughs]

Steven Sinofsky 8:02 Well, I, look, I, I think that first I, I would not, I'm just not a fan right now of buying any, into anything by year because whatever-

Martin Casado 8:10 Yeah

Steven Sinofsky 8:10 ... year you wanna buy into, i- in that, in 2027 we're just gonna be having a fight over what we meant by-

Martin Casado 8:16 Right, right, right

Steven Sinofsky 8:17 ... by the metrics.

Martin Casado 8:18 For real. [laughs]

Steven Sinofsky 8:18 And it just turns into like OKRs-

Martin Casado 8:20 Yeah, yeah, yeah, yeah, yeah

Steven Sinofsky 8:20 ... for an industry, which is just like a ridiculous-

Martin Casado 8:22 Yeah, yeah

Steven Sinofsky 8:22 ... place to be.

Martin Casado 8:23 That's really funny.

Steven Sinofsky 8:23 But, but I-

Martin Casado 8:24 [laughs]

Steven Sinofsky 8:25 ... I, I think that everything takes 10 years, and, and the thing is, but you can't predict anything in 10 years.

Martin Casado 8:30 Right.

Steven Sinofsky 8:30 So how do you even reconcile that? And, and I think that it, there, you, you know, you just have to re- recognize that the, we're on an exponential curve, so no one's predictive powers work.

Martin Casado 8:42 Right.

Steven Sinofsky 8:43 And, and it's, it's just gonna keep happening. It's not gonna plateau. It's not gonna, you know, all of a sudden we're done. And that's what makes this a different kind of platform shift. Like you, if you just, you look at the progress, and that's the same that went through with storage, that went through with bandwidth-

Martin Casado 8:59 Yeah

Steven Sinofsky 8:59 ... that went through with productivity on, on computing, on, on connectivity around the world.

Martin Casado 9:04 Yeah.

Steven Sinofsky 9:04 Like you, because it's exponential, you can't predict it, and it's just folly to sit around and try to predict.

Steven Sinofsky 9:10 Now, you could do science fiction-

Martin Casado 9:11 Yeah

Steven Sinofsky 9:11 ... and you could say in the future when we all have our personal AI with all this other stuff, and then that's great, but then you say it's gonna happen in 2029-

Martin Casado 9:18 Yes

Steven Sinofsky 9:18 ... you're an idiot.

Martin Casado 9:19 Yes.

Steven Sinofsky 9:20 A- and so-

Martin Casado 9:21 That, that sounds totally correct, right? Because basically, uh, three years ago you would not have been able to conceive of Cloud Code, so, or Cursor or, or, you know, name your, your background agent writing code. So it's like what is the point of having some date at which you're, you're naming something? And, um, and so we've actually seen probably vastly more progress in the past just two years of, of actual applied AI than we would've thought, and yet does it matter that one or two of the predictions didn't play out? Like, no. Um, so, so I think it's probably more interesting to think about like where is the technology from more of a classic Moore's law standpoint and like how much compute do we have, how much data are we working through, um, how powerful are these models?

Steven Sinofsky 10:02 I mean, just let me ask you, like as semi-old, like the [laughing] You know, like-

Martin Casado 10:07 Guilty

Steven Sinofsky 10:07 ... well, I mean, like, like nobody after AI collapsed and machine translation and m- and machine vision failed-

Martin Casado 10:16 Yeah

Steven Sinofsky 10:16 ... there, you couldn't find anybody who thought that those would become solved problems.

Martin Casado 10:21 Yeah, totally.

Erik Torenberg 10:21 Hmm.

Steven Sinofsky 10:21 Or like, or after neural nets-

Martin Casado 10:23 Yeah

Steven Sinofsky 10:23 ... imploded and like literally you were teaching-

Martin Casado 10:25 Or expert, or expert systems or whatever, yeah

Steven Sinofsky 10:26 ... or expert systems, but you were teaching and like, like if you tried to teach neural nets-

Martin Casado 10:31 [laughs]

Erik Torenberg 10:31 Yeah, yeah

Steven Sinofsky 10:31 ... like the students would rebel-

Martin Casado 10:33 Yeah, yeah, yeah

Steven Sinofsky 10:33 ... because you were wasting everybody's time. You know, in, in, like in 1989, like Hinton couldn't get funded trying to-

Martin Casado 10:41 Oh, yeah, yeah

Steven Sinofsky 10:42 ... to do neural networks.

Martin Casado 10:43 Neural networks, yeah, yeah.

Steven Sinofsky 10:43 I, I took, like I, grad school was this three volume history of artificial intelligence thing. Neural nets was like eight pages.

Martin Casado 10:49 I, you know, ironically, I remember when ML was the cool thing-

Steven Sinofsky 10:52 Yeah, yeah

Martin Casado 10:52 ... and neural nets was the old thing. [laughing] And now like, you know, ML is like the old thing and neural nets are the cool thing again. [laughs]

Steven Sinofsky 10:57 Right, or NLP.

Martin Casado 10:58 Yeah.

Steven Sinofsky 10:58 Like and so-

Martin Casado 10:59 Totally, yeah

Steven Sinofsky 10:59 ... so the fact that like, so we will return to all of these problems-

Martin Casado 11:03 Right

Steven Sinofsky 11:03 ... that couldn't be solved.

Martin Casado 11:04 Yeah, yeah.

Steven Sinofsky 11:04 Like even like this, the w- w-

Martin Casado 11:06 For sure

Steven Sinofsky 11:06 ... everyone's favorite one, oh, it doesn't understand math.

Martin Casado 11:09 Right.

Steven Sinofsky 11:09 Like okay, that is a solvable problem-

Martin Casado 11:11 Right, for sure. Of course

Steven Sinofsky 11:12 ... because math is solvable.

Martin Casado 11:13 Right.

Steven Sinofsky 11:14 Like there's just no one put the math layer in to understand what a number was-

Martin Casado 11:17 Right

Steven Sinofsky 11:17 ... and to, you know, hard code it and just build in an expert system for math-

Martin Casado 11:21 Yeah

Steven Sinofsky 11:21 ... which is actually a well understood thing-

Martin Casado 11:23 Yep

Steven Sinofsky 11:23 ... 'cause we've had Maxima since like 1975.

Martin Casado 11:26 Mm-hmm.

Erik Torenberg 11:26 Yep.

Steven Sinofsky 11:26 You know?

Martin Casado 11:27 I, I think it's important to like maybe for us to describe how hard it is to predict

Martin Casado 11:31 anything, right? So let's take recursive self-improvement. This is one of my favorite ones. So the theory of recursive self-improvement is you have a graph or you have a box which is the thing, and then there's an arrow that goes back to the box which says improve. And then of course you look at that and you're like [laughing]

Steven Sinofsky 11:45 Works.

Martin Casado 11:46 Right. So I guess, you know, like from an intuitive lay perspective, every time you have a box with an arrow back in it, you're like, "Okay, we're, we're done," right? But like if you know anything about nonlinear control theory, answering that question is one of the most difficult question that we know in all of technical sciences, right? Like does it converge? Does it diverge? Like does it asymptote? Right? So for example, you could recursively self-improve-If you're doing basic search, but you asymptote, right?

Steven Sinofsky 12:11 Right.

Martin Casado 12:11 And so, like, saying recursive self-improvement from, like, a deeply technical perspective says almost nothing.

Steven Sinofsky 12:17 Mm-hmm.

Martin Casado 12:18 It says... But, but, but unfortunately, because we tend to anthropomorphize AI, we say recursive self-improvement, and all of a sudden we're like, "And then it, like, overcomes energy boundaries-

Steven Sinofsky 12:29 [laughs]

Martin Casado 12:29 ... and human intelligence," [laughs] and then-

Steven Sinofsky 12:29 Well, that's how it goes from being a toddler to being, like, an eight-year-old. It just because it, it figured out how to learn.

Martin Casado 12:34 It recursively self-improves, right?

Steven Sinofsky 12:35 Yeah, yeah, yeah [laughs] .

Martin Casado 12:35 And so, I mean, the reality is, like, nonlinear control systems, which are feedback loops that are adaptive, we don't even have the math for, for, for a relatively simple system to understand what happens. You have to actually know the distributions that come out and go into them. And so these things are gonna improve. They're gonna continue to improve. Maybe they'll improve themselves, but just because they do improve themselves doesn't mean they, they can continue to do it, and this is kinda part of this entire journey is we're learning about these systems.

Steven Sinofsky 13:00 Right.

Martin Casado 13:00 Again, the good news is I think we're talking a lot more sensibly now than we were a year ago, and hopefully that will continue. I don't... Hopefully, [laughs] hopefully the discourse can recursively self-improve so we're just more sensible.

Steven Sinofsky 13:09 Well, the good news is that's involving humans.

Martin Casado 13:10 [laughs]

Steven Sinofsky 13:10 So we don't actually have to worry.

Martin Casado 13:12 [laughs] Oh, that's right.

Steven Sinofsky 13:12 But I, I think that, I mean, you, you must be seeing this even with, with customers. I mean, like, take the conversation about, like, hallucinations-

Martin Casado 13:20 Yeah

Steven Sinofsky 13:20 ... and things like that, how, how dramatically that's altered-

Martin Casado 13:23 Yeah

Steven Sinofsky 13:23 ... in just the past two years, say.

Erik Torenberg 13:25 Yeah. In, in, on two dimensions, actually. So on one dimension, the, the problem of hallucinations has improved, so the, as the models get better, as our understanding of how do you, you know, whether it's RAG or whatever, what, you know, even the, even the, the problem of, uh, of, of actually the efficacy of the context window has, has improved. So you have the technical improvements, um, you know, kind of across the stack, and equally, you have a kind of a cultural understanding to some degree within the enterprise, uh, as to, like, okay, actually, no, these are, these are non-deterministic systems. They're probabilistic. So, so you're starting to see almost a culture shift, which is, okay, uh, you can, you can actually, uh, implement AI in, in es- essentially more and more critical use cases because the employees that are using those systems understand that they do actually have to do the work to verify it. And then the only question is, is what is that ratio of, of time it took to verify versus if I had done it myself and how much efficiency gained for whatever that workflow is? Um, but we are, we're going from probably, like, two and a half years ago where there was, you know, this instant excitement as, as to, "Oh my God, this is going to be the greatest thing of all time," to a reality check within three to six months 'cause everybody was like, "Hallucination is gonna be the, the massive, you know, kinda problem," to now a couple years later after that, which is like, okay, like, we're, we're seeing the hallucination rates shrink. We're seeing the quality of the outputs increase, and we understand that you do have to go and review the work that these AI, you know, agents are doing. And that, that takes on a different form debas- depending on the use case. So in the form of coding, that means just, like, you just have to go review the code in the, in the, uh-

Steven Sinofsky 15:02 Which you had to do anyway.

Martin Casado 15:03 Which-

Erik Torenberg 15:03 Yeah.

Steven Sinofsky 15:04 People seem to be forgetting.

Erik Torenberg 15:05 You, you had to do it anyway-

Martin Casado 15:06 Yeah, yeah, yeah, yeah

Erik Torenberg 15:06 ... but, but, like, there was probably at least a little bit of, like, theory as to, like, what part you, you should go review with extra level of detail because you kinda knew the person you were working with.

Martin Casado 15:15 But it al- it al- it, it also implicitly limits the value of AI, which people are uncomfortable with, right?

Erik Torenberg 15:19 Right, right.

Martin Casado 15:19 Which it just basically says it helps people that will know more than the AI does-

Erik Torenberg 15:23 Yes

Martin Casado 15:23 ... and as soon as it knows more than a, you know... Like, it starts to actually kind of bisect the utility.

Erik Torenberg 15:27 Yeah. Yeah, basically, it's, it's super interesting, which is the experts are now becoming, uh, the a- the productivity of an expert is, is outpacing everything else.

Martin Casado 15:34 Yeah, of course.

Erik Torenberg 15:34 Which is, which was this, you know, I think we could have probably predicted it based on historical events, and I think you've got some good theories about how, you know, the, the skill, the, the, the type of skills, um, that, that are, that, you know, that kind of, the right user for these models for the kind of use case. So we're seeing that, you know, where the expert engineers are like, like, "I don't mind that it's a slot machine where I'm pulling it and I see what comes out because I know I can, I can still get 10X productivity."

Martin Casado 15:58 I can still get to where... It gives me a good idea of-

Erik Torenberg 15:58 And, yeah, and I get it good enough that, that, that it's worth that, that productivity gain, whereas if you were, like, not an expert engineer and you did the slot machine, you probably would try and go and deploy-

Martin Casado 16:07 Yeah

Erik Torenberg 16:07 ... you know, all the ones that were also wrong.

Martin Casado 16:09 And, and, and you don't, actually don't know which lever to pull.

Erik Torenberg 16:11 Right.

Martin Casado 16:11 Which is, uh, i- i- and a big thing is, like, literally knowing the, like, what to ask for and what language to use will get you-

Erik Torenberg 16:15 Yes

Martin Casado 16:15 ... a

Martin Casado 16:16 better result.

Steven Sinofsky 16:16 Well, that, I, I think that this is just an incredibly important point that you're making, and it, it really gets to the heart of what it means to use a tool. Like, you know, you put me-

Erik Torenberg 16:25 Yeah

Steven Sinofsky 16:25 ... in front of, like, a 12-inch chop saw and say-

Martin Casado 16:28 [laughs]

Steven Sinofsky 16:28 ... like, "Go fix the fence," really, really bad idea.

Martin Casado 16:31 [laughs]

Steven Sinofsky 16:32 I mean, I could go buy one.

Martin Casado 16:33 Still bad idea.

Steven Sinofsky 16:33 I lo- I could cruise the Home Depot and-

Martin Casado 16:34 There's a, there's a reason the tools counter [laughs]

Steven Sinofsky 16:35 ... and, and I'm like, "Ooh, dang, man. I don't have a DeWalt."

Martin Casado 16:39 [laughs]

Steven Sinofsky 16:39 And I could buy it, but it's really not a, a particularly good idea.

Martin Casado 16:44 Right.

Steven Sinofsky 16:45 And, and I think that how these platform shifts happen-

Martin Casado 16:48 Yes

Steven Sinofsky 16:48 ... and why there's so much excitement-

Martin Casado 16:49 [laughs]

Steven Sinofsky 16:49 ... over coding is that, well, the best way for a platform shift to take hold is it's the, the experts that are... The, the closest you have to an expert in the new platform is who becomes the most enthusiastic-

Martin Casado 17:02 Yeah

Steven Sinofsky 17:02 ... and the biggest users overall. Like, I, I've been practicing yoga ov- over at, um, the Cubberley Community Center in Palo Alto 'cause the studio's closed for remodel. But, but what's neat is that was, like, the OG place for computer clubs.

Martin Casado 17:14 Oh, nice.

Steven Sinofsky 17:14 Like, in the early 1990s and the late '80s, like, if you ever wanted to meet the computer club... And you would go, and, like, this is, like, halt and catch fire. Like, you-

Martin Casado 17:23 Oh, nice

Steven Sinofsky 17:23 ... and it's, like, like, a bunch of people with soldering irons and shit, and, like, they're-

Martin Casado 17:28 Yeah.

Steven Sinofsky 17:28 That's who... And, and, you know, when it, when it didn't work, when something was broken, that wasn't like, "Oh man, these things are terrible. I'm wasting all my time."

Erik Torenberg 17:35 Right.

Martin Casado 17:35 No, yeah.

Steven Sinofsky 17:35 That was, like, the whole meeting-

Erik Torenberg 17:36 Yeah.

Martin Casado 17:36 Right

Steven Sinofsky 17:37 ... was like, who could get-

Erik Torenberg 17:38 Right

Steven Sinofsky 17:38 ... like, one of these new-

Erik Torenberg 17:39 That's awesome

Steven Sinofsky 17:39 ... discrete graphics cards-

Martin Casado 17:40 Yeah, yeah, yeah

Steven Sinofsky 17:40 ... to actually work? And do you bug the driver? Or just, "Can anyone print? Is there anyone in this room who can print in this new thing called PostScript?"

Martin Casado 17:47 Yeah.

Steven Sinofsky 17:47 And I, I think that's what's really happening right now, and so first it's obvious it should happen with development and coding first.

Erik Torenberg 17:53 Yeah. Yeah.

Steven Sinofsky 17:54 Because they're the most forgiving-

Erik Torenberg 17:56 Yeah

Steven Sinofsky 17:56 ... and the most understanding of, like, what's a bug, what's a thing that can never get fixed? And the thing to watch for is no one is saying that coding can't get fixed.

Erik Torenberg 18:05 Right.

Martin Casado 18:05 Yeah.

Steven Sinofsky 18:05 Like, whatever it's generating that's bad for, like, a 2X coder rather than a 10X coder-

Martin Casado 18:10 Yeah

Steven Sinofsky 18:11 ... no one is saying, "Well, that'll never be fixed."

Erik Torenberg 18:13 Right.

Steven Sinofsky 18:13 And then the next thing that's gonna happen is gonna be what I think is just gonna be, like, the, the creation of words, like the, the marketing document, the positioning document, all of this long form stuff where if you're really good at that job, you, you can

Erik Torenberg 18:27 You know the right questions to ask.

Steven Sinofsky 18:29 Yep.

Erik Torenberg 18:29 You know what looks good.

Steven Sinofsky 18:30 Yep.

Erik Torenberg 18:30 And then you can, you can get really domain-specific, like on the next, you know, the next level is like, "Oh, I need to understand, like, um, a competitor."

Steven Sinofsky 18:37 Yep.

Erik Torenberg 18:37 Which then is using real information from the internet in real time, not just statistical. And then you're like, "Well, they already know what the competitor does."

Steven Sinofsky 18:45 Right.

Erik Torenberg 18:45 Like they're, they're... And then my favorite scenario is the one that just constantly just has these aha moments is attack this thing I just wrote.

Steven Sinofsky 18:53 Yep.

Erik Torenberg 18:53 I'm not interested in you getting, adding em dashes and making it a little bit better.

Steven Sinofsky 18:57 [laughs]

Erik Torenberg 18:58 I just wanna know, what did I miss?

Steven Sinofsky 18:59 Yeah.

Erik Torenberg 19:00 You said one, I think recent- on this last one about like, here's my earnings statement.

Steven Sinofsky 19:04 Yeah.

Erik Torenberg 19:05 The, the-

Steven Sinofsky 19:05 Disimprove it

Erik Torenberg 19:06 ... the, for people, that's the thing you read-

Steven Sinofsky 19:07 Or tell me the, yeah, analyst

Erik Torenberg 19:08 ... that you read after the, uh, uh, uh, to the analyst. That now, like attack it like an analyst.

Steven Sinofsky 19:13 Yeah.

Erik Torenberg 19:13 And there's like 6,000 hours per company of analyst questions.

Steven Sinofsky 19:18 Yep.

Erik Torenberg 19:18 It knows what they're gonna-

Steven Sinofsky 19:19 Yeah

Erik Torenberg 19:19 ... they only ask like three questions anyway.

Steven Sinofsky 19:20 Yeah.

Erik Torenberg 19:20 Ooh, expense line, you know? And-

Steven Sinofsky 19:22 Right

Erik Torenberg 19:22 ... and I, I feel like this is the thing that-

Steven Sinofsky 19:25 Do not watch this if you're an analyst. [laughs]

Erik Torenberg 19:26 Yeah. [laughs] And this is not any advice about being an analyst or any-

Steven Sinofsky 19:30 It's fine. [laughs]

Erik Torenberg 19:31 But I, this is what, what's really gonna happen with-

Steven Sinofsky 19:34 Yeah

Erik Torenberg 19:34 ... with writing, and then it's gonna happen with PowerPoint and slides, and then it's gonna happen with video, and it, it-

Steven Sinofsky 19:39 But it's right, but it's, it's really important to call out, um, which is you're getting the consensus mean response. And so in the limit, it's offloading a lot of kind of busy work if you're a professional. Like if you're a professional, you actually know all of these things, you just don't have the time to go through all of it.

Erik Torenberg 19:54 Yeah.

Steven Sinofsky 19:54 And you may not remember it. So, so in a way it's, it, it's, it's, it's productivity helpful, but it's not, you know, solving some problems that where you know you are a particular expert in. And this is maybe why for those that are non-expert, it's a little bit more threatening because it can do that job.

Erik Torenberg 20:09 Yeah. Well, the, the maybe to, to bridge a view and probably throw in a different tangent like so, so Steven, you're asking like, so where is the enterprise now? So that was the coding piece. I think where you're seeing this is, uh, you know, kind of clear understanding, which is, okay, uh, the, what I'm gonna get out will be correlated to what I put in, so how precise I put the prompt, what, what like, like I think prompting doesn't go away anytime soon simply because the leverage you get on the set of instructions you're gonna give the AI at the start is still gonna be massive. So we see-

Steven Sinofsky 20:39 Wait, wait. What, what, what, what would... If prompting went away, what would you end up with?

Erik Torenberg 20:42 Well, I mean, two years ago, I think the, like people were like, like you'll just tell the AGI what you want it to produce.

Steven Sinofsky 20:48 Oh, it's just one prompt.

Erik Torenberg 20:49 Yeah. I'm, I'm gonna get my-

Steven Sinofsky 20:50 There's just, there's just one prompt. Like you unbox it and you say-

Erik Torenberg 20:52 Yeah

Steven Sinofsky 20:52 ... "Go do something."

Erik Torenberg 20:53 My, my agent is gonna-

Steven Sinofsky 20:54 Be, be a software engineer.

Erik Torenberg 20:54 Right. No, literally that was the, like that was like an open debate. And it was like no, you're probably missing the fact that what is in my head is gonna be unbelievably germane to the thing that I'm trying to produce, and like I have to somehow give you that context. Like there's no world where you have that context without me telling it to you, and now you're seeing it. Like you're seeing these incr- incredibly unhinged prompts which are like pages long.

Steven Sinofsky 21:14 Yeah.

Erik Torenberg 21:14 And the output you're getting from that is actually like way better than if you didn't give it that context. So I think there's a clear understanding of, of that side on the enterprise use cases, and then a clear understanding that you've gotta go and review it. And then, and then on, on this point about like, well, you know, y- you know, what, what is-

Steven Sinofsky 21:29 Wait, I, I just have a... We forget that formal languages came out of natural languages for a reason.

Erik Torenberg 21:34 Mm.

Steven Sinofsky 21:34 Like we didn't start-

Erik Torenberg 21:35 Yeah, yeah

Steven Sinofsky 21:35 ... we didn't like start with like-

Erik Torenberg 21:36 Oh.

Steven Sinofsky 21:36 We didn't start with like formal language like, oh, it's much easier to speak in English. Let's speak in English. It's the opposite.

Erik Torenberg 21:41 Right.

Steven Sinofsky 21:41 It was like we had this natural language. We're like, it's very tough to convey the information that I want to.

Erik Torenberg 21:45 Right.

Steven Sinofsky 21:45 You and I are both experts. We understand the solution space, so let's communicate more efficiently.

Erik Torenberg 21:49 Right.

Steven Sinofsky 21:50 Right? So to think that this somehow wouldn't happen without language is silly.

Erik Torenberg 21:51 And, and that's what jargon is.

Steven Sinofsky 21:53 Of course.

Erik Torenberg 21:53 Like jargon-

Steven Sinofsky 21:54 Right

Erik Torenberg 21:54 ... is just a formalized way that people who have domain expertise talk to each other.

Steven Sinofsky 21:58 That's exactly right. Yeah.

Erik Torenberg 21:58 And so, so the thing that, that is kind of, kind of the most like fun to kinda think about right now at least is, and, and maybe you give us a little history lesson on this in, in kind of interesting parallels. So when does the style of

Erik Torenberg 22:14 work change because of the tool?

Steven Sinofsky 22:16 Ah.

Erik Torenberg 22:16 Versus-

Steven Sinofsky 22:17 Yeah, yeah

Erik Torenberg 22:17 ... the tool sort of adapted to the style of work. And so what I'm starting, and we're like only in day one of this, but what I'm starting to see kind of some, some patterns emerge, which is we thought agents would go and learn how we work and then automate that. And then the quest- and so basically agents conform to how we work. The question is when is the moment when we conform to how agents are best used?

Steven Sinofsky 22:38 Yep.

Erik Torenberg 22:38 And you're, you're seeing this in a couple areas, so you're seeing this in engineering to start with, which is like people are saying, "Okay, I'm gonna have agents and then sub-agents for parts of the code base, and then I'm gonna give them kinda read-me files that the agents read, and then, and then I'm gonna actually optimize my code base for the agent as opposed to the other way around in other forms of knowledge work." So within how we use Box, um, with, with our AI product, like you're starting to see people like basically tell the agent like its complete, you know, job, and the, the workflow is now starting to be almost like the agent is almost dictating the workflow in the future as opposed to it's just mapping to the existing workflow. So I don't know like what the history is on this of like when does, when does the work pattern itself shift because of what the technology is capable of, but I think, I think probably where this goes has to be some version of that, which is, which is it's not gonna just be the agents just plop into how we currently do our work and then, and then just automate everything. I do think you start to change what we actu- what the work is itself, and then, and then agents actually go in and accelerate that.

Steven Sinofsky 23:38 Well, as important as that is, it's actually more important.

Erik Torenberg 23:40 Okay.

Steven Sinofsky 23:40 Like because what, what happens is where it, there, there's, to reuse the, a word in a different, this anthropomorphization of work, what happens in is that the first tools actually anthropomorphize the work.

Erik Torenberg 23:54 Uh-huh.

Steven Sinofsky 23:55 And so like if you go back, this is every single evolution of computing. I mean, like how long did it take for Steve Jobs to get rid of the number buttons on a smartphone?

Erik Torenberg 24:03 Right, right, right, right. [laughs]

Steven Sinofsky 24:04 Like, like they, they still had number buttons. Or like you look at, at cars, and until Elon got rid of all the controls-

Erik Torenberg 24:10 Right. Yeah

Steven Sinofsky 24:10 ... everybody kept all of the controls. I don't wanna get in that fight. But, but like the, what happened with every technology shift is, you know, if you, if you were to look at what accounting software looked like in the '60s before IBM said s-stop. W-we all use double entry, but we need to have people skilled in how computers can do the accounting-

Erik Torenberg 24:29 Uh-huh

Steven Sinofsky 24:29 ... not how people can, because we're never gonna figure out how to close the books-

Erik Torenberg 24:33 Right

Steven Sinofsky 24:33 ... if we have to automate this whole r-room of people with green eye shades-

Erik Torenberg 24:37 Right

Steven Sinofsky 24:38 ... that have a manual process based on how far apart the desks were.

Erik Torenberg 24:41 Right.

Steven Sinofsky 24:42 And, and everything that happened with the rise of, of PCs and personal productivity started off, and I always use this example because I've watched it happen like five times now, which is the, the first PCs that did word processing, the biggest request was how do I fill in like expense reports?And so the whole, this whole world grew up of tractor-fed paper that was pre-printed with the expense report. Right. And so then software, we wrote all of this code, like are you using an Avery 942- [laughs] ... expense report or is it a New England Business Systems A397? And, and like, you know, and then you had like these adjustments in the print dialogue, like .208 inches, and you, you moved little things around, and then you would print out like- [laughs] ... ate dinner, $22, and that was all you printed. Right. And then someone said, "You know, we could use the computer- [laughs] ... to actually print the whole thing." Right. [laughs] And then, like fast-forward, and finally Concur said, "You know, why just take a picture? Why not just- Right ... take a picture of the receipt, and then we could do all of it?" And so then the whole thing gets inverted, and, and every single business process ended up being like that. And, and then there are things that really, really do change the tools. Right. Like when email came along, you know, it used to be to prepare an agenda for a meeting, somebody would open up Word and type in all the things, and then print it out, and everybody would show up to the meeting with this very well format. And now, and then like- Yeah, new case ... email came out, and that whole use case for Word- Right ... just evaporated. Yeah. And, and then an email agenda became no formatting, nothing, just like, "Here are the eight things we're gonna talk about." Yeah. And you show up, and everybody's like, "Did you get the agenda?" You know what, what's interesting about the AI one is it's kind of t- it's like we're seeing the same thing, but vis-a-vis AI. So nobody really predicted the generative stuff, and we've had AI for a very long time. So we had chatbots, we've had, you know... And so you had these kind of like AI shape holes in the enterprise for a long time, and a lot of the mistakes that we see today is people are taking the generative stuff and trying to kind of cram it into the old models- Mm-hmm. Yeah, yeah ... when it's really a new behavior that's emerging, that's very m- much more in... Like, it used to be you'd se- centrally sell, you know, AI to some platform team, and then they would kind of try to get the NLP thing to work or the voice to work for like talking to people on the phone for support, and it was this kind of very central. A lot of the adoption that we see is like much more individual, for example. And so I just think that there is a- Right ... bit of a mismatch, as we're seeing now, that is getting ironed out, too. Well, and, and so I, I think the question is, is, yeah, are we in the phase where we're trying to graft the agents and, and work in basically the, what we've been doing for 30, 40 years- Yeah ... of software, and is this gonna be actually like a, like a, like the first real step function shift we've seen in what the workflow itself should look like? Oh, we- Yeah ... but we are. I mean, like you, if you, you know, remember, people like p- I, I tried to jam the internet into Office. Right. And it, and, and- It was fun to watch ... no, but I mean, you were, you were not watching. Yeah. [laughs] But, but, but like, but, but everybody around was trying to jam the internet- Right ... into their product because that's the only way you could envision it. Right. And it, it- Right ... didn't really... Like, you, you were like, "Well, where else would the internet go?" [laughs] Like, there's no word processor on the internet. Right. Right. Like, there's no spreadsheet on the internet. Yes. And, and then the other people would be like, "Well, let me just try to implement Excel using these seven HTML tags with no script." Uh-huh. That turned out to not be a really good idea either. [laughs] The best was like, "Let's do PowerPoint." Well, how do you do it? You give them five edit controls, tell them their bullet points, and then we'll generate a GIF on the back end and send it back to you as the slide. Yeah. [laughs] Okay, that, that, that was not... And so there was that whole, like that- I think actually maybe the, the main point is just the durability of Office. But- It transcends all- It does ... all disruptions. [laughs] I, I like to think it pretty much rises above everything. Yeah, exactly. [laughs] But, but the thing is, is that that's where we are now- Yeah ... is everybody... A- and you know, like e- Do, but do you think then- No ... I mean, j- just to dig a little bit, so do you think this is similar to the internet in that it's a consumption layer change? 'Cause I always viewed the internet as very much a consumption layer change. Like, I go to a, you know, instead of going to my computer, I go to the internet. But otherwise, things kind of are the same where AI's got this weird quirk which for the first time I can recall, programs are abdicating logic to a third party. Like, we've always abdicated resources, right? Yeah, yeah. Like, so it'd be like, "Okay, I'll use your disks or whatever," but like I'm writing the logic. But this time it feels like we're changing the consumption layer. So like, you know, when my son, you know, talks to an AI character and, you know, he's not going to wellsfargo.com, he's going to an AI character, and so like that's changing kind of how we're interacting with the computer. But also these programs are no longer kind of written by a human- Yeah ... in the same way. So I feel like the change is maybe a bit more sophisticated. Oh, I think that, but this is the, this is why it's a platform shift- Right ... and not just an application shift. Right. Yeah. Like where, where each l- each platform shift changes the abstraction layer with which you interact with computing, but what that also does is it changes th- what you write the programs to. Yeah. Do you, do you remember ever abdicating logic to- Oh, y- here's a great, like here's an example of how disruptive this can be. The, the first word processors in, in, in the DOS era, the character mode era, they all implemented their own print drivers and clipboard. So if you were Lotus and you wanted to put a chart into a memo, you, you, you couldn't 'cause you didn't have a word pro- you didn't sell a word processor, so you actually made a separate program- Uh-huh ... to make something that the leading word processor could consume. Oh. And if you were WordPerfect, your ads said, "We support 1,700 printers." Huh. Like, and you won reviews because you had 1,700 and Microsoft had 1,200. Oh, that's so... [laughs] And so then along comes- That's a great one, actually ... and then so Windows comes along, and, and if you were s- and if you were trying to enter the word processing business, step one, I need to hire a team of 17 people to build device drivers for Epson and Okidata and Canon printers 'cause you can't get them anywhere. Microsoft came along and for Windows built print drivers and a clipboard, and all of a sudden, and also Macintosh did it. Yeah, yeah. All of a sudden, you, there was a way that two applications that had no a priori knowledge of each other- Good deal. Yeah ... but of course, if you were WordPerfect or Lotus, that's a disa- Yeah, yeah. Of course, yeah ... you got creamed by that- Right ... because your ability to control- Right ... your ev- right. And so, and what happened was a bunch of developers were like, "Wow, this is cool," 'cause now I'm just by my... When we did C++ for Windows-Like we were like where w- the demo, in fact, at that Cubberley Community Center, I would go and I would show brand new Windows programmers in 1990 like, "Hey, you don't have to write print drivers and use the clipboard." And like literally standing ovation of, you know-

31:12 Yeah [laughs]

31:12 ... all 10 people at the thing.

31:13 [laughs]

31:13 And, and but, but they were like more than happy-

31:17 Yeah

31:17 ... to let data interchange between product, 'cause they were like, "That's nothing but opportunity for me."

31:22 Can, can we... C- you g- uh, like they m- they probably from an emotional standpoint felt exactly the same way as like a Vibe Coder does today.

31:29 Oh, yeah.

31:29 Which is like, you've just given me the-

31:31 No

31:31 ... platform that [laughs] and it was just a print driver.

31:33 The, the writing code for Windows book was like this big.

31:35 [laughs]

31:36 But the writing a device driver-

31:38 Yeah

31:38 ... for an Epson printer was this big.

31:39 Yeah, yeah, yeah.

31:40 Writing it for a Canon printer was this big.

31:42 That's incredible.

31:42 And so, so this-

31:43 I'm just, but I'm ju- I'm just actually trying to think of like the, but the paradigm shift is the same, which is there's been many times where we've reduced the amount of work a developer takes. What I just don't remember ever where the programmer abdicates logic. Like, so for example-

31:55 SDM didn't?

31:56 Not logic. Like I, I would always say what is correct and what's not correct, right? And so-

32:01 I think you undersold it, though. [laughs] Could... No, this is the thing, by the way, everybody, that Martin invented and worked on-

32:08 Oh, yeah [laughs]

32:08 ... and stuff. Like, but, but it's a big deal, trust us.

32:10 May- maybe we should postmortem your pitch at the time if, if you-

32:12 Well, no, but I-

32:12 ... you wanna pitch this, yes. [laughs]

32:14 ... but I, I mean... Well, no, no, I, look, let me like logic specifically, which is I am writing an app. My app is whatever, some vertical SaaS app for a certain customer base. The answer the app gives is based on logic that I've written historically, right? Like if I run it on the cloud, the cloud is not producing an answer, it's providing resources. If I'm using your device driver, it's, you know, providing access to a device resources. But if I'm like, "Hey, large model, tell me the answer here"-

32:42 Mm.

32:42 ... you're actually abdicating application logic.

32:44 Uh, I-

32:44 I'm, I think maybe you're right. Maybe this logic, it feels, it feels-

32:45 No, I just say I think, I think what you're, you're, you're almost playing like incumbent in, in the sense of trying to-

32:52 Oh

32:52 ... no, trying to like decide this is abdicating the logic and this isn't.

32:57 Right.

32:57 When in fact like it, it really was like a huge competitive advantage for WordPerfect. And, and they didn't wanna give it up, and they, they fought against it. And the number of people who didn't wanna do like great clip... And e- the next example, of course, is the browser-

33:11 I think this is a good metric

33:11 ... where people literally gave up. Like you, in a, in Windows or in Mac, you could rasterize anything you wanted. You wanted a button that you pushed and it spun and it animated like a rainbow, you could do that in your product. But then the web came along and you're like, "Wow, I have to use a gray button that says Submit."

33:29 Mm. [laughs]

33:30 And, and that was like-

33:31 Yeah, I guess I guess to some point-

33:32 And, and, and-

33:32 ... we do use a bunch of third-party things that will-

33:34 Well, it, but it took a long time for those to show up.

33:37 Right.

33:37 And so early in the internet, the magazines in particular, and, and the printed media were the ones who absolutely wouldn't go to the internet because they would not give up-

33:45 Ah, that's fair

33:46 ... their ability to format.

33:47 Mm-hmm.

33:47 And they, and this is, is another part about the, the tooling and where, where what happen- what's gonna happen with AI is that, like a huge amount of the productivity software space today is like the preparation of output. Like Office is-

34:00 I love this. This is great

34:00 ... basically a format debugger.

34:02 Right.

34:03 Like all it is is like 7,000 commands for how to do kerning and bold and italic. And like it turns out AI not only doesn't care-

34:10 You don't... Yeah

34:11 ... you could ask it to make whatever you want.

34:13 Right. Yeah.

34:13 Like you could just say, "I'd like this to be a double index pie chart thing." That, that's not a thing, I just-

34:18 [laughs]

34:18 But, but you, you can do that and it, it will just figure out something that looks like that, and you'll go, "Ooh, cool."

34:23 Yeah.

34:24 And this was where to this disempowering and experts and who's not an expert, when, when productivity software arose, the big thing about it was that there were people who figured out how to like make like killer charts.

34:35 Yeah.

34:35 Like Bennett Lievon's like killer chart guy.

34:37 Yeah. Yes. Yeah.

34:38 And there were people who were like, every meeting started with, "How did you make that chart?"

34:42 Yeah.

34:42 Like, I could be on an airplane-

34:43 Yeah

34:43 ... and somebody would be like making a shitty chart.

34:45 Oh, so, uh, so interesting. So like in this case, the application is like, actually, what's the way to, to visually represent the data?

34:50 Right.

34:50 Which is absolutely now being... Yeah.

34:52 Right. And it turns out-

34:53 This is a, these are great examples

34:53 ... well, because like 90% of the people never really got to be expert-

34:56 Yeah, exactly

34:56 ... at doing that task, even though 90% of the tool-

34:59 Right

34:59 ... is about, like to even... So what happens is each generation-

35:01 But the programmer didn't abdicate the logic in this case. This is the user.

35:04 We, but, you know, what's the user, what's the programmer in that?

35:08 Right.

35:08 And, and in fact, what the programmer was doing was like, like we would invent a thing called wizards or whatever we, you know, and that would make a whole bunch of choices for you, style sheets or whatever.

35:18 Yeah.

35:18 And so in a sense, we were making a bunch of choices for the user, which to the experts looked like disempowering the experts-

35:25 Yeah, yeah, yeah

35:25 ... who were tweaking all. And so I, this is all like this, there, there's some Steve Jobs quote that he loves about Schopenhauer, how if you've seen the, the conjurer-

35:33 The, the grand continuum of, of CS class, yes. Yeah, yeah, yeah

35:34 ... if you've seen the conjurer, it's not a trick anymore.

35:37 Yeah, yeah, yeah.

35:37 And I really feel like this is like the third or fourth time that this has happened just in, in my lifetime-

35:42 Right

35:42 ... of watching this.

35:43 Well, so something that's really caught my attention, because it's the most senior people I know, um, is that a lot of very senior developers are spinning up a lot of background agents, like code agents, and they're interfacing at like the GitHub PR level.

35:55 Right.

35:56 And so it's not obvious to me why you'd do a bunch as opposed to one, and it's not obvious to me why you wouldn't interact directly. So it feels like something's going on here, but I'm not quite sure what, and I would love your thoughts.

36:06 Well, the, the, my read on it, um, uh, and then the, the que- I, I guess I would kind of sort of throw out like what then happens next as a result of this, 'cause, 'cause it, to me, it's actually a little bit of an epiphany on what the future work design could look like in this world, 'cause engineers, back to the prior conversation, are just the first to experience this. But I think what, what my, my read from, from talking to kinda similar folks that are like all in on this is, um, is this mix of basically, uh, the, the effectively the context rot problem, which is, you know, the more that we put in the context window, uh, the, the more it gets confused, the, the lossier the answers get. And so you have to have some kind of a way to partition what, what an agent should work on. And we see this in building agents internally, which is, you know, the, the panacea that I think we maybe would've hoped for is like, well, you just put a million tokens into the context window, and then obviously...

36:56 Oh, so you're saying this is almost like a counter-

36:59 Yeah

36:59 ... trend to the-

37:00 Yeah

37:00 ... the AGI is almost like the, is like the opposite.

37:02 It, it's the opposite, but it, it's, it's, uh, it only works because the models are so good.

37:07 Yeah.

37:07 But-

37:07 But you're giving more things, more specific tasks-

37:09 Yes

37:09 ... rather than one thing, less specific tasks.

37:11 Right, and so the-

37:11 That's interesting, yeah.

37:11 So, so but like I think this is why it's happening. So basically the, the craziest version of this is I was talking to somebody who

Erik Torenberg 37:18 Who is in startup land, and they have, they have, to your point, they have all these sub-agents.

Martin Casado 37:22 Yeah.

Erik Torenberg 37:23 But what's amazing is that it maps one-to-one, uh, to each microservice in, in their code base. And so they have an agent per microservice.

Martin Casado 37:30 Yeah.

Erik Torenberg 37:30 They have effectively a README for the agent, and that agent owns the microservice, and they... I don't know the, the, the specific number, but let's just say you could have, you know, dozens or hundreds of these things going on, and you're, you're effectively mitigating this issue, which is if you just said, "Here's my entire code base, you know, go run wild," you know, to one agent, it's, it will just, you know, produce worse and worse code over time-

Martin Casado 37:52 Yeah

Erik Torenberg 37:52 ... because it's gonna have context rot. It's not gonna know exactly what, what, what you're trying to do in that one area of the microservice, but the sub-agent model seems to be working for that paradigm.

Martin Casado 38:01 I love this counter-pattern.

Erik Torenberg 38:02 Yeah.

Martin Casado 38:02 'Cause everybody's like, "They're gonna..." Like, you know, "Models will be- get, you know, smarter, and you'll give them higher level tasks, and they'll do things longer."

Erik Torenberg 38:09 Yes.

Martin Casado 38:09 This is a counter one. I wanna tweet that, but you have more Twitter followers [chuckles] so-

Erik Torenberg 38:12 Oh, we can, we can collectively do it.

Martin Casado 38:14 [chuckles]

Erik Torenberg 38:14 We'll do it. But, but the, but then, so then the question is, okay, so let- let's just assume this works in engineering. You, you have this interesting dynamic which is, well, then, that means that, like, some of the coding practices will be pretty different in the future. We've talked about this idea of, you know, the individual engineer becomes the manager of agents, so that was already kind of, I think, a well-understood path. This is like a supercharger of that, of that concept.

Martin Casado 38:34 Yep, yep.

Erik Torenberg 38:35 And then the, and then the question is like how does that translate to almost every form of work? Because if I am now, you know, the lawyer and working on cases, and I can have 20 sub-agents that all, you know, do a different case and then basically br- you know, come back in some kind of task queue that I'm going through, like ob- obviously, one, just the, the, the sheer leverage now you get is, is gonna be insane. Um, but I do think the way that you, you know, mi- might even organize the work and, um, and what the, what the, you know, workflows within an organization are, are, you know, inevitably gonna change as a result of that.

Steven Sinofsky 39:08 Oh, but y- I mean, I think, I think this just like goes, gets to the, you know, essentially that, that the, the flow in the workflow has been serialized or linearized-

Erik Torenberg 39:19 Yeah

Steven Sinofsky 39:19 ... based sometimes on knowledge, but other times on tooling.

Erik Torenberg 39:23 Yeah.

Steven Sinofsky 39:23 And so what happens when, when the tooling changes is you just get this realignment of what's truly serial and, and what's not. Like, if you're, if you're planning an event for a company-

Erik Torenberg 39:33 Right

Steven Sinofsky 39:33 ... which is still gonna keep happening, you know, like, "Oh, I have to book the venue. I have to invite all these people. We have to create all these materials." Well, they're actually not particularly gated on each other.

Erik Torenberg 39:42 Right.

Steven Sinofsky 39:43 But if you have-

Erik Torenberg 39:44 We do them, yeah

Steven Sinofsky 39:44 ... an events person-

Erik Torenberg 39:45 Right

Steven Sinofsky 39:45 ... they're gated.

Erik Torenberg 39:46 Right.

Steven Sinofsky 39:47 And, and so now an events person can-

Erik Torenberg 39:49 Right

Steven Sinofsky 39:49 ... start spinning up all of these, these different elements, and then they're gonna come back like, "I've gotten as far as I can on collateral until I get a logo for this event."

Erik Torenberg 39:58 Right.

Steven Sinofsky 39:58 Like, "I've gotten as far as I can on invites until I get the date and the time and the, the venue."

Erik Torenberg 40:03 Right.

Steven Sinofsky 40:04 And I think there's no, there's no reason why you can't spin up all those in parallel because, of course, how does that happen today? Well, if you're a company and you use Box, and you've done... This is your 58th event, you know, you have a folder called Event.

Erik Torenberg 40:16 Right, right.

Steven Sinofsky 40:17 And people take the folder and go Event 59, and they, they make a copy of it and all the stuff in it, and, and, well, if you think about that workflow, that's exactly what a series of, of different background tasks-

Erik Torenberg 40:30 Right

Steven Sinofsky 40:30 ... or agents could go do. And so I think the reason that you could be doing all that in coding is, well, there's a, there was a natural, there's a natural way to break that up because there's a bunch of programming that's not-

Martin Casado 40:41 Right, but there, but there's the other side, but there's also a bit of an indictment on the ability of you to give it a high level-

Steven Sinofsky 40:46 Oh, yeah, yeah

Martin Casado 40:46 ... you know, it, it kind of suggests that the human being needs to be, you know, giving them more granular orders. Otherwise, you know, to start a company, you'd issue one prompt. You'd go to the beach for six months-

Erik Torenberg 40:56 Right, right

Martin Casado 40:56 ... and you'd come back, and you'd have a full company.

Erik Torenberg 40:57 Wh- which is a, which is this almost re-anthropomorphizing effect, which is, which is like it turns out we s- we did, we did kind of figure out division of labor. Uh, we, we figured it out in the context of an, of, of a lot of physical, you know, kind of analog limits that we, that we clearly had, uh, that agents won't have. But we now, you know, there- there's no kind of, you know, total free lunch, so you have this context rot issue, which is, which is that you do actually have to subdivide the tasks at some point. So then the question is, like, what are the right-

Martin Casado 41:27 I mean, it may not be a context rot, if you could. Like, the Occam's razor here is you need to give them specific instructions for s- for specific tasks.

Erik Torenberg 41:34 Right.

Martin Casado 41:34 And if you give them higher level instructions-

Erik Torenberg 41:36 It's gonna-

Martin Casado 41:36 ... independent of context-

Erik Torenberg 41:37 Right

Martin Casado 41:37 ... they just don't know what you want-

Steven Sinofsky 41:38 Right, and this gets to the-

Martin Casado 41:39 ... or might not get the right thing

Steven Sinofsky 41:39 ... formal language part.

Martin Casado 41:41 Yeah.

Steven Sinofsky 41:41 Like, at some point-

Martin Casado 41:41 No, of course

Steven Sinofsky 41:42 ... if you try to use, like, the Uber frontier to get the whole thing done, you, you have to tell it the whole thing, right?

Martin Casado 41:49 Yeah, exactly. That's right.

Steven Sinofsky 41:50 A- a- and that just seems like a lot of work, whereas if you have to tell it less because the part of the model you're using knows more-

Erik Torenberg 41:57 Right

Steven Sinofsky 41:57 ... it's basically a different way of thinking about templates or a different way of thinking about starting artifacts or scoping the context i- in a generic word of context.

Erik Torenberg 42:06 Well, but, but then there's this... I mean, uh, it might, though, be the right, uh, architecture in general if you assume that, that, you know, there's n- we're never gonna get to a point where the model is just 100% perfect, right? And so it might also be the right, uh, kind of architecture design because at some point you're gonna have... You don't want an agent, uh, or a set of agents to go so far down a path when there was a step that it needed to check in with you on-

Steven Sinofsky 42:30 Right, right

Erik Torenberg 42:30 ... because there's just the compounding effect of that. So you do need to kinda subdivide the work also because if you do have gating, you know, moments that are gonna have a bunch of dependencies, the agent does need to know, like, at what point should I roll that back up to the user.

Martin Casado 42:43 Yeah, again, a- a- a- against the c- the, the, the common narrative, now that I think about it, it seems that the trend is, is prompts are getting more complex-

Steven Sinofsky 42:52 Yes

Martin Casado 42:52 ... not less.

Steven Sinofsky 42:53 Right.

Martin Casado 42:53 And we're seeing more agents, not less, doing more narrow tasks, which is almost this kind of counter AGI narrative. It's almost like these are much more specialized and much more deep-

Steven Sinofsky 43:02 Yes

Martin Casado 43:02 ... working with much more specific instructions.

Steven Sinofsky 43:03 A- and there's like a sort of a history of this, thisWow, maybe we can actually solve it if we're specialized-

Martin Casado 43:09 Yeah

Steven Sinofsky 43:09 ... a little more. And it, like if you take expert systems, at first they thought expert systems would just be experts, and they would just know. And then, like by the time you got to the actual published research, like at Stanford, it was like, "This is an expert system in deciding on what type of infectious disease you have-

Martin Casado 43:25 Yeah, like di-digestive

Steven Sinofsky 43:26 ... as long as you have one of these seven." [laughs]

Martin Casado 43:27 No, literally. There, there was, there was a paper, it was like there's just one digestive disorder that actually is a minimal type [laughs] for the system.

Erik Torenberg 43:33 I, I do, well, but I do want to, though, because I, uh, you wouldn't wa- Like, there, there is one big difference, which is somehow the model itself is packing in the inherent intelligence or capability to solve all of these. Like, like, like we are benefiting from the fact that, that at least you can build these all on Claude 4-

Steven Sinofsky 43:50 Yeah

Erik Torenberg 43:50 ... and GP-

Steven Sinofsky 43:51 For sure

Erik Torenberg 43:51 ... 5 and, and that-

Martin Casado 43:52 And that all on a-

Erik Torenberg 43:52 That gives you a lot of leverage

Martin Casado 43:53 ... that all on a computer too.

Erik Torenberg 43:54 Yes.

Steven Sinofsky 43:54 Let me-

Martin Casado 43:54 But yeah, I mean

Steven Sinofsky 43:55 ... try to show, like demonstrate this one with an old person example on this one, which was like early in the PC era, the, the, there were word processors and spreadsheets and graphics and databases, and a lot of people were like, "Why are there these four programs? There should only be one program." [laughing] And, and I, my answer to that, like which often involved screaming, was, "Have you been to an office supply store?"

Erik Torenberg 44:17 Uh-huh.

Steven Sinofsky 44:17 'Cause if you go to an office supply store, there's like paper with numbers-

Martin Casado 44:21 Mm

Steven Sinofsky 44:21 ... and then there's blank s- rectangles of paper, and then there's transparency paper for pr- And like this has been around a really long time. There's some reason-

Martin Casado 44:29 Yeah, there's some-

Steven Sinofsky 44:30 ... that, that these are different

Martin Casado 44:31 ... some physical human context for how-

Erik Torenberg 44:32 How many minutes did it take you for, uh, for you to know Google Wave wasn't gonna work?

Steven Sinofsky 44:36 Zero.

Erik Torenberg 44:36 Okay, okay, okay. [laughs]

Steven Sinofsky 44:37 Like it was-

Erik Torenberg 44:38 It was, it was instantaneous.

Steven Sinofsky 44:38 Ah, it was instant.

Erik Torenberg 44:39 Okay, okay, okay.

Steven Sinofsky 44:40 And, and no, I mean, and, and but this was the thing. There was a product, ancient Mac product that was lauded by the industry called, uh, ClarisWorks.

Erik Torenberg 44:47 Yeah.

Steven Sinofsky 44:47 Which was like, oh, it does ... You could have a spreadsheet-

Erik Torenberg 44:49 [laughs]

Steven Sinofsky 44:49 ... inside a word processor. And my first reaction is, have you seen a person use a spreadsheet? Because their monitor can't be big enough.

Erik Torenberg 44:56 [laughs]

Steven Sinofsky 44:56 So they just want as many cells-

Erik Torenberg 44:58 Yeah

Steven Sinofsky 44:58 ... as you could possibly have.

Erik Torenberg 44:59 Yeah.

Steven Sinofsky 45:00 And you're sitting there saying it has to fit on an eight and a half by 11 sheet of paper on a fi- on a Mac.

Erik Torenberg 45:04 Yeah.

Steven Sinofsky 45:05 And I think that one of the things that happens is, is that these lenses that humans bring to-

Erik Torenberg 45:10 Yeah

Steven Sinofsky 45:10 ... specialization, like really, really matter. And if you think about the medical profession, and you think about going from a, a GP to the radiologist, to a specialist, to a nurse practitioner, through the whole series, they're each gonna look at and use AI in a different way.

Erik Torenberg 45:26 So then the only thing would be, okay, so that was, that level of specialization and division of labor emerged over a 100-year period-

Steven Sinofsky 45:34 Right

Erik Torenberg 45:34 ... with, wi- you know, alongside tools and, but, but also with, uh, driven by a lot of the physical constraints and realities of, of how organizations emerge. So the only question would be, in a post-agent world, in 10 years from now, do those divisions of labor look exactly the same, or do those shift also because the agents collapse, you know, some of the functions, and is there some blurring?

Erik Torenberg 45:55 And then is there just a new set of roles? Like, like clearly there's a role in a bunch of organizations emerging, um, which is like, no, I'm just like, my role is like I'm the AI productivity person, and like I just like have a way of, of, you know, creating all new forms of productivity in the organization with AI. So like clearly we'll have a bunch of new roles, but is our current division of labor gonna also collapse in some interesting ways because of AI?

Steven Sinofsky 46:17 Well, I, I think that, like if you actually s-stick with the medical example, they're, we're just gonna wake up and there's gonna be way more people with way more specialties.

Erik Torenberg 46:24 Right.

Steven Sinofsky 46:25 And, and AI will have created-

Erik Torenberg 46:27 So you think it adds

Steven Sinofsky 46:28 ... more jobs.

Erik Torenberg 46:28 Yeah.

Steven Sinofsky 46:28 And in the interim, the-

Erik Torenberg 46:29 Do you think AI causes more specialization over time?

Steven Sinofsky 46:31 Absolutely.

Erik Torenberg 46:32 Yeah.

Steven Sinofsky 46:32 'Cause everyone's gonna, every human is gonna be way better-

Erik Torenberg 46:34 Right

Steven Sinofsky 46:35 ... and, and more knowledgeable about. And I think this is a thing that, that has, has really happened with computing that people forget. Like, there used to just be like this morass of marketing-

Erik Torenberg 46:46 Right

Steven Sinofsky 46:46 ... and R&D.

Erik Torenberg 46:47 Right.

Steven Sinofsky 46:48 And all of a sudden, like just, just ... And there used to just be coding, and then there was coding and testing and design and product management and program management and, you know, usability and research and all of these specialties. And all of those had their own tools.

Erik Torenberg 47:01 Right.

Steven Sinofsky 47:01 Go to a construction site. I, I remember growing up, these, our neighbors built a house. We lived in an apartment, and they built a house, and there was Clem, the carpenter.

Martin Casado 47:08 Yeah.

Steven Sinofsky 47:08 And you built a house with a guy named Clem, who used all the tools and everything. And now, like you build a house, and it's like this 20-person list of sub-subcontractors, all who have whole companies that do nothing but like put in pavers, you know? And, and that's what it's gonna be.

Martin Casado 47:23 I mean, there, there's been a, there's been a long disaggregation in the history of IT, right? Like everything in the same sheet metal, then, you know, disaggregate the OS and the hardware, then you disaggregate the apps.

Steven Sinofsky 47:33 Right.

Martin Casado 47:33 Um, and then it was kind of interesting, like in the last 15 years, we saw the app, and like independent functions got disaggregated, right? It's like almost everything became, like, like an API would become a company, right? You'd have like-

Steven Sinofsky 47:44 Yeah

Martin Casado 47:44 ... Twilio's like Auth became a company, like PubSub became a company, et cetera. And so it may very well be the case that every agent, it becomes like a whole new vertical and a whole new-

Steven Sinofsky 47:55 Right

Martin Casado 47:55 ... specialization.

Steven Sinofsky 47:56 Well-

Martin Casado 47:56 And then, and then you can n- actually build a company ar-around it.

Steven Sinofsky 47:59 Yeah.

Martin Casado 47:59 Like it may, it may be the case that today, just like with APIs, one company will have a whole bunch of agents.

Steven Sinofsky 48:04 Right.

Martin Casado 48:04 It may be the case in the future that a third party will provide that agent as an independent company.

Steven Sinofsky 48:08 Well, it's, it's so-

Erik Torenberg 48:08 Right

Steven Sinofsky 48:08 ... it, it's, the, the opportunity, to your point, is, is really there for that-

Erik Torenberg 48:13 Yeah

Steven Sinofsky 48:13 ... because, like there, it used to be like the, the impedance to creating a company and distributing-

Martin Casado 48:20 No, exactly

Steven Sinofsky 48:20 ... was, was infinite.

Martin Casado 48:21 It used to be, it-

Steven Sinofsky 48:21 And, and so, you know, if you look at-

Martin Casado 48:22 It used to be ridiculous to think that a single API like Auth could become a company, but then, you know, of course it became a-

Steven Sinofsky 48:27 Or, or it used to be ridiculous to think you could build a whole company out of signing documents.

Erik Torenberg 48:30 Right.

Steven Sinofsky 48:31 And, and like the, not just-

Erik Torenberg 48:33 Yeah

Steven Sinofsky 48:33 ... a whole company, but then all of a sudden you realize, wow, the addressable market for that is huge.

Erik Torenberg 48:37 Right.

Steven Sinofsky 48:37 And it's way bigger than signing because of all the stuff that got done that was just-

Erik Torenberg 48:42 Right, surrounding

Steven Sinofsky 48:42 ... uh, baked into a company causing headcount and waste and abu- and fraud and abuse.

Erik Torenberg 48:47 Well, I, I think you can kind of underwrite thousands of, of, of these companies emerging. So-

Martin Casado 48:52 Yeah

Erik Torenberg 48:53 ... uh, Jared Friedman had a tweet, um, uh, about basically like go deep on a, on a workflow. Um, you know, take, basically do the job of, of some part of the economy, payroll specialist, and then build an agent for that.

Martin Casado 49:06 Yeah.

Erik Torenberg 49:06 And it's not obvious that there's not literally a, like 1,000 of those.

Martin Casado 49:09 Yeah, yeah, yeah.

Erik Torenberg 49:09 So by every vertical-

Martin Casado 49:11 That's a whole new segmentation

Erik Torenberg 49:11 ... and every line of department and-

Steven Sinofsky 49:12 I just love this because this is like literally the anti-AGI spiel.

Erik Torenberg 49:14 Yeah.

Steven Sinofsky 49:15 It's basically-

Erik Torenberg 49:15 Right

Steven Sinofsky 49:15 ... following like the long arc of computer science-

Erik Torenberg 49:17 Right

Steven Sinofsky 49:17 ... where as the market grows, the level, the granularity can create a company.

Erik Torenberg 49:20 Well, it's also economic growth. Like take that-

Steven Sinofsky 49:22 Right

Erik Torenberg 49:22 ... payroll example.

Martin Casado 49:23 Exactly, exactly.

Erik Torenberg 49:23 Like today, just like Salesforce, which is always my favorite example, like the idea of having a produc-productive sales force used to just be a consultancy.

Martin Casado 49:31 Right.

Steven Sinofsky 49:31 And the only way you could ever fix it was hiring a consultancy to show up and analyze what everybody does, and then do a report-

Martin Casado 49:37 Right

Steven Sinofsky 49:37 ... that says, "This is how you need to reorg." And it usually meant go the opposite of whatever you had.

Martin Casado 49:41 [laughs]

Steven Sinofsky 49:42 A- and then they would leave. And then, you know, people tried, but there was no cloud. So to build-

Martin Casado 49:48 Right

Steven Sinofsky 49:48 ... like CRM, you had to do all that consulting work, and then roll it out. And, and then it was static, and you couldn't maintain it.

Martin Casado 49:55 Right.

Steven Sinofsky 49:55 And then all of a sudden there's like, "Oh, here's Marc Benioff, and here's a whole way to do all this." And not only that, the people actually like it.

Martin Casado 50:02 Right.

Steven Sinofsky 50:02 And they think they're better at selling, selling because they're, they're using their phone and they're putting in a, a few notes about this client-

Martin Casado 50:09 Right

Steven Sinofsky 50:09 ... which helps everybody. And I think that's what's really gonna happen with all this. And so suddenly something that looks really, really small-

Martin Casado 50:17 Right

Steven Sinofsky 50:17 ... becomes like a whole thing because there's no problem with distribution.

Martin Casado 50:20 That's awesome.

Steven Sinofsky 50:20 There's no problem with customization. You know, we'll actually have ways to solve security and privacy-

Martin Casado 50:25 Yeah

Steven Sinofsky 50:25 ... at, just like we solved reliability and, and things like... And, and I think it's just... And I mean, look at, you know, the stuff that you're a world expert in, in the stack of internet technol- of networking technologies.

Martin Casado 50:35 Yeah.

Steven Sinofsky 50:35 I mean, uh, you would've asked me 15 years ago, "Would CDN be companies?"

Martin Casado 50:39 [laughs]

Steven Sinofsky 50:40 I never would've... I, I'm like, that doesn't make any sense. Like, how could you have a company that's a cache?

Martin Casado 50:44 Yeah.

Erik Torenberg 50:44 Yeah. I think that people are probably way too afraid of the model providers kind of eating them. Um, and I th- I think it was-

Martin Casado 50:52 Oh, yeah, for sure

Erik Torenberg 50:52 ... I think it was basically, uh, a phenomenon in the first wave, which was if you were just doing like basic like... Like, if you had figured out that you could do something on GPT, you know, 2 and 3, where it was a text interface that produced more text, like yes, ChatGPT ate you. Like, like that, that clearly happened.

Steven Sinofsky 51:09 Yeah.

Erik Torenberg 51:09 But basically since then, most enterprises want kind of applied use cases for AI and AI agents. And so, so it's not obvious that the current crop of companies, if you're doing AI for healthcare, if you're doing AI for life sciences, if you're doing AI for financial services, if you're doing AI for coding at the right parts of the stack. AI, AI, AI for coding may be the one asterisk area which will be hypercompetitive simply because the model companies like don't want to use somebody else's product to build their own models.

Steven Sinofsky 51:36 Right. Right.

Erik Torenberg 51:36 And so that kind of almost forces them to get really good at AI for coding.

Steven Sinofsky 51:39 Yeah.

Erik Torenberg 51:39 But with that as the one kind of, you know, exception, I think basically we're just in a five-year period right now where you're gonna have to build agents for every vertical, every domain, and there's a playbook that's starting to emerge of what that needs to look like.

Martin Casado 51:52 I mean, so I, I think there was kind of a technical head fake that happened early on, which was pre-training. So the pre-training really was a 10 out of 10-

Erik Torenberg 52:00 Mm-hmm

Martin Casado 52:01 ... technical innovation. I, I can't tell you like two years ago if somebody was... Like, I had a friend that was building, um, like their own aging model, post-training aging model. Like, "We're gonna make it so good at aging." Like, you know, this is a, this is a, a, um, a text-to-image model, and they wanted to make it so like old people looked really good at it. And then of course, the next version of like Midjourney, whatever, comes out-

Erik Torenberg 52:20 Yeah

Martin Casado 52:20 ... and it does a better job of it. And the thing with pre-training was you're just kind of consuming all of the world's-

Erik Torenberg 52:24 Right

Martin Casado 52:24 ... existing data. You're draining all of that energy, and it perfectly generalized, right? But it feels like technically that's passed, and now we're more in post-training and RL-

Erik Torenberg 52:33 Yeah

Martin Casado 52:33 ... which is a lot more domain specific. And so-

Erik Torenberg 52:35 Well, and the moment that you have access to some set of data that is only-

Martin Casado 52:39 Exactly

Erik Torenberg 52:39 ... just for that enterprise, and so who gets permission to access that data? Who gets permission to do the workflow on it? It's gonna be applied companies.

Martin Casado 52:46 Yeah. So yeah, and in this, if we had an infinite number of tokens, then the models would just continue to generalize, but it's pretty clear that that's not happening.

Erik Torenberg 52:53 Yeah.

Martin Casado 52:53 And so now we're going into, which we all understand very well, which is now companies have to choose which domains they go into.

Erik Torenberg 52:59 Right.

Martin Casado 52:59 And they gotta solve the long tail of problems there and get access to the data, et cetera. So-

Steven Sinofsky 53:02 And I also think that there's, that the shadow, having been the shadow, the shadow cast by large companies over, "We're gonna put you out of business and stomp you," it, it's ridiculous.

Martin Casado 53:11 Mm.

Erik Torenberg 53:11 Yeah.

Steven Sinofsky 53:12 And it, it has never in any technology wave lived up-

Martin Casado 53:15 Yeah

Steven Sinofsky 53:15 ... to the fear that people have. Look, if you built a new word processor in 1995, you were an idiot.

Martin Casado 53:20 Mm-hmm. [laughs]

Steven Sinofsky 53:21 Like, that was not the thing to go build.

Erik Torenberg 53:24 Yeah.

Steven Sinofsky 53:24 You know, and, and but, you know, there was a time just 10 years earlier where like companies built standalone spell checkers.

Erik Torenberg 53:31 Mm-hmm.

Steven Sinofsky 53:31 Like, it was just a thing. You went to the store and you bought a spell checker. And, and like it had more words than the other spell checker. And, and so the, the, the thing is that that's not being said now, which we should do a whole one on, is like what is the actual platform?

Erik Torenberg 53:45 Yeah.

Martin Casado 53:46 This is great.

Steven Sinofsky 53:46 Because you-

Martin Casado 53:47 This is a great topic

Steven Sinofsky 53:47 ... like it's very g- it's all well and good to say that the, the large models will go subsume every application. The thing is, the minute they start doing that, no one will be in their platform.

Martin Casado 53:56 Yeah, this is a great topic.

Erik Torenberg 53:56 Right.

Steven Sinofsky 53:56 Because like, like no developer is gonna sit around and say, "If you're gonna subsume me-

Erik Torenberg 54:01 Right

Steven Sinofsky 54:01 ... then..." And, and this is, there's a phrase that it's Sherlocking on the Mac-

Erik Torenberg 54:05 Yep

Steven Sinofsky 54:05 ... and the Apple world-

Erik Torenberg 54:06 Yeah

Steven Sinofsky 54:06 ... to this thing. And so it does, it has a real chilling effect, and that's one of the things all the model people are gonna learn very, very quickly.

Erik Torenberg 54:12 There's a chilling effect, but there's also just, I think there really is just a problem of like it's hard to go deep in 50 categories.

Steven Sinofsky 54:18 It, it-

Erik Torenberg 54:18 Like, you just, you just can't, you can't-

Martin Casado 54:20 Model pre-train, I think everybody's scared 'cause pre-training-

Erik Torenberg 54:22 Yes. Yes

Martin Casado 54:22 ... was actually the one thing that was good at that.

Erik Torenberg 54:24 Well, it-

Martin Casado 54:24 And then now they have to actually too. Yeah, I agree.

Erik Torenberg 54:26 You do have to like... At, at some point it becomes purely just an execution issue, which is like, like I, I don't know how anybody would set up a company to be, to be able to beat 50 startups across 50 different domains.

Steven Sinofsky 54:37 No, it's ridicul- and in fact, like it's only good-

Erik Torenberg 54:40 Right

Steven Sinofsky 54:40 ... because the, what happens is, is that the, the, the big company

Steven Sinofsky 54:44 raises, the, the big company raises the awareness of a whole category.

Erik Torenberg 54:47 Right. Right.

Steven Sinofsky 54:48 And then you just swoop in and you go, w- you, to them you're, I'm just a feature.

Erik Torenberg 54:54 Right.

Martin Casado 54:54 Yeah.

Steven Sinofsky 54:54 But to, to you, I'm my, this is my whole life.

Erik Torenberg 54:57 Right.

Steven Sinofsky 54:57 And, and you're gonna win. Look, I just, I always come back. There's a whole company that just signs things.

Erik Torenberg 55:01 Right.

Martin Casado 55:02 [laughs]

Steven Sinofsky 55:02 I, I, I like I can't, I cannot believe there's a whole company-

Erik Torenberg 55:05 Yeah

Steven Sinofsky 55:05 ... that just signs things.

Martin Casado 55:06 I have so much to say about this topic. I mean, even minimally, if you graph like, like the cost to produ- the, the, so the willingness to pay for, um, an inference versus the cost to serve it.

Erik Torenberg 55:19 Mm.

Martin Casado 55:19 Something like for most companies, for most spaces, 20% of the inferences are 80% of the cost. So like actually the problem of the application is just to choose those ones on, which tend to be-

Erik Torenberg 55:28 Right

Martin Casado 55:28 ... more domain specific.

Steven Sinofsky 55:29 Yeah, yeah.

Martin Casado 55:29 This is the problem of inviting the three of us on here-

Erik Torenberg 55:31 Okay. [laughs]

Martin Casado 55:31 ... which is like getting us to talk is easy.

Steven Sinofsky 55:32 We just overbooked like the next two hours.

Martin Casado 55:35 Just getting us to shut up is the trick. [laughs] And so-

Erik Torenberg 55:37 Yeah. Guys, thank you so much for coming on. This was fantastic. [upbeat music]