Scott Wu: Why Devin will write half of Cognition's code

1. 0:00 – 9:13

   Introduction to Scott Wu and Devin

   1. SWScott Wu0:00

      Our whole team is only like 15 engineers. We use a ton of Devin when we're building Devin. Most folks on the team are definitely working with up to five Devins at once. And so Devin merges like several hundred pull requests into production in the Devin code bases every month.

   2. LRLenny Rachitsky0:12

      What percentage of your PRs are Devin versus humans right now?

   3. SWScott Wu0:16

      It's in the neighborhood of a quarter or so.

   4. LRLenny Rachitsky0:19

      Where do you think this will be at the end of the year?

   5. SWScott Wu0:21

      Honestly, uh, we expect it to be a decent bit more than half.

   6. LRLenny Rachitsky0:24

      You guys are so ahead of how companies work with AI engineers.

   7. SWScott Wu0:28

      AI is going to be the biggest technology shift of our lives. So most of the big tech revolutions that we've had over the last 50 years like personal computer and the internet and the mobile phone, they all had this big hardware component that was a big part of the distribution. Folks who were building for those industries kind of saw their market grow and grow and grow, basically steadily year over year as the number of people with mobile phones increased, right? As the number of people connected to the internet increased. One of the things which is already I'd say different in AI is just how explosive the technology can be. There's no wait on hardware distribution. It means that the space is just growing so exponentially.

   8. LRLenny Rachitsky1:01

      How is the act of being an engineer and building changing?

   9. SWScott Wu1:04

      I think there's going to be way more programmers and way more engineers a few years from now. Pretty quickly the form factor of what it means to be a programmer obviously is going to change, but at the end of the day of course, the discipline is all about just being able to tell your computer what to do. And so in that lens I really think that programming is only going to become more and more important as AI gets more powerful.

   10. LRLenny Rachitsky1:20

       Today my guest is Scott Wu. Scott is the co-founder and CEO of Cognition which makes a product called Devin, the world's first autonomous AI software engineer. Unlike other AI tools that I've highlighted on this podcast, Devin is designed to act like an actual remote engineer that you chat with like you would with any other human engineer through Slack or through its dedicated website. When Devin launched about a year ago, it was very much a junior engineer. Over the past year, they've made a lot of progress and Devin is now being used by tons of companies in production. We chat about how their engineering team of 15 uses Devins to build Devin including how every engineer uses about five Devins each to help them code and move faster, how a quarter of their pull requests today are committed by Devins, and that they expect this to be over 50% by the end of the year. We also talk about how Scott imagines software engineering is gonna look in the future, and how the role of an engineer changes from a coder to an architect. We also get into the eight pivots that they went through before landing on this path, why Scott believes AI tools like this will lead to more engineer hiring versus less, also where the name Devin comes from, and so much more. This episode is going to blow your mind. I highly recommend you listen to it if you're at all interested about where engineering, product building and AI is going. A huge thank you to Claire Vo for suggesting a bunch of great questions for this conversation. If you enjoy this podcast, don't forget to subscribe and follow it in your favorite podcasting app or YouTube. Also, if you become an annual subscriber of my newsletter, you get a year free of Linear, Superhuman, Notion, Perplexity, and Granola. Check it out at lennysnewsletter.com and click Bundle. With that I bring you Scott Wu. This episode is brought to you by Interpret. Interpret unifies all your customer interactions from Gong calls to Zendesk tickets to Twitter threads to App Store reviews and makes it available for analysis. It's trusted by leading product orgs like Canva, Notion, Loom, Linear, monday.com and Strava to bring the voice of the customer into the product development process, helping you build best in class products faster. What makes Interpret special is its ability to build and update customer-specific AI models that provide the most granular and accurate insights into your business, connect customer insights to revenue and operational data in your CRM or data warehouse, to map the business impact of each customer need and prioritize confidently, and empower your entire team to easily take action on use cases like win-loss analysis, critical bug detection, and identifying drivers of churn with Interpret's AI assistant Wisdom. Looking to automate your feedback loops and prioritize your roadmap with confidence like Notion, Canva, and Linear? Visit E-N-T-E-R-P-R-E-T.com/lenny to connect with the team and get two free months when you sign up for an annual plan. This is a limited time offer. That's interpret.com/lenny. Many of you are building AI products, which is why I'm very excited to chat with Brandon Fu, founder and CEO of Paragon. Hey, Brandon.

   11. NANarrator4:15

       Hey, Lenny. Thanks for having me.

   12. LRLenny Rachitsky4:17

       So, integrations have become a big deal for AI products. Why is that?

   13. NANarrator4:21

       Integrations are mission critical for AI for two reasons. First, AI products need context from the customer's business data such as Google Drive files, Slack messages or CRM records. Second, for AI products to automate work on behalf of users, AI agents need to be able to take action across these different third-party tools.

   14. LRLenny Rachitsky4:40

       So where does Paragon fit into all this?

   15. NANarrator4:42

       Well, these integrations are a pain to build and that's why Paragon provides an embedded platform that enables engineers to ship these product integrations in just days instead of months across every use case from RAG data ingestion to agentic actions.

   16. LRLenny Rachitsky4:56

       And I know from firsthand experience that maintenance is even harder than just building it for the first time.

   17. NANarrator5:01

       Exactly. We believe product teams should focus engineering efforts on competitive advantages, not integrations. That's why companies like U.com, AI21 and hundreds of others use Paragon to accelerate their integration strategy.

   18. LRLenny Rachitsky5:15

       If you want to avoid wasting months of engineering on integrations that your customers need, check out Paragon at useparagon.com/lenny. Scott, thank you so much for being here and welcome to the podcast.

   19. SWScott Wu5:29

       Thanks so much for having me. Excited to be on.

   20. LRLenny Rachitsky5:31

       I'm really excited to have you here because you are building and you've been building something that is very different from what a lot of other AI companies have been doing for a long time, although they are starting to converge (laughs) to where you guys are now. We're gonna talk about that. And it's also just such a unique point in the history of AI and just the journey of AI, and so it's really cool to be chatting right now. And I feel like we're gonna chat again in a few years and be like, "Wow, we were so right about so much and so wrong about so much."

   21. SWScott Wu5:59

       Yeah.

   22. LRLenny Rachitsky6:00

       And so I'm excited to have you here. Let's start with talking about Devin, giving people an understanding of what, just what the heck Devin is. This is the main product that you guys build. What is the simplest way to understand what is Devin?

   23. SWScott Wu6:10

       Absolutely. And so Devin is a fully autonomous software engineer that is gonna work on tasks end-to-end. And so there are a lot of great tools for all- all parts of the stack of the AI code workflow. What Devin does is it is a- a- a full asynchronous workflow. And so you can tag Devin on an issue in Slack, you know, you're talking about De- uh, uh, an issue and you tag Devin, you can tag Devin in Linear, you can have Devin... A- and Devin will make pull requests in your GitHub. And so it's very much built to, to work with engineering teams a- a- as your junior engineer.

   24. LRLenny Rachitsky6:37

       Amazing. Okay. So I remember when you guys launched this, there was like this big pitch of this is your new AI engineer. And it was, it was really good at a lot of stuff. It wasn't great at other things. It's been a year now about since you guys launched, is that right?

   25. SWScott Wu6:50

       Yeah, yeah.

   26. LRLenny Rachitsky6:51

       What's the s- best way to think about, like, the level of seniority that engineer had back in the day when you guys launched, and then the level of seniority of engineer today, if that's, I don't know, measure of how to think about Devin?

   27. SWScott Wu7:02

       Yeah, and it's crazy to think about, by the way, because, you know, a year ago when we did the initial launch, I mean, people didn't really believe that an agent was possible, (laughs) right? And, and it was a very... I mean, it was a very different time. You know, it's, uh, uh, uh, it's like start of 2024, you know, things with model capabilities were definitely quite a bit earlier on. Uh, reasoning especially was, was quite a bit earlier on. A- and, uh, and- and yeah, I mean, uh, in the time since then, it's obviously developed a lot, I think, in terms of, uh, practical skills. You know, we, we... There, there are some comparisons we make. Sometimes we kind of say, "Well, when we got started, it was kind of like a, a high school CS student, and then as time went on, it became more of like a college intern, and now it's like a junior engineer." But, but I would say though that tho- those are more like rough guidelines because... I- I- I really like the phrase jagged intelligence, for example, be- because there are... Obviously, there are certain things that it is much better at than a human, there are certain things that are- it's much worse at than a human. And I think over the last year, we've, we've learned a lot, especially about not just coding agents but agents in general, just like really building out, like, how all of us should be working and interacting with agents as part of our flow. And so a lot of the things that we built, I mean, it's... You know, there was no Slack, uh, there was no GitHub i- integration, there was no Linear, there was no interactive planning phase working back and forth. There was no way to touch up Devin's code. And, and so a lot of the, the, the features that we've built on the product side since then have really been about basically, yeah, figuring out how to, how to make working with Devin and handing off tasks to Devin as smooth of an experience as possible.

   28. LRLenny Rachitsky8:27

       That's so interesting. So a lot of the work has gone not into how do we just make Devin the best possible engineer, but it's how to work with this new type of entity that we haven't ever worked with.

   29. SWScott Wu8:37

       I think it's a 50/50 of both. You know, I, I think the, uh, the, the capabilities obviously, you know, ha- have improved a ton, and we've seen these get better and get measurably better. But I think the other side of it is everything to do with, yeah, really the product interface and the tools and so on. And, and I think, you know, I think today, folks generally know how to use chatbots and to work with chatbots, right? And that's an interface that, that people are familiar with. And, and obviously with agents, it's... You know, it's, uh, the... It's, it's still like a real curve, I think, to, to learn how to use them and how to get the most out of them. And so it's, it's really exciting to see a lot of others starting to, to build and do a lot more in the agent space as well. But I, I, I think this is the kind of thing that we're, we're all really

2. 9:13 – 10:23

   Scaling and future prospects

   1. SWScott Wu9:13

      figuring out together as a space.

   2. LRLenny Rachitsky9:14

      What can you share about just the scale of Devin at this point, whatever you're comfortable sharing, and then just where do you think the level of Devin's coding abilities will be in a year?

   3. SWScott Wu9:24

      So we work with companies of all stages and sizes. You know, uh, uh, on the smallest end, it goes to, you know, startups of just one or two people who are using Devin to, to build out a lot of their kind of like, uh, initial prototypes or initial product all the way up to, you know, big public companies, Fortune 100 companies or, uh, or public banks or things like that who are using Devin, like, across their, um, across their engineering teams. In general, it's... You know, we've, we've seen a huge range of, of the use cases there. And obviously, the kinds of engineering work that you're doing at a one or two-person startup is very different from the kind of work that you're doing at a, at a public bank. But, but throughout, it's all been... Uh, basically, yeah, being, being that junior buddy of yours that- that e- that makes you go faster and, and really multiplies you, I would say. You know, I, I think there's, uh... It can multiply you as an engineer obviously by just, like, letting you work with your own team of Devis, uh, instead of having to be kind of like fully synchronous on a single task. And then it's also kind of like m- multiplying your team and multiplying your, your team's knowledge base, because Devin really accumulates a lot of the knowledge, uh, from, from working with every member of your team and is able to bring that into each new session.

3. 10:23 – 17:26

   Devin's origin story

   1. SWScott Wu10:23

   2. LRLenny Rachitsky10:23

      Awesome. We're gonna show people how it actually works later in the podcast. We're gonna do a few live demos. But let's actually go to the beginning of the journey. What's just the origin story of Devin? How did this all begin?

   3. SWScott Wu10:33

      The, the founding team, I mean, most of us have known each other for, for years and years and years actually. And, and for, for almost everyone, this is our first time working together, but we've known each other a long time, and we all actually had our own kind of journeys in, in AI for the last, last decade or so. Um, and so for myself, you know, uh, I, I ran a company called this... Uh, before this called Lunch Club, which was an AI for a professional networking product, and I ran that for about five years. And, you know, my co-founders... One of my co-founders, Steven, was, was one of the first engineers at a company called Scale AI, which has obviously grown a lot and done very well. Uh, my other co-founder, Walden, was an early engineer at a company called Cursor, which has also obviously grown a lot and done really well. And, uh, a- a- a- and our whole team kind of like... Was kind of like that, you know? Many of us knew each other from, from, from competitive programming and math competitions, but we had stayed very closely in touch, you know, in the, i- in the decade since then, and we've all kind of all had our own journeys. And so, you know, we had, we had one person who was running teams at Neuro. We had one person who was at Waymo, someone who had their own YC Tools startup, uh, for, for machine learning. And, and, uh, and we were really excited to build something together. A- a- and this was around, like, late 2023, um, so about a year and a half ago at this point. And yeah, when we got started, I m- I mean, I think there, there were a couple of things that we felt really strongly about, and one was that reinforcement learning was really working and was going to be the next big paradigm shift in capabilities. You know, back then, it was, you know, the initial ChatGPT launch in 2022, and, and, and those models were to first order what, you know, what we would call imitation learning i- in, in AI, right? Which is basically, you know, you have the model read all the texts that you can find on the internet and then train it to talk like somebody on the internet would talk, right? A- a- and, and there, there are kind of, o- obviously a lot more details on top of that, but that's, that's kind of the first order pass of, of what was really done. And it was amazing, right? I mean, it passed the Turing test. It, it was able to respond and to, to have encyclopedic knowledge about a lot of things. And I think this new paradigm, which we've gotten into over this last year or year and a half, is, is really high compute RL, which is, which is a very different paradigm, right? Which is basically the ability to go and...... do work on tasks and put something together and then be evaluated on whether that was correct or incorrect, and use that knowledge to decide what to do and, and, and to learn from that, right? And so, you know, uh, we felt very strongly that that was gonna happen. I think it, for us is, code was the natural thing to work on for a couple reasons. One, because, you know, we're all programmer nerds ourselves (laughs) and so teaching AI to code is about as cool as it gets for us. But, but also because, you know, code has this whole automated feedback loop, right, where you can run the code. And, and that is the kind of automated feedback that really feeds into, to the RL, which makes these models so great at coding. And then the other thing that we felt very strongly about was that the product experience was going to shift from, you know, what I'll call, like, text completion to agents, basically, right? Uh, and, and, and to first order, I would kind of say, you know, there, there have been a lot of great experiences in text completion. You know, it's, it's been used for marketing, it's been used for customer support, it's been used for education. And in coda, obviously, as well is, you know, the GitHub Copilot was, was, was kind of really the, the, the, the dominant product of the- that initial wave, right? But I think the, the, the big shift that we really felt we would see is moving from kind of this, this text-to-text model to an actual autonomous system that can make decisions, that can interact with the real world, that can take in feedback, that can iterate and, and take multiple steps to solve problems. A- and, you know, now we call that agents, but, but, but that was what we were really excited about at the time. So it was always coding, it was always agents, and in some ways that kind of, it feels like it's, it sh- it should have been, you know (laughs) , like, uh, been, been clear from the start. But even with that it's, I feel like we've pivoted like eight times or something within coding agents, you know, over the last year and a half, so...

   4. LRLenny Rachitsky14:08

      I just noticed recently all the AI, top AI companies, sort of, not all, but many of them, the product that is winning is different, has a different name from the company, which is not typical. Cursor is AnySphere, Bolt is StackBlitz, you guys are Cognition Labs, uh, like v0 is Vercel. And it just tells me, like, these all emerged later in the company's journey and they tried a bunch of stuff and like, "Oh wow, this thing worked." And it's so-

   5. SWScott Wu14:32

      You-

   6. LRLenny Rachitsky14:32

      ... interesting that it's so common amongst these top AI companies.

   7. SWScott Wu14:34

      Yeah. And there's even, I mean, OpenAI, ChatGPT, Anthropic and Claude and Google.

   8. LRLenny Rachitsky14:38

      Mm. (laughs)

   9. SWScott Wu14:38

      Yeah. It's, it's, it's funny. Yeah, yeah, yeah.

   10. LRLenny Rachitsky14:40

       (laughs)

   11. SWScott Wu14:40

       I agree. So, so, you know, when we got started, it, it wasn't even really a company. I mean, it was more like a project or a, or a hackathon almost. You know, we, we got, uh, a bunch of... Uh, we, we booked an Airbnb basically for a couple weeks. This was around Thanksgiving time. And, and, and just got a, a, a, a bunch of people together who were just excited to, uh, to hack on some projects and build something cool. And it's funny actually, the first, the first thing that we were building for actually was more like solving, like, uh, these more, like, contest programming problems and using, like, an agentic loop to, to really do better on that. And so obviously if you run your code on test cases, you can evaluate, you know, there, there's a lot of agentic work that you can do there to try and do better. And, and that, that we spent some time on that initially. And then, you know, we, we've kind of gone from ha- I mean, the, the, the, the story of the whole company for us in some sense has been going from hacker house to hacker house, you know? (laughs) So, so, so after that, you know, we had another hacker house, and that's where kind of some of the initial ideas for Devin came, and, and really building, like, a software engineering agent and not just, like, a, a coding agent and having it interact with a lot of these tools. But, but even then, there were so many iterations and, you know, even, like, the idea of talking to Devin, for example, was like a... You know, it was something that we had to come up with, right? Initially it was just like you hand off a task and then it works and then it shows you, like, this whole finished code, right? And now obviously it's like you can jump in at any time, you can give feedback on the plan, you guys can scope out the task together, you know, when you're working with Devin. A- a- and a lot of these things we, we had to develop, obviously, and certainly we've learned a lot about the use cases, the form factor. We've made a lot of big improvements, uh, and, and step function improvements on the capabilities and, and Devin's ability to use tools and debug and make decisions. And, and so it's, yeah, it's been a fun journey. I mean, I think, like, I, I, I would say that the grounding question for us really as the... which is one that we think about all the time is, is really just like, what, what is the future of software engineering, you know? A- a- and how, how should we be working with AI to write code? 'Cause I think at the end of the day, of course that's, that's, that's what underlies all of the, the, the, the product decisions that we make, so.

   12. LRLenny Rachitsky16:34

       I like that you're asking the juicy question I wanted to get to. Before I ask it, is... Just for the history books, how... When did you guys start kind of hacking around and when did Devin launch? What was that... How long was that period?

   13. SWScott Wu16:44

       Yeah. So we started in November of 2023, which was in... Yeah. It was just, like, hackathon mode. We officially made it into a company around the start of 2024, and then our, our initial launch was in March. And so it was like non-stop... I mean, it's, it's been non-stop for the entire last, you know-

   14. LRLenny Rachitsky17:00

       Okay.

   15. SWScott Wu17:00

       ... 17 months (laughs) but-

   16. LRLenny Rachitsky17:01

       Fast.

   17. SWScott Wu17:02

       You know, it's, it's getting, getting to the launch and then obviously working with enterprises and, uh, developing the product a lot more, uh, building in... Uh, building in and, and getting it to work for a lot of practical use cases, a- a- a- a- and then doing, g- you know, getting it, making it fully available self-serve in December of last year. And, and now we've rolled out 2.0 obviously just a few weeks ago. And so it's been a, been a very busy time for us.

   18. LRLenny Rachitsky17:25

       Understatement of the century.

4. 17:26 – 22:19

   The idea of Devin as a person

   1. LRLenny Rachitsky17:26

   2. SWScott Wu17:26

      (laughs)

   3. LRLenny Rachitsky17:27

      Let me ask this question 'cause you touched on it a bit, this whole idea of Devin as a person and this idea-

   4. SWScott Wu17:31

      Yeah.

   5. LRLenny Rachitsky17:31

      ... of creating a personality. For Devin it's unlike any other, I believe, AI app. No one else has, like, a name and like a... You don't think of it as a person. What made you guys decide to go with that approach? And just how do you design it to work well that way?

   6. SWScott Wu17:44

      I would say we're... It's a decision we're pretty proud of, I would say. I, I, I mean, I think there's a lot of different product experiences out there, and I think the thing that really makes Devin unique in what it does is that you can really hand off. And, and, and more and more what we've seen, honestly, is, uh, that, that I, I think a lot of kind of explaining the D- the Devin experience for folks is really just explaining to, uh, explain it as, "Yeah, this is your junior buddy." You know? And, and, and that goes for a lot of the, the parts of the flow where in the onboarding, for example, you know. Initially I would say, like, we've definitely had a lot of users come in and just kind of see the blank screen and not really know or, or they'd, they'd ask, "Hey," like, "I'm gonna do this whole big re-architecture of the whole code base." And, and basically, you know, what, what, what we've learned over time is to basically get folks to think more like, "Whoa, whoa, whoa." You know? "Let's work on getting the re- repository set up first." Like, "Let's make sure we, we hand Devin a couple one-pointer tasks so it can get familiar with the code base." You know, "Let's get it to think..." I- if Devin needs to be able to test the code or run the linter or CI or, or things like that, obviously we wanna make sure Devin's got its own virtual machine set up to be able to do that. A- a- and similarly, like, I think the usage pattern. You know, I, I think o- often it wasn't clear and, and obviously you can sit and just kind of watch Devin do it action by action, uh, a- a- and work that way, but, but we, we found that the, you know, the, the best workflow really as a team building off stuff was to, to work with multiple Devins and to run them asynchronously and to, to kick them off and, and to only jump in basically as you needed to provide feedback or, or, or, or steer the plan or anything like that.And so in many ways, I think it's... I think Devin as a name really is, is, is our attempt to kind of capture the soul of that as a product, where it really is, you know, treating it like a- a- a bit more of an autonomous entity that, that you can, you can hand off tasks, that you can work with, that you should be teaching and learning with over time.

   7. LRLenny Rachitsky19:28

      I wanna come back to an area you started us down and then I took us away from, which is impact on software engineering and then how software engineering's gonna change. So there's kind of two parts to this. Just like when people are using Devin today, say in the ne- like this year, how, how has the act of being an engineer and building changing for those companies? What does that look like?

   8. SWScott Wu19:49

      You know, by the way, we're all, we're all software engineers ourselves, you know? It's like, uh, I- I'm a programmer by training and still a programmer at heart certainly. Uh, a- and, and, you know, I- I think the way that we have always thought about it is there's layers of abstraction and there's tools. Uh, a- and one way I would say it at a high level is kind of, you know, I- I think of AI in general as, yeah, I mean, computers are obviously getting more and more intelligence and are able to do more and more. And, you know, it's, it's possible there may come a day where computers truly do everything that we do and humans are not (laughs) responsible for any of it. You know, I- I don't expect that to come particularly soon. Um, but, but, but I guess what I would say is, you know, until that point, for as long as we're still part of the equation, one of the most important things to do obviously is, you know, for, for us as humans is, is to instruct our computers on, on what we want and what we wanna build and what we wanna do, right? And, and software engineering is, you know, we think of it today obviously as Python and C++ and JavaScript and, and all these things. But at the end of the day, of course, the discipline is all about just being able to tell your computer what to do. And so in that lens, I really think that programming is... if anything is only gonna become more and more impo- important a- as AI gets more powerful. And, and I think the thing that's really, that's, that's, that's really exciting for us is, yeah, is, is like really like seeing that, that kind of iterative transformation. And so you asked how, what things look like today and I, I would say yeah, it's, it really is like having like a junior buddy, uh, or really a team of junior buddies that you can work with, right? And so every engineer on our team pres- you know, we use a ton of Devin when we're building Devin. And so Devin merges like several hundred pull requests into production in the Devin code bases every month, you know? Uh, which is, I mean, our whole team is only like 15 engineers and so it's (laughs) it's a, it's a pretty sizable fraction of all the code that we write. A- a- and the way that we use it is basically, yeah, everyone's got their whole team of Devins, you know? If, if you're going to be like looking through various issues, if you're going through feature requests, if you're going through bugs, if you're going through new paradigms that you wanna build, um, then it is naturally the case that there's a lot of handoff points where you just say, "Hey, @Devin, here's what's going on. Can you, uh, please take a, take a pass at this?" Right? And sometimes Devin will be able to do the task 100% autonomously and just makes the PR and then you merge the PR and that's great. Sometimes we wanna be able to, to, to jump in for the 10 or 20% that really needs your help. Maybe there's, uh, a few details with how exactly you wanna scope it or how you're architecting this feature or maybe you wanna go and test the front end at the end yourself to make sure it looks exactly the way that you want and, and give your, your one or two lines of feedback after that, right? But a lot of it is, is really, yeah, is kind of like, yeah, learning, learning to, learning to work with Devin, uh, to, to be able to just do more

5. 22:19 – 25:17

   How a team of “Devins” are already producing 25% of Cognition’s pull requests

   1. SWScott Wu22:19

      in parallel and build more.

   2. LRLenny Rachitsky22:20

      What percentage of your PRs are Devin versus humans right now?

   3. SWScott Wu22:25

      Yeah. I, I, I'd have to look but it's in the neighborhood of, o- o- of a quarter or so of all of our PRs.

   4. LRLenny Rachitsky22:31

      Wow.

   5. SWScott Wu22:31

      Yeah, yeah.

   6. LRLenny Rachitsky22:32

      That's... And then what was it like six months ago?

   7. SWScott Wu22:35

      Oh, it's, uh, yeah. It's, it's, it's grown a ton for... I mean, we've seen it grow ex- exponentially internally at ourselves as well. Uh, and, and so it's, you know, it's, it's kind of an interesting one where again it's always both the capabilities and the product in- interface and so, you know, I think the intelligence has incr- increased a lot. But the other thing of course is that, you know, w- we've spent a lot of time in figuring out how to build and, um, to, to, to really kind of build for an interface where you can get Devin's value on tasks where Devin is able to do the 80 or 90%. And so Devin is obviously not, you know, it's obviously not perfect and it'll make mistakes and so on. And a lot of the question is basically, yeah, h- how do you scope out your initial task with Devin and then just kind of set Devin off and, and have it go and do the things that you wanna do? You know, how do you come in at the end and review and give feedback? How do you make sure Devin learns over time? How are you able to kind of just check in as, as, as needed and course correct if you want to?

   8. LRLenny Rachitsky23:24

      Okay, so today about quarter of your PRs are Devins. Where do you think this will be at the end of the year? What would you guess?

   9. SWScott Wu23:31

      I think by the end of this year we expect it to be more than half. And, and, and I mean as, as time goes on, you know, one of the things that we've seen is just... it's you're, you're able to do more and more and more work, uh, uh, asynchronously, right? And you're able to hand off more and more. You know, I- I think the soul of programming, the soul of software engineering has really been about, um, through, through all the areas. You know, not just now but, you know, even when it was Assembly, right, and even when it was, uh, it was Pascal and even when it was punch cards or whatever. You know, I think the soul of it has really been basically just about defining the problem that you're w- you're facing and really thinking through exactly what is the solution you, that you wanna build. You know, thinking through the architecture, thinking through the details and really kind of mapping out in, in your mind exactly what you want to, to build basically, uh, and what you want to have your computer do. And I think that's, uh, you know, that's, that's what makes software engineering really great and I think that's like the funnest part of software engineering. I- I think at the same time that's probably, you know, i- in the neighborhood of, of 10% o- of the average software engineer's time, right? Because 90% of the time is, you know, you've got this Kubernetes error that you've gotta debug and you have to see what went wrong and did the system crash or, you know, you left some port open and this is, you know, messing up or, you know, there's, there's a bug report that you have to take care of or you've gotta migrate your code or you gotta upgrade to a new version or, or things like that. You know, a lot more kind of like implementation. A- a- and one of the, the, the ways that we've kind of thought about Devin, uh, a- and building Devin is, is, is really allowing engineers to, to go from bricklayer to architect so to speak. And a lot of it is, is yeah, just I... getting to the point where, where you can do the high level directing and you can basically specify things exactly how you want. You know, I- I think there's very much about still having the human in control and having the human able to do the full specification but just multiplying the magnitude of, of what you can do and what you can build, you know, in, in one day or one hour or

6. 25:17 – 30:21

   Important skills in the AI era

   1. SWScott Wu25:17

      however long.

   2. LRLenny Rachitsky25:18

      So in the future, say someone is trying to get into software engineering, thinking about becoming an engineer. First of all, do you think people should, you know, classic question everyone's getting these days, should you still learn to code? So that's just-

   3. SWScott Wu25:30

      Yeah.

   4. LRLenny Rachitsky25:30

      ... I'd love your perspective there. And then two, for people that are engineers today, what skills do you think will be more and more important and then less important in this discussion of moving from bricklayer to architect?

   5. SWScott Wu25:40

      Yeah, for sure. I, I love this question. First of all, the question of, you know, whether you should still learn to code. My answer would be absolutely yes. I, I think to a large extent, you know, computer science, wh- when you take computer science classes and when you learn these fundamentals, sure you're learning a little bit about, you know, how a particular language's syntax works or something like that. But honestly, most of what you're learning really is about the ability to logically break down problems for number one. And two, I would say is just, yeah, the model of a computer and, and a lot of these decisions and a lot of the abstractions that we've built over time, right? Like, what is a database and how should you think about a database? You know, what is a garbage collection system and how do those work? And, and, and all of these different pieces. And, and the reason I think that's important is because it's, it's the same with a lot of these other... You know, a- arguably we've already go- kind of gone through these, these, these phases in programming, and I think this next one is going to be, you know, somewhat faster and somewhat bigger, but, but in many ways a similar flavor, which is, you know, when you work with Python today, obviously a lot of things are already abstracted away from you. And in some sense, you know, someone from 50 years ago might already call Python, you know, you just get to explain in English what you want and now the computer does it for you, right? A- a- and that's great and I think it's, it's really powerful. It's, it's opened it up. I mean, we have far more programmers obviously than we ever have before because of that. But, but I would say certainly, you know, a- a- as you're building your skills as an engineer, it really helps a lot to, to understand the abstractions and to be able to peel all the layers beneath, right? And so, you know, folks will use Assembly, for example, if they're really performance optimizing a piece of code. But also, you know, it's... In order to build good systems and to understand these things, you certainly want to understand these abstractions of, you know, how, how does networking work? What, what is TCP/IP like exactly, or what happens with this Python code when it gets interpreted or, or all of these details. And I think similarly, I think we will get to a state where yeah, you know, with, with no experience at all, you're going to be able to build some pretty cool stuff and, uh, and to do some pretty amazing work just by explaining what it is that you want. But I think that, you know, for, for quite some time, you, you really want to be able to, to think precisely about the details, to peel back the abstractions, to, to be very, very precise about what it is that you want to build and how.

   6. LRLenny Rachitsky27:48

      And then for skills that you think are more and more valuable for engineers, like where should engineers today be leaning more and more into, and versus like, you know, forget this, I don't need to think about this anymore?

   7. SWScott Wu28:00

      For sure. And I, and I think architect, I mean it's... You know, we already have a term for architect in engineering and I think it is directly the, directionally the right term. And, and I think a lot of it is really, you know, it's, it's I think one thing to kind of just do a routine implementation and, and write boilerplate code and things like that. And I would say that, uh, you know, in many ways AI coding has already made us much faster at that, right? But I, I think a lot of the core questions of, of understanding very complex systems and, and working in the context of the whole company and thinking about, you know, the product that you're building or the work that you're doing, uh, a- and understanding, okay, what are the problems that we want to solve? How do we want to solve those problems? What is exactly the solution that we want to build? What are all these key decisions and trade-offs that we're going to be making? Basically, I think folks who are able to do that really, really well are just going to be, be able to leverage themselves more and more. And so if anything, I think there's going to be, I think there's going to be way more prod... Way more programmers and way more engineers, you know, a few years from now than there are today. And I think, uh, I think pretty quickly the form factor of what it means to be a programmer obviously is gonna change, and in some sense it already has. But, but, but I, I, I think there's just gonna be so much more for us to build, you know. I think one of the great things... Folks talk about Jevons paradox all the time. I mean, software is truly the, the kind of, uh, the shining example of Jevons paradox, where we have always managed as a society to, to find more and more things that we want to build software for and build more code for it. And, and I really think there's, there's a lot more out there to do.

   8. LRLenny Rachitsky29:23

      For people that don't know Jevons paradox, can you briefly explain it?

   9. SWScott Wu29:26

      Absolutely, yeah. So Jevons paradox just says that as the, the price of something goes down, it can still be the case that, uh, the total spend on it actually goes up. And, and so, you know, you can think about this with money, you can think about this with time or resources, but, but the direct version here is I think as it becomes easier and easier to program and as programming becomes more and more effective, I think we're gonna have a lot more programmers. You know, it's, it's... I, I think in a kind of zero sum view you might say, "Well, we're gonna be, we're gonna be 10 times faster at software engineering and so it means that we're gonna need 10 times fewer software engineers, right?" But I think in practice what really is gonna happen is actually we're gonna build even more than 10 times as much code and because, you know, all, all of the work that we do is so capped obviously on, on our ability to actually build and execute and iterate, we're gonna have so many great ideas out there. We're gonna have so many great products out there. Uh, people are going to build a lot more personalized experiences, for example, and, and, and there's gonna

7. 30:21 – 34:37

   How Cognition’s engineering team works with Devin's

   1. SWScott Wu30:21

      be a lot to do.

   2. LRLenny Rachitsky30:22

      Going back to the way you guys use Devons. So you said that every engineer has kind of this fleet of Devons. How many Devons per engineer do you find most people are working with these days at-

   3. SWScott Wu30:31

      Yeah.

   4. LRLenny Rachitsky30:31

      ... at your company?

   5. SWScott Wu30:32

      Yeah, so it's very asynchronous and so obviously you can kick them up, uh, and start them up and, and shut them down basically a- as you see fit. But, but most folks on the team are definitely, yeah, is... Are, are often working with, with up to five Devons at once, I would say. Uh, a- and it's a nice flow where it's, you know, it's... You think through, all right, what are the five things that we want to get done today? Uh, one, two, three, four, five. You have Devon one do number one, you have Devon two do number two, Devon three... A- a- and the thing about it is a lot of it is... It, it... And for what it's worth, you know, I think it's taken us some time to, to really kind of like... To, to adjust to it and get to the point where it's, uh, it's really intuitive for us. But I, I think it's, uh, yeah, it's, it's definitely a different experience where you're, you're handing off most things asynchronously and, and the goal for each of your tasks is to...... be there for the parts that really need your expertise, where e- either you really, really need to define exactly what it is that you're solving for and what you're building, or maybe some of the more complex parts where you want to, to, to steer Devin towards, you know, particularly what kinds of changes you wanna make. You know, "I want the class to be set up this way and I want, uh, uh, uh, you know, we should go and change all the downstream references to this as well," or whatever. But basically having, having Devin do the bulk of the work asynchronously with you.

   6. LRLenny Rachitsky31:43

      And then how many engineers do you guys have roughly?

   7. SWScott Wu31:46

      Yeah, so our engineering team today is about 15 people.

   8. LRLenny Rachitsky31:48

      15, one, five?

   9. SWScott Wu31:49

      One, five, yeah.

   10. LRLenny Rachitsky31:50

       Holy moly. Okay. And then each one has five-ish Devins?

   11. SWScott Wu31:53

       Yeah.

   12. LRLenny Rachitsky31:54

       Uh, so there's five times the number of Devins as engineers. What I love about this is this is just, like, a glimpse into where the future's going. You guys are so ahead of how companies work with AI engineers, and so seeing how you operate is gonna be es- a sense, essentially how most companies will end up operating.

   13. SWScott Wu32:11

       Yeah. A- a- and, and for what it's worth, you know, it's, we've, we've already seen this shift, I would say ourselves, where it's, uh, in terms of the team obviously, it's, you know, folks don't spend that much of their time just writing out boilerplate or, uh, or, or, or just kind of doing pure, like, implementation of features. And, uh, and, and people get to spends like th- much more of their time focused on really just, yeah, thinking about the core questions of, yeah, how, how do we make Devin better? What is the right interface for Devin? You know, what is the right flow or, or, or, or the right set of features that's, that's really gonna make this, uh, uh, as great of an experience a- as possible? And, and that's, that's how we like things, obviously.

   14. LRLenny Rachitsky32:45

       Have you... When is the point you reach where you're, there's takeoff of this being the bi- you know, like, your Devin starts moving so much further ahead of everyone else? Like, once you have enough Devins doing all these things, they're just, like, whatever, and you're 10 years, 20 years, 30 years, 100 years ahead.

   15. SWScott Wu33:00

       H- honestly, I think as a community, you know, I, I think the, the, the kind of all, all, all of us as engineers around the world, right, I think we're gonna have to think about this and build for this and kind of adapt to these new technologies. But what I would say is, is yeah, I, I, I think more and more, and especially as capabilities get better, but, but certainly, you know, even in steady state today, I think, you know, I think more and more I think things are going to shift towards this kind of asynchronous flow. And, and one of the reasons I would say for that is in the real world you're just capped by real world constraints, right? And, and I think that one way to put it is, is kind of like, and, and don't take these numbers exactly, but, but, you know, it's, it's kind of like the, the first order math of it is of, of course, you know, being able to, to, to write files or to complete this function or complete this line or things like that, you know, it, it helps a ton and so it's a really great experience, right? There's a lot of parts of, of, of building software that obviously are almost not that at all, right? It's, you know, if you have a bug that you're trying to fix and so you, uh, you, you, you spin up the local server, you click around on your own products on the front end and try to reproduce the bug yourself. You know, once you have the error, you, you take a look at Datadog and, and you see what happened and you try to find other errors in the logs. You know, you look at those files and you see what went wrong. You make some edits, maybe you go and, like, rerun the whole process again now that you, you know, just to make sure your change looks right, right? A- and, and that's a lot of, you know, what it means to be a software engineer. Right? And, and, you know, these are processes that take real time. I, I think we're going to shift more and more towards this agentic workflow because, you know, that's in some ways it's kind of like the, the way to really get to the, you know, 200%, 500%, 1000% gains that, that, that, that we'll be getting to with software engineering over the next

8. 34:37 – 42:20

   Live demo

   1. SWScott Wu34:37

      few years.

   2. LRLenny Rachitsky34:38

      Okay, enough talk. Let's show people what the heck this actually looks like.

   3. SWScott Wu34:42

      Yeah. Let's do it.

   4. LRLenny Rachitsky34:42

      Uh, you've got a couple demos prepped that show a few use cases that you found helpful.

   5. SWScott Wu34:47

      Yeah. Yeah.

   6. LRLenny Rachitsky34:47

      So you're gonna pull up your screen and then we'll-

   7. SWScott Wu34:49

      Let's do it. So-

   8. LRLenny Rachitsky34:50

      ... we'll kick it off and then we'll talk as it's happening.

   9. SWScott Wu34:52

      Yeah. And so, so the whole process obviously of working with Devin is working asynchronously. Uh, and so, uh, I, I, I thought it'd be cool for us to actually just watch Devin a little bit in action and then, you know, we can, we can go through some other examples o- of, of work that Devin's done or things that Devin does for us, even on our team, for example. But, but, but, but then we can check back and asynchronously with our, with our Devin after.

   10. LRLenny Rachitsky35:12

       Let's do it.

   11. SWScott Wu35:13

       Um, and so I'll share this real quick. And the, the, the key thing I, I, you know, that I would just emphasize here is a lot of it obviously is really just about, yeah, th- thinking about a- as a software engineer or as engineers ourselves or engineering teams, um, um, engineering teams, PMs and so on, you know, h- what are the, what, what are the things that we would wanna build, that we would want to hand off? And so, uh, you know, we have Devin set up with our own Devin code base, for example. And so I'll, I'll go ahead and, and kick off a Devin for that. And so I'll just say, "Hey, @devin, I'm on with my friend Lenny."

   12. LRLenny Rachitsky35:45

       Hi, Devin.

   13. SWScott Wu35:46

       (laughs) Um, "Can you modify Devin web app to..." Let's, let's, let's feature, let's feature your, um, your, your newsletter as part of the, uh, as part of the Devin, uh, website.

   14. LRLenny Rachitsky35:59

       Let's do it.

   15. SWScott Wu36:00

       (laughs)

   16. LRLenny Rachitsky36:00

       Like, on, on the real Devin website.

   17. SWScott Wu36:03

       Feature Lenny's site.

   18. LRLenny Rachitsky36:04

       You're gonna lose all your features.

   19. SWScott Wu36:06

       And so we're gonna kick this off. As you can see, Devin gets started instantly and, and, and goes ahead and respond. And again, you can work with this asynchronously. You can work with it synchronously as well. For, for this, we'll, we'll just kind of go in a little bit and see exactly what's going on. But as you can see here, Devin's going through files and, uh, and taking a look through a lot of stuff. And so we, we can, we can follow here basically a- as we need to and, and see what makes sense. You can see Devin's already, uh, called out a few particular pieces, right? Where there's the sidebar, uh, which we have implemented on the, on the front end, and, and there's pieces there and we're gonna have, uh, a new component and that component's gonna link to Lenny's website. That all sounds good. Devin's asking us any questions, if there's anything that we have here. Same story here, where it's kind of you can let Devin make its own decisions and, and hand off, or you can, um, uh, you can go ahead and, and be, uh, kinda, kinda give some more thoughts, right? Um, should the button open in a new tab or within the application? I'll say, let's, let's open it in a new tab.

   20. LRLenny Rachitsky37:03

       And you could answer these at any point, like is it waiting for the answer?

   21. SWScott Wu37:05

       You can answer these at any point. You can hand off, hand back off.

   22. LRLenny Rachitsky37:07

       But it's not gonna be like, "Just goddamn it, I just wrote it this way."

   23. NANarrator37:10

       (laughs)

   24. SWScott Wu37:10

       (laughs)

   25. LRLenny Rachitsky37:10

       "Why didn't you tell me earlier?"

   26. SWScott Wu37:11

       That's right, yeah. One of- one of the (laughs) ma- one of- one of the big pieces, you know, with- with Devin is, Devin will always be enthusiastic, you know, will always be ready to put in the hours. (laughs) Uh-

   27. LRLenny Rachitsky37:20

       Thanks for changing scope. Thanks Scott.

   28. SWScott Wu37:21

       (laughs)

   29. NANarrator37:22

       (laughs)

   30. SWScott Wu37:23

       Uh, a- and so- so we'll give Devin a chance to work, and it's going to go through these files and it'll make a pull request for us and we'll see, uh, and go from there. But I- I thought it'd be fun to- to show some other examples of- of Devin in action as well. One of the, uh, the- the examples, actually this morning which I just used Devin for, is I asked Devin to- to help me brush my own facts up, uh, for this podcast. (laughs) Um, and so obviously a huge fan of the podcast and the newsletter, uh, I asked Devin, "Hey, Devin, you're gonna be on the podcast, could you please research everything you can about him and make a nice website quiz for me so that I can, uh, make sure I know my facts," right? And so Devin, this was just this morning, uh, I asked Devin to do this, and I'll kind of just show what Devin did. It looks like, yeah, went to Wikipedia first. Uh, unfortunately it's not a page from Wikipedia which is... L- L- Lenny, we'll work on that, I guess. (laughs)
9. 42:20 – 44:50

   Devin’s codebase integration

   1. LRLenny Rachitsky42:21

      Something that I've- I've learned as I've been talking to more and more AI, uh, building companies and apps is there's a big difference in how large of a code base they could integrate into.

   2. SWScott Wu42:31

      Yeah.

   3. LRLenny Rachitsky42:31

      And that's a big deal for companies that are existing versus startups, people that have large existing code base. What's the- how should people think about h- what kind of code base Devin can plug into?

   4. SWScott Wu42:41

      Yeah, yeah. So we go all the way to- to the biggest code bases possible, right? A- and one way I'd kind of put it is, you know, how the way that- that- that we as engineers would think about a- a large code base is certainly, you know, when- when you're making changes or when you're thinking about a particular task, you're not, um, you're not bringing in every single line of the code base at once, right? You have a high level of traction that- that you're able to think about and look into, right? And- and then you're obviously able to zoom in and get to kind of higher resolution on each of these different things, right? And so Devin works in much the same way, where, you know, the first thing it'll do is it's gonna kind of figure out like the high level architecture of- of what's going on here and what this is built for and so on.But within each of the comp- each of the components, it's obviously also gonna be able to, to zoom in and give some more detail about each of these. And so here's, you know, FP8 to B float 16, uh, and how exactly a lot of that is set up, right? Here's each of the, uh, the different parts of the codebase. And so similarly it's, you know, we, we've built this to, to be scalable.

   5. LRLenny Rachitsky43:38

      It's essentially coming back to the engineer as architect is now it's helping you understand the architecture.

   6. SWScott Wu43:44

      Yeah.

   7. LRLenny Rachitsky43:44

      Um, it's kind of circling back to that.

   8. SWScott Wu43:46

      Yeah. Yeah, exactly. And one of the fun use cases that we've seen actually with folks is, is they'll often actually, uh, use Devin, uh, get Devin's help to, to onboard new engineers on the team, right? And, and you know, when you're new and you're joining, there's obviously a lot of questions that you have about the codebase or about how things are set up. It also sometimes can be a little bit awkward to, you know, to ask your mentor or your manager the questions and if you're worried that they're gonna be really dumb questions, right? And, and so it's nice to just be able to ask Devin and to, to, to go through Devin's wiki and to understand these internal representations, right?

   9. LRLenny Rachitsky44:14

      I think that's really interesting because it comes back to your point that Devin is not just a junior engineer, it's what you call a jagged, uh, jagged engineer.

   10. SWScott Wu44:21

       A jagged intelligence.

   11. LRLenny Rachitsky44:22

       Jagged, jagged intelligence where like it's almost an, uh, like a staff engineer at understanding the codebase. Usually you have to ask an engineer that's been there a long time, "What does this do? Where's this thing? How does this work?" And it feels like Devin's very good at that.

   12. SWScott Wu44:34

       Yeah, yeah, yeah. Obviously and the, the, the retrieval and kind of processing a lot of code and, and a lot of tokens at once is, you know, something that, that language models are really great at, right? And so, so basically being able to get those gains in, in the places that you need them is, is really great. Yeah. Um-

   13. LRLenny Rachitsky44:49

       Sweet. All right.

10. 44:50 – 46:53

    Automation with Linear

    1. LRLenny Rachitsky44:50

       You got a couple more use cases.

    2. SWScott Wu44:50

       Cool. Yeah. Then, then, yeah, one, one last show is just, you know, we, we just rolled this out last week actually, but it's, uh, a full kind of Devin automation set up with Linear, right? And so, you know, if you have tasks that you're doing on the DeepSeek repository, for example, and it's all set up, all you have to do is you just add the Devin label and Devin will come through and it'll give you this, right? And it's gonna give you its thoughts on what the task so- looks like. And you know, you can take a look at each of the particular files that you see or it, it'll point out snippets that it thinks are important. Um, and from there, if, if you feel good about, uh, if you feel good about what was built, uh, or, or, or, or the conclusions that we came to, then you can just start off the Devin session that will go and actually do that work. So...

    3. LRLenny Rachitsky45:30

       That is insane. Like, that sounds like such a simple idea.

    4. SWScott Wu45:33

       Yeah.

    5. LRLenny Rachitsky45:33

       But essentially what you're saying is there are tasks in Linear that are fixes and features and now Devin just goes off and can just do them for you.

    6. SWScott Wu45:44

       Yeah. Yeah. And so, so it's definitely like it's, it's a, it's a hands-on process, you know, you c- you certainly want to be involved when Devin is scoping out the task or giving you its thoughts. And the nice thing too, by the way, is Devin will give you its confidence level and, you know, here's how likely I think, uh, uh, I am to, to, to really understand this piece or that piece or whatever, right? But, uh, uh, but, but, but it helps make things a lot faster, right? And, and, and to your point, you know, it's like a, a, a lot of product managers, for example, obviously love to, to be able to use Devin and Linear to understand things better at the codebase better or, or, or things like that. And you know, Claire Vo from, for example, from, uh, from LaunchDarkly is, uh, is a big Devin user. And, and she loves, uh, basically going and scoping out tasks or asking data questions or asking, "Hey, what's, you know, what's going on?" Or, or, or like, "Is this merged into production yet?" Or, you know, uh, "Is this a feature flag right now?" Or, "What percent of people are, are getting this or that feature?" And it's, it's a, it's a clean way basically to, to, to, to make that intelligence much more accessible.

    7. LRLenny Rachitsky46:40

       I love just like with the integration with Linear, that you can still keep it really simple. You know, you add a little ticket like, "Hey, uh, this link to this homepage would do this." And Devin will be really good at understanding what you mean and then show you, here's what I'm thinking. Is this right?

11. 46:53 – 52:56

    What Devin does best

    1. LRLenny Rachitsky46:53

    2. SWScott Wu46:53

       Yeah. Yeah. Cool. Okay. So, so, so yeah, so Devin did finish working. It seems like there's something going on with the CI and it's debugging that right now, but it went ahead and put up the initial first pass pull request and we can take a look and-

    3. LRLenny Rachitsky47:04

       Let's do it.

    4. SWScott Wu47:04

       ... it's, uh, uh, this is the Devin website obviously in, in this custom deploy. And, uh, we have Lenny's newsletter right here.

    5. LRLenny Rachitsky47:11

       Let's ship this to production. Let's go.

    6. SWScott Wu47:14

       (laughs)

    7. LRLenny Rachitsky47:14

       Or I'll be so confused.

    8. SWScott Wu47:15

       Yeah.

    9. LRLenny Rachitsky47:15

       Uh, that's amazing. Okay. Show it again real quick.

    10. SWScott Wu47:18

        Yeah, yeah.

    11. LRLenny Rachitsky47:18

        So it just added it to the homepage of, of, of Devin's cognition labs.

    12. SWScott Wu47:22

        Yeah, yeah. So, so Devin obviously has access to our Devin codebase.

    13. LRLenny Rachitsky47:25

        Right.

    14. SWScott Wu47:25

        It does a lot in here and so it's super familiar with all the pieces here.

    15. LRLenny Rachitsky47:28

        Beautiful.

    16. SWScott Wu47:28

        And it said, uh, yeah.

    17. LRLenny Rachitsky47:29

        I like how that looks. This is, this is linear.

    18. SWScott Wu47:30

        Look, we've got Devin's search, we've got Devin wiki.

    19. LRLenny Rachitsky47:32

        This is gonna-

    20. SWScott Wu47:32

        And we've got Lenny's newsletter. (laughs)

    21. LRLenny Rachitsky47:33

        ... drive some great growth. I'll link to your site, you link to my site, we'll get some page rank going.

    22. SWScott Wu47:37

        Yeah, yeah, yeah. (laughs)

    23. LRLenny Rachitsky47:39

        Okay. Is that a good exam- oh, there it is. What a beautiful website.

    24. SWScott Wu47:43

        Yeah.

    25. LRLenny Rachitsky47:43

        For my newsletter. Is that just like a good example of the kind of thing Devin's very good at? Like, here's a very specific thing to change on the website. How should people think about what Devin is very good at and maybe where it starts to fall apart?

    26. SWScott Wu47:54

        You know, the, the way that we often describe it is I think Devin is best when it is working on tasks that are well-defined. You know, it's one way to put it is we- you want, you want to be giving Devin tasks, not problems, right? And a lot of these things, you know, like what, what you just saw, which was kind of like a, a quick front end feature request or a bug fix or adding testing and documentation or, or things like that. You know, one of the things that, that makes a loop really nice obviously is a quick way to iterate and test. And so with something like this, obviously super easy for us, for example, to just go pull up the, the preview, uh, and see that the link worked, right? Obviously it would be easy for Devin to do as well. Devin will often go and log into Devin and start a Devin session and make sure it's, you know, it, it, when, when it's working on our own codebase, which is kind of hilarious. (laughs) But yeah, you, you generally want something that is kind of like eas- easy to verify and easy to test is, is, is the main thing. And you can work on bigger projects or bigger asks as well, obviously, but in that case you should certainly expect to, to need to steer Devin more to make sure you're really, you know, to, to make sure it's, it's going the right direction.

    27. LRLenny Rachitsky48:55

        It's interesting 'cause that's very similar to the way people talk about synthetic data and reinforcement learning, creating data that's very easy. There's like a very definitive answer, yes and no.

    28. SWScott Wu49:04

        Yeah.

    29. LRLenny Rachitsky49:04

        And it's very clear. Hmm.

    30. SWScott Wu49:06

        Yeah. Yeah.
12. 52:56 – 57:13

    The future of AI in software engineering

    1. SWScott Wu52:56

       naturally made sense over time.

    2. LRLenny Rachitsky52:57

       Let's talk about the landscape then of just other companies in this space, which is something a lot of people are always thinking about. There's all these different approaches, you guys are going full on AI engineer, there's obviously IDE companies, there's also just like models being built that are really good at engineering. Uh, everyone's kind of starting to build agents now, you guys are ahead on this in a lot of ways. Like OpenAI just recently said they're gonna build a software engineering agent. Anthropic's got something there, you know, Cursor and Windsurf have their own little agents, and Replit. Thoughts on just kind of where you guys fit in in the landscape and then where you... how you think you win long term? How do you think about that?

    3. SWScott Wu53:31

       Yeah. Yeah. And for what it's worth, you know, I, I think all of these are incredible teams. I think, uh, you know, r- really smart and r- and really forward-thinking folks who are, who are building a lot of great products out there. A- and it's, uh, I, I, I think there's, there's, there's a lot to do, honestly, you know, over the next few years with, with the advent of AGI or, or whatever you want to call it. You know, I think, uh, one, one of the quotes that I love is, is in 2017 if you asked if we had AGI, the answer is no. And in 2025, if you ask if we have AGI, the answer is, "Well, you have to define AGI," and, you know, it depends on your subject. Yeah. Right. A- and, and I think it does kind of get to the point of, of, I mean, there, there is a lot of really amazing stuff happening. You know, I, I think that it, it's easy to underrate, I would say, just, just how, how big of a shift it is that we're seeing, right? Where I think there are a lot of great products out there, for example, over the last 10 years, 20 years, 30 years that have made each of these kind of like, uh, kind of these different niches o- of the, the, the life cycle of building a product a little bit easier, for example, right? There's, there's great products out there for instant response, there's great products out there for logging, there's great products out there for billing. There's, you know, a- all of these different tools, right? A- and, and the obvious thing is, you know, what we're seeing with AI is all of these spaces are, are going to be moving multiple times faster, you know, and th- this... it's going to be like an order of magnitude shift, if anything, right? And so I, I think from our perspective, you know, we've o- obviously had a, a very specific lens that, that we've bet on this whole time, and that is, you know, autonomous coding agents, and there's, there's a lot of problems to solve there, to be honest, right? Like the, the... there's still a ton to do on the core capabilities, certainly, and, you know, we see cases all the time where it's like, "Wow, why did Devin make that decision? That seems..." you know? (laughs) No human engineer would have ever done that. You know, there's, there's all sorts of spots where, you know, i- with the product interface, there's obviously a lot to think about. And I think it's, it's by the way, not just like a single thing that, that, that we're working towards, but something that will change with every addition of capabilities. Like I kind of think of it as like there's, there's 20 generations of agent product, you know, agent coding experiences to come. You know, I, I think the, I think the one that we'll get to over the course of several years is probably something where you don't even look at the code at all, right? And you're actually just looking at your own product and you're just able to look and, and specify and say, "Hey, you know, I...... this button should be a little bit rounder, let's do that. And, and by the way, let's add a new tab here. A- and maybe we should save this information. Let's, let's start up a database table and let's index it on X, Y, and Z columns. And you're just, you know, basically working with your products in real time, a- and having your agent build out those things for you. Obviously, there's gonna be a lot of generations, (laughs) you know, in between the, here and there. But, but I think the product experience itself is gonna change every single time. And then obviously, you know, there's, there's all of the practicality of just getting it out there in the world. And so, you know, th- fo- folks obviously need to, need to learn how to use the ne- the new technology, there's a lot to do to, to deploy and to all of the messiness of real world software. You know, there's a lot of COBOL out there still, there's a lot of Fortran out there (laughs) still. There's, you know, there's lots of kind of abstractions and, and details that folks have done. And so I think from our perspective, you know, we have f- w- we have been, been since the beginning have been laser focused on agentic coding. And that is the one thing that we've really believed in, it's the one thing that we've designed for. And, you know, that goes all the way to, uh, even the revenue model with ACUs and, and, and having the usage based setup. It goes into obviously all the product experiences of thinking how, okay, like where do you want to talk to Devin? You know, you wanna be able to talk to Devin in Slack. You wanna be able to, to spin this up from your issue tracker. You want to be able to, uh, all of these things. A- and then of course the capabilities. And so I, I, I think it's, uh, I, I, I don't think there's any one easy answer. I, I, I think it's obviously a combination of things, but, but this is really the, this is, this has been the space that we've lived in and, and spent all of our time in for, for the last year and a half. And it's, it's gonna be that way for the next five

13. 57:13 – 1:01:57

    Moats and stickiness in AI

    1. SWScott Wu57:13

       or 10 years too. So.

    2. LRLenny Rachitsky57:14

       Along these lines, a big question everyone always has in AI is moats and defensibility. It's a question I've been asking every founder that comes on. How do you just think about how to build a moat in this space when it's so much easier to build and these models are, you know, so much is built on these models that are themselves advancing so quickly?

    3. SWScott Wu57:30

       I, I'd give one slight, uh, kind of tweak on that, which is, I, I think it's often less about moats and more about stickiness. And what I mean by that is, you know, moats are in some sense, typically what folks mean by a moat is something that means that a competitor couldn't even enter, you know, the market. A- and, and I agree that at a high level, you know, a lot of different folks have different layers of AI spectrum, you know, uh, the, the, the foundation labs or the application layer or so on. You know, I, I don't think there's any kind of like hard barrier that would prevent others from entering. I think what does exist is stickiness, which I would kind of define as, you know, once you have a product experience that you really like, are you excited to keep using that experience or is there a kind of like, you know, i- is there an effect where it is, it is just as easy from now on to just switch onto a new one and n- learn a new one and so on, right? And I think from that perspective, I think there's a, there's a few things that are really great about coding agents in particular. You know, one I would say is there is a lot of just inherent kind of stickiness and learning and buildup over time, which is that as you use Devin and as your wh- whole team uses Devin, it's the same thing with, with an engineer, right? If, if you're joining on day one versus, you know, so it's, you've been at the company for five years, you know, (laughs) you wrote half the code yourself. Y- you've touched every file, you've built every single piece, right? You know all the engineers. A- and so similarly, it's like Devin will really learn and build its representation of, of your code base and of your stack and of your process over time. Um, and we'll be able to do a lot more with that. And then the other piece of it, which I think is really, uh, is, is really, uh, exciting I- I'd say, is there really is a lot to do, uh, of what I would kind of call like a multiplayer aspect of, of code, which if you think about it is, is how a lot of things get done in the real world certainly, right? And so, um, you know, it's, it's one thing to have, you know, your own experience which you use yourself as just an engineer. But, for example, ourselves, like we see this all the time where, uh, some engineers are working with Devin and teaching Devin things, and as I mentioned, like folks will have Devin onboard their new engineers and, and kind of convey that knowledge to them, right? Or similarly, it's like yo- you know, I'll start a session with Devin in Slack and I'll say, "Hey, you know, it'd be cool if, if we could do this thing." And some eng- other engineer will chime in and say, "Oh, by the way, like, the reason we did it initially was X and Y. And so Devin just makes sure when you do this change that you, you know, still support that workflow." And Devin will say, "Okay, sounds great." Right? Or Devin will make a PR, you know, I'll be working with Devin, we'll make a pull request in GitHub and somebody else will be reviewing that PR or give some comments and, and Devin will work on that too, right? You know, you'll, you'll be in linear. A- and so, so all, all these kind of spaces, it really does just kind of set up for an experience where basically where, where, where Devin can just grow in the value that it can provide for your whole org over time. And so I think from that perspective, like if anything, you know, it's we, we want there to be a, a lot of innovation and a lot of new products and, and so on. You know, I, I, I don't think that the goal is to try to lock other people out of (laughs) of building, you know? There's a lot of stuff to build and I think there's gonna be a lot of different experiences. I think from our perspective, what we think about is more like how can we make Devin more and more and more useful as you're using it more?

    4. LRLenny Rachitsky1:00:27

       That's very similar. Uh, we had Michael from, uh, Cursor, the CEO of Cursor on the podcast, and he had a similar point of just he thinks moats are just kind of like consumer, like Google is the way. He thinks it's like Google, where people can easily switch, you just have to stay the best and that's the answer.

    5. SWScott Wu1:00:43

       Yeah. Yeah.

    6. LRLenny Rachitsky1:00:44

       Uh, and it feels like you're adding to that of just like but also if you can create some stickiness where it is, uh, very hard to leave because it's so good at what it's doing and it's built knowledge and, and integrated into your workflows, uh, that it builds on that. And that stickiness.

    7. SWScott Wu1:00:57

       Yeah. And, and I think one of the things that's, that's, that's, you know, that's nice about our space too is software engineering for better or for worse is, is, is has a very clear tide of value, you know? (laughs) And, and what it means is, I, I guess one way to put it is, is there is always kind of like a clear next level, at least for the next while, you know? I think there, there could be some point where you're just like, all right, just build the entirety of YouTube for me, you know? And Devin just does the whole... It's like there's probably been like 100 million hours of human engineering time built building YouTube, building the algorithm, building all the infrastructure, all, all of the, the everything, ev- every little detail. And, and like, you know, there's maybe there's some time where Devin just does that out of the box, you know? (laughs) That, that's obviously gonna be a long time from now. I, I think on the interim on, on every level in between, obviously it's...You know, it- it- it makes a difference, the- the- the quality of- of software engineering, and I think one of the cool things with developers obviously is developers are really willing to- to- to- to learn new experiences and to put in effort if it means that they're able to have a higher and higher quality experience, so...

14. 1:01:57 – 1:04:14

    The tech that enables Devin

    1. SWScott Wu1:01:57

Episode duration: 1:32:31