Successfully coding with AI in large enterprises: Centralized rules, workflows for tech debt, & more

1. 0:00 – 2:44

   Introduction to Zach Davis

   1. CVClaire Vo0:00

      vibe coding is not an acceptable enterprise development strategy. I love it. I can do a hundred commits a week by myself, on my side project, on my startup, but when you're working on a code base in a platform like LaunchDarkly that powers trillions and trillions of experiences every day, you can't take the same strategies and tactics that a vibe coder could take.

   2. ZDZach Davis0:22

      One of the things that I realized is what's good for humans is also good for LLMs, and so I really started with: How do we make sure that the repo is well set up for humans to know how to work in it? So we have front-end organization, we have accessibility, we have a JS style guide. So all... It's this very detailed documentation that we've put into the repo itself, rather than have it in other places. And this way, LLMs can access it, humans can access it, et cetera.

   3. CVClaire Vo0:47

      I think all the engineers out there are, like, crossing their fingers and hoping that there's one rules protocol to rule them all that shows up. And I think what you've shown is you can just create that yourself, and then that makes it much more scalable. [upbeat music] Welcome back to How I AI. I'm Claire Vo, product leader and AI obsessive, here on a mission to help you build better with these new tools. Today, we have a great episode for anybody trying to deploy AI agents in a real engineering team with a real codebase, not just vibe coding. We have Zach Davis, director of engineering at LaunchDarkly, who's gonna show us how he sets up centralized rules and docs for all his AI agents, uses AI to burn down tech debt, and keep his hiring bar high. Let's get to it. This episode is brought to you by WorkOS. AI has already changed how we work. Tools are helping teams write better code, analyze customer data, and even handle support tickets automatically. But there's a catch: these tools only work well when they have deep access to company systems. Your copilot needs to see your entire codebase. Your chatbot needs to search across internal docs. And for enterprise buyers, that raises serious security concerns. That's why these apps face intense IT scrutiny from day one. To pass, they need secure authentication, access controls, audit logs, the whole suite of enterprise features. Building all that from scratch, it's a massive lift. That's where WorkOS comes in. WorkOS gives you drop-in APIs for enterprise features, so your app can become enterprise-ready and scale upmarket faster. Think of it like Stripe for enterprise features. OpenAI, Perplexity, and Cursor are already using WorkOS to move faster and meet enterprise demands. Join them and hundreds of other industry leaders at workos.com. Start building today.

2. 2:44 – 4:00

   Overview of AI tools used at LaunchDarkly

   1. CVClaire Vo2:45

      Zach, I'm so excited to have you here because I feel like I maybe turned you into an AI fiend at this point. [chuckles] Before the show, we were talking about how many tools you're now using. So before we dive in, can you just tell us a quick list, maybe not-so-quick list, of all the AI tools the technology team at LaunchDarkly are now using?

   2. ZDZach Davis3:09

      Yeah, absolutely. Let's see. So on the design side, uh, w- we're, again, we're exploring a bunch of things, so there's gonna be a bunch of tools. So, uh, Lovable, v0, uh, Figma Make we're using. Uh, on the product side, obviously, we're using ChatPRD. Um, and then on the engineering side, for code, heavy Cursor users, heavy Devin users. Um, we're also using now, like, Cursor's background agent. I personally use Windsurf 'cause I like Windsurf, but most of the rest of the org does not. Uh, we are using... We're trialing Augment. We are looking into Claude- Claude Code. We're doing all the things. We're also looking into PR review, so we use Copilot for code review, and we use, uh, Cursor for code review as well.

   3. CVClaire Vo3:51

      Okay, and I feel like 18 months ago, we were using maybe a little GitHub Copilot, but [chuckles] not, not much-

   4. ZDZach Davis3:59

      Yeah.

3. 4:00 – 5:44

   The importance of having someone responsible for driving AI adoption

   1. CVClaire Vo4:00

      ... yeah, not much more. And one of the things that I really liked that you and I did together, and you were, you were a champ in coming along this journey, is we really decided that in order for AI to be really effectively adopted by a team like LaunchDarkly's engineering organization, which is over 100 people, we really needed to put some concerted effort behind it and put a person in charge. And lucky you, you drew the, uh, either short or long straw, however, [chuckles] however that works. You know, what do you think about teams approaching this kind of engineering-wide transformation, and what kind of organizational and cultural things you need to do to make it possible?

   2. ZDZach Davis4:37

      I do think having a person who's kind of, I don't know if "in charge" is the right word, but whose responsibility it is to drive that kind of change. And I think that having someone who's close to the code helps a lot because you don't really know what's working and what's not working unless you're in the code, at least on some basis. And so that can be a manager, that can be a director or someone, but it has to be someone who's actually trying these things, I think matters a lot. And, uh, yeah, you-- I think you were looking for someone to take that role, and I was, I think, skeptical, right, of how well... Things were working really well for you over w- on your side job on ChatPRD, but when I tried to do the same things in our codebase, I was struggling. And so I really came at it from a standpoint of, "I wanna understand what works and what doesn't," and, and either be able to push back on you and say, "Hey, you know, [chuckles] like, it's great over there, but on, on this, you know, larger codebase, it's not working," [clears throat] or be able to actually drive change. And now I'm at the, like, "I'm, I'm on board. Let's, let's drive

4. 5:44 – 6:42

   Why vibe coding isn’t acceptable for enterprise development

   1. ZDZach Davis5:44

      change." Uh, I think that matters a lot.

   2. CVClaire Vo5:46

      Yeah, and one of the things that I think is really important for our listeners, especially ones that are at growth stage or larger companies, is vibe coding is not an acceptable enterprise development strategy. Like, I love it, right? I can do 100 commits a week by myself on my side project, on my startup, and, you know, I, I can recover from quality issues. You know, the maintainability of the code, the issue right now, that's not my big business issue. But when you're, you know, working on a codebase in a platform like LaunchDarkly's, that powers-... trillions and trillions of experiences every day, you can't take the same strategies and tactics that a vibe coder could take and bring those into not just an individual developer's workflow, but an entire team's workflow. And so what have you kind of discovered as you try to figure out how to make these tools work for a larger team?

5. 6:42 – 7:55

   Making engineers successful with AI on their first attempt

   1. ZDZach Davis6:42

      With smaller teams, you have more flexibility in terms of how you approach these things.

   2. CVClaire Vo6:47

      Mm-hmm.

   3. ZDZach Davis6:47

      With larger teams, you have more enablement and, and stuff like that. We're kind of in the messy middle, and it's-- I, I found it more difficult to, uh, sort of like operationalize that and make everyone successful. And, and what I found was everyone was on their own journey, uh, to try to be successful with AI, and that just doesn't scale very well, right? So, um, yeah, you really need to come up with a system in order to, to make everyone more successful, right? What I want is when those skeptical engineers jump in, and they try Cursor, they try whatever for the first time, or they try it for the first time in a while, I want them to be successful so that they get that aha moment. And if you just leave them on their own, then, then you're not gonna get there.

   4. CVClaire Vo7:32

      Totally. And we had... I think you and I had mutually had the experience of developers having their first experience actually be negative. Whether it was with Cursor or with Devin, it was like: See, I, I knew this was never gonna work, [chuckles] and here's my first-pass proof that it didn't work. And so you really did a lot of... What I appreciate about you is you did a lot of technical work to make sure people were successful. We'll definitely talk a

6. 7:55 – 10:19

   Creating centralized documentation for both humans and AI agents

   1. CVClaire Vo7:55

      little bit more about some of the kind of culture and operations piece, but I actually wanna get- dive into what you did in the codebase to make it easier to work with Cursor and Devin. So can you walk us through some of those things that you did?

   2. ZDZach Davis8:07

      Yeah, absolutely. So in my IDE here, uh, you can see one of the things that I realized is what's good for humans is also good for LLMs, and so I really started with: How do we make sure that the repo is well set up for humans to know how to work in it? And so we have this docs directory, and I pulled a bunch of stuff from Confluence, from Google Docs, from other places in the repo, and I, I put it all in here, right? So we have front-end organization, we have accessibility, we have a JS style guide. So all... It's this very detailed documentation that we've put into the repo itself, rather than have it in other places. And this way, LLMs can access it, humans can access it, et cetera. In addition, we had these-- we had Cursor Rules before. We had a Claude MD file, and I wanted to consolidate that. And so, uh, instead of a .cursorrules, I have this .agentsrules, and the idea is to kind of centralize all of this knowledge in one place. And so you can see here, uh, I have something like TypeScript Essentials, which has a really, uh, kind of like quick, like, the, the quick hits of what's really important, and then it, uh, it also links off to, like, the comprehensive docs and says, "Hey, go... If you wanna find out more, you know, go look at the JS style guide." And so then our Cursor Rules actually just point to that, right? So with our Cursor Rules say, "Hey, if you want TypeScript guidelines, go find this file in .agents." Um, and then, uh, I talked about Augment earlier. We were trialing Augment. I set this up yesterday, and I had- I asked the Augment, uh, agent to just create this. I pointed it at the Cursor Rules, and I, uh, pointed it at our agents rules, and I said: Can you just create this file? And so it, uh, it did the same thing. And this way, we don't have to duplicate everything across multiple tools or tool files, and it's much easier to get stuff working well by default. And the whole idea with this is, again, I want people to be successful, uh, out of the gate a- and having this kind of centralized place. Having all this documentation in the repo just makes it way easier for, for tools to be successful by default.

7. 10:19 – 12:33

   Using feature flagging rules to improve AI outputs

   1. CVClaire Vo10:19

      Yeah, one thing that I hear a lot is people are really frustrated with the tool-specific rules. They're like: Why do I have a Claude.md? Why do I have a Cursor Rules? Why do I have these GitHub rules? Especially if you're experimenting with the number of tools that you're trying, each tool has isolated their rule set in an individual file structure. And I think all the engineers out there are, like, crossing their fingers and hoping that there's, like, one rules protocol to rule them all that shows up. And I think what you've shown is you can just create that yourself. Create a directory in your repo, put a consolidated set of rules, make your sub-rules for each of those tools, point to those rules. So say, when you're looking... You know, when you're working on front end, reference these rules, and then that makes it much more scalable. The other thing that I wanna call out for you is, I think what you said at the beginning, you're using, I don't know, a dozen tools, probably like three to five IDEs across the engineering organization. Probably within any individual engineer, you're testing, like, I want Cursor today and Devin tomorrow, and if you don't have your rules set up for all those tools, then you're starting from scratch every time. And so I really like this idea of, of, of rule setup and then consolidation. I'm curious, do you feel like the rules have improved the quality of the outputs significantly?

   2. ZDZach Davis11:40

      Yeah. Yeah, absolutely. So here I'm looking at, at our feature flagging rules, and it's interesting because we are a feature flagging, uh... We, we have a lot of feature flagging code in the codebase, and one thing we noticed was that some of the models, some of the tools, would get confused about whether we were asking it to create feature flags on the, like, LaunchDarkly product, or whether we were actually trying to get it to do stuff in the code. And so there's, there's a bunch of stuff that I did to just be really specific about how to make- how to be successful when creating feature flags. I want you to return a link, that kind of stuff, and it really has... It has made a difference. Yesterday, literally, uh, one of our product managers was doing a task with Devin, uh, and was able to tell it to put a flag- put the feature behind a flag, and, uh, Devin went and used the MCP and, and hit the flag, and, and everything worked.

8. 12:33 – 14:28

   Advice for getting started with rules

   1. CVClaire Vo12:33

      ... So I have another question about rules, because LaunchDarkly's giant monorepo, ten, ten years old [chuckles] or something like that, it's, it's got a lot of code in it, front-end, back-end, tests, all this stuff. What do you think, if you had to give some advice to peer engineering leaders who are approaching the same problem, what are the must-have rules from your point of view in the code? You know, I saw, like, a lot of front-end stuff, but what, you know, what are the quick hits of what you think should belong in a kind of Cursor rules or a rule set?

   2. ZDZach Davis13:02

      I would say the best tip is ask the agents, right? To, to get you started. And so Devin actually has a great wiki, um, that ha- for each repo that Devin works on, it creates a wiki, and it has a ton of really good information in it. And so I actually started with Devin, and I said, "Hey, can you, like, this is what I'm trying to do. Can you create basically the human-readable docs for this?" And so Devin did a pass and, and created a, a bunch of docs, suggested some structure. We went back and forth, and I kind of tweaked things, and then I took that output, and I went through it with kind of like a fine-tooth comb because I think it matters, right? It, it matters to get those details right. And then once I had the human-readable docs, I went to Cursor, and I said, "Hey, [chuckles] like, can you, can you take these docs, and can you take your existing Cursor rules, and can you turn those into a dot agents file?" And so it was a combination where you, you can kind of lean on the agents a little bit to help you get unstuck and get started, and then also use your knowledge of, of what's important in the repo. And, um, the other thing is that I was looking at, where are people getting stuck? I knew that people on the front end would struggle with getting, like, testing, basically, writing unit tests. Um, it would write Jest tests, and we use Vitest and stuff like that, and so putting in specific rules to make sure where people get stuck, uh, we have rules to, to help the agents be more

9. 14:28 – 24:33

   Demo: Setting up Devin’s environment in a large codebase

   1. ZDZach Davis14:28

      successful.

   2. CVClaire Vo14:28

      Would you mind pulling up Devin and actually-

   3. ZDZach Davis14:32

      Yeah

   4. CVClaire Vo14:32

      ... giving an example of generating a rule? And I have an idea for you-

   5. ZDZach Davis14:36

      Sure

   6. CVClaire Vo14:36

      ... which is, like, rules around generating data visualizations since we've done so much, and just, you know, see what it comes up with.

   7. ZDZach Davis14:42

      Yeah. Here, here's an example of, of both asking Devin Wiki a question and then also using Devin to create a, to create a rule. So we can say, uh, to the Devin Wiki, we can say, um, "What are the libraries used for charting on the front end?"

   8. CVClaire Vo14:59

      I'm curious, while this is loading, how long did it take you to set up Devin's environment? It's something that, you know, everything's easier with a little vibe coding Greenfield app, except for-

   9. ZDZach Davis15:09

      Yep

   10. CVClaire Vo15:09

       ... setting up Devin's [chuckles] environment. It's just as hard.

   11. ZDZach Davis15:13

       Yeah.

   12. CVClaire Vo15:13

       I'm curious what your experience has been configuring Devin to work in a large repo.

   13. ZDZach Davis15:18

       Yeah. I would say to get up and running with Devin, uh, I, I got started pretty quickly, and we have kind of- we have a separate flow for front end and, and back end. We have a concept of sort of like front-end-only mode, which proxies against a, another back end. And so I was able to get Devin's machine up and running pretty quickly in just front-end-only mode, and then I was able to take on front-end tasks using Devin. One of our other engineering managers actually came in and, and saved the day on the back end to get the full, like, running our whole end-to-end, uh, up and running with Devin, and that took him, I think, uh, a, a little bit more time than it took me. But the nice thing is you can do that sort of incremental, you know, y- you do what works. You don't have to have Devin running your full app locally in order to, to get value out of it, and so it's, it's just about kind of like doing it piece by piece. And again, if it's hard to get Devin up and running, it's probably hard for your human developers to get up and running, so there's always, uh, incentive to, to make those things better.

   14. CVClaire Vo16:19

       Yeah, I will give, just 'cause you, you said it, you said, you know, "Devin's environment doesn't have to be running for you to get value," my number one Devin prompting tic- trick is: don't run this locally, just give me the code, and I'll test it for [chuckles] for you.

   15. ZDZach Davis16:32

       Yeah.

   16. CVClaire Vo16:32

       So-

   17. ZDZach Davis16:33

       Yeah

   18. CVClaire Vo16:33

       ... sometimes I by, I bypass that process entirely. Okay, so you asked this question of Devin Wiki, "What are libraries used for charting on the front end?" And it gave, um, answer, Recharts, um, some other things. And so what would you... One, is this information pretty accurate? And, and two, what would you do with it?

   19. ZDZach Davis16:48

       Yes, it is accurate. We are using multiple libraries, and so that was one of the things I was curious about, is, is we've brought in several libraries and, and we're kind of trying to figure out how to consolidate, and so it picked out that we're using Recharts, we're using VizX. Um, it lists ECharts as a secondary library. I don't know if that's strictly true, but generally, this seems very correct. And so I, I like to use Devin Wiki to just sort of ask basic questions about the repo, make sure I, I understand what we're doing. But if I actually wanted to create a rule... So you can't take action from, from Devin Wiki. So I, what I want to do is I want to create a new document, a human-readable, you know, human-centered document in, in my, our doc/frontend about how to use charting libraries, and then I want to also add a rule to .agent/rules. Um, so I'm just gonna give this to Devin, and I am gonna see how it goes. Uh, so Devin's going to spin up a new session here.

   20. CVClaire Vo17:46

       And one thing I wanna call out for folks listening on the prompt is you specifically said you want it to create a markdown document. Markdown is every engineering agent's favorite, um, file type, so that's a good way just to give a little bit of structure to your code. It also tends to pretty print and, and be human-readable and easy, easy to view in, in GitHub and all that stuff.

   21. ZDZach Davis18:11

       Yeah.

   22. CVClaire Vo18:11

       And so what you're doing here is just asking Devin to make those docs for you, and this is one of my favorite use cases of Devin. I think you know this. Um, it's my favorite Devin hack, which is I have a GitHub action on every PR that writes docs for the PR and adds to a change log programmatically with Devin. I found that it's a very good-... technical writer. You know, sometimes the code is okay, but the technical [chuckles] writing is, is very clear and very good.

   23. ZDZach Davis18:40

       Yeah, I think that's exactly right. The, the Devin wiki is very good. It knows a lot about your codebase. It has this very explicit way of learning and, and understanding your codebase, and so it is very good about kind of describing that back, and, and as you said, doing it in a, in a solid technical writing way.

   24. CVClaire Vo18:58

       Yeah. And then one of the other things that I wanna call out for people that are maybe listening and not watching is we are chit-chatting because we're waiting for Devin to spin up a virtual machine. So for those that don't understand kinda how Devin works, it actually spins up a virtual environment that reflects a development environment. It's gonna open it up, it's gonna read your codebase, it's gonna do all this stuff. And so, you know, it takes a minute to actually boot into an environment, a little different than running something like Cursor locally.

   25. ZDZach Davis19:26

       Yeah, that's exactly right. Where, where these other tools are s- just using whatever you have locally, Devin is running its own machine, which has a lot of upside. Um, it can run a browser and, and see a browser. It can do a lot of things that don't come out of the box with these other tools, uh, with the downside that it takes a little time. Depending on, on your repo, it, it takes a little bit of time to actually set that machine up and, and get it running.

   26. CVClaire Vo19:50

       But like you said before, if your machine is slow to cold boot for Devin, it's probably [chuckles] slow to set up locally for an engineer. So again, align incentives on getting your repo to work well for both your agent coworkers, as well as your human, your human colleagues.

   27. ZDZach Davis20:09

       Yep. That is, that is my favorite thing, is all the things that have been hard for humans forever, and we have just kind of swallowed it and said, "Well, that's the way this works," become even more important, uh, today with, with these LLM tools to, to solve and, and improve.

   28. CVClaire Vo20:25

       Well, I think it becomes more important to solve and improve, and then I also think it becomes easier to solve and improve them.

   29. ZDZach Davis20:30

       Yeah.

   30. CVClaire Vo20:30

       If I said, you know, two or three or four years ago, "Zach, go document everything in the repo, high quality, human-readable docs. You just go do it by yourself," it would take forever to generate high-quality docs that, you know, really reference our code and understand the nits and details. And I think the fact that even you can spin up docs so quickly is so transformational to how you can... And I know we'll see this in a little bit, like burn down tech debt, make your engineers happier. And so I do think, you know, a lot of engineers have this skepticism that adoption of AI tools is really about moving faster, shoving more junk in the code, like just getting feature bloat. And I actually do think for mature engineering organizations, it is also an opportunity, if you approach it correctly, to take care of some of the things that you've have just hated forever i- in either how you run your software, how your team operates, or the code itself. And so that's one of the advantages I think people underestimate in larger organizations because they blur the line in their mind of AI-assisted engineering with vibe coding, which is not what we're talking about right now.
10. 24:33 – 27:55

    Devin’s plan overview

    1. ZDZach Davis24:34

       So Devin has a plan now. One of the things I like about Devin is it, it gives this confidence now, like how, how confident is it in the task at hand? Uh, which-... is nice because sometimes it's not confident, [chuckles] and, uh, you- it's, it's better not to proceed. This, this is something that, uh, as we mentioned, Devin should be really good at, and so I feel good about its ability to execute this, but it will give you sort of an overview. If I thought- if I read through this, and I didn't like, like, what it was doing, it's gonna run prettier on the markdown files, which actually I think is a good idea. But if I didn't think that was a good idea, I could update its plan while it's deciding what to do next.

    2. CVClaire Vo25:10

       Yeah. The, the other thing that I enjoy about Devin is nine times out of ten, its confident- confidence gets higher as it goes. So it always starts, like, medium confidence, but I have to investigate, and then it's, like, high confidence, I know what to do. But occasionally, it fails me deeply-

    3. ZDZach Davis25:28

       Mm

    4. CVClaire Vo25:28

       ... and I have bullied it so much [chuckles] that it, like, starts to progressively lose, lose confidence, and then it's, like, low confidence. I haven't been successful so far. So I find the confidence assessment pretty, pretty accurate.

    5. ZDZach Davis25:41

       Yeah. Okay, so now it is creating... It's created multiple markdown files. So it's created a charting-libraries.md file. Um, and we can actually, if we want, we can jump over. So there's a shell, there's also the code, so I can actually go look at what it's creating while it's creating it. Um, so charting libraries guideline, it's creating that in our .agent/rules/frontend. It looks like I think it also created one in docs. So this is the human-readable version, um, which I'm not gonna go through in detail, but looks... It has examples. It, uh, you know, has the different libraries. I like all of that. And then in the agents rules, it's sort of a consolidated, you know, must, must use this when... I like seeing that. I would go through here and really make sure it's accurate and what we want. And then it's a little long, I think. You know, for, for me, I want to keep the, the agents rules pretty concise, so you're not reading the context, and, and just so it's not too much. And I would also wanna make sure that it links out to the full documentation, is another trick that I like to do, so that a tool can decide to pull in that additional context if it wants.

    6. CVClaire Vo26:52

       Well, one little trick that I learned from another How I AI guest, is that if you notice, Cursor reads long files in chunks of two hundred lines, and so his goal was to keep these files under two hundred lines, so that it's not chunking the content. And so I saw yours is just, like, a little bit over two hundred. So one of the things you might add to your rules for rules is try to keep your rules files under two hundred lines, for example.

    7. ZDZach Davis27:21

       Yeah.

    8. CVClaire Vo27:21

       Now, again-

    9. ZDZach Davis27:22

       That's-

    10. CVClaire Vo27:22

        ... I don't, I don't know if that's actually [chuckles] helpful or true, but it is a tip somebody gave me, so I'm passing it along with no personal context. [chuckles]

    11. ZDZach Davis27:30

        Yeah. No, I mean, that, that, that's actually, again, that's a good tip for humans, just like it's a good tip for LLMs.

    12. CVClaire Vo27:36

        Yeah.

    13. ZDZach Davis27:36

        And you, you said something that I think is really interesting, which is I actually- I have a README... I have a human README about the rules so that people understand how to create new rules, but I should actually probably have something geared towards LLMs-

    14. CVClaire Vo27:50

        Mm-hmm

    15. ZDZach Davis27:50

        ... so that when LLMs are adding new rules, they're doing a better job of it.

    16. CVClaire Vo27:54

        Yep.

11. 27:55 – 36:40

    Demo: Creating a prioritized tech debt reduction plan

    1. CVClaire Vo27:55

       Okay, so I see this. It looks great. So it's created a human-readable docs, um, in your docs folder, a, um, rules for your LLMs. You're gonna review this, you're gonna do a PR and merge those docs into, to the repo, maybe take a look at them, edit them. And you've used, you know, Devin Wiki, Devin Agent, um, and then it's spun up this, this codebase to write, write those docs. And so I think this is a really great flow. I think people are gonna learn a lot from this. You know, one of the things you said earlier was that tech debt is your favorite use case for these AI tools. I love to hear it, because this is how I try to pitch senior engineers and senior engineering leaders like you to really adopt these tools when they're really skeptical. Can you walk us through how you actually approach burning down tech debt using these tools, where it's made it easier, maybe?

    2. ZDZach Davis28:51

       Yeah, absolutely. So here we are in Cursor, and, uh, I'm gonna show you that same agents directory. There's... So I showed you agents rules before. We also have agents/migrations, and so this has a couple files in it. Um, it has a CSS module conversion file, which I created to, to help us, um, convert CSS files to, to modules, CSS modules. And then it also has... I just added this one the other day, which is the one that I would like to show. And so, uh, what it is, is basically it's a combination of instructions for agents and a, a checklist, basically, like a task list of, of what to burn down. And so the problem that I was running into is that our front-end unit tests, when you run yarn test, it just, there's so much noise in the console that it's really distracting. There's some actual legitimate problems in there that are just kind of being warned about and ignored, and I wanted to pay that down. But it's, it's one of those things that is annoying, but it's not quite annoying enough for someone to own it. And also, it's such a big problem, it's really hard for one person to just kind of, like, take that and pay that down. And so-

    3. CVClaire Vo30:06

       Well, and, and I'll say, imagine as an engineer, you go to your product counterpart, and you're like: "Hey, I just wanna spend, like, a week or two just making our test logs just a little less noisy, so my life's just a tiny bit e-" Like, it's such a hard pitch to make for work like this. It's super important, and, like, the pitch can work on the [chuckles] right leader, but again, like, this is the kind of thing that's hard to justify in a fast-moving org.

    4. ZDZach Davis30:33

       Yeah. So I'm actually- I think what I'm gonna do is I'm gonna ask, uh... I, I will talk, I will talk through kind of how this works.

    5. CVClaire Vo30:41

       Mm-hmm.

    6. ZDZach Davis30:42

       And in, in the background, I will have Cursor, uh, take the next step. So-

    7. CVClaire Vo30:45

       Okay

    8. ZDZach Davis30:46

       ... uh, I'm actually gonna say... So it has this context, it knows I'm in this file, and I'm just gonna say: "Can you take the next tier of tasks?" I can see here there's a tier one, tier two. There's three files. I think that's reasonable. "And, uh, fix them." And I'm just gonna say click Go.... and we're gonna see what happens. Um, okay, so in, in the meantime, what I did to actually produce this is I ran Yarn test, and I, I piped the output to a log file, which I'm, I'm not, like, a super techy tech person, and so I actually asked Cursor how to do that effectively. Uh, and then I had a log file, and I pipe- I gave that to Claude, uh, to Claude Code, and I asked Claude to basically create this file. And, uh, so what it did is it went through all... It, it actually had trouble with how big that file was-

    9. CVClaire Vo31:34

       [chuckles]

    10. ZDZach Davis31:34

        ... but it was smart about, uh, working around that. And so it found out that we have something like 1,200 extra lines in a, in a test run that we don't need to be there, that we don't really want there. And, and then it quantified this, or it sort of grouped this into different types of warnings, and then which files are, are the worst offenders. And so then o- once we had this file, I said, "Great, like, go- can you go fix, like, the worst, the, the tier one worst offenders?" And so it actually went and has, has done that successfully. That's been merged in, reviewed and merged in. And then I can do stuff like this, where I just say, like, for any... The thing that I like about this is you can just give this to any agent now. I can Slack Devin and say, " @Devin, can you pick up the next task in the front-end test noise cleanup?" Uh, I can do it here in Cursor, uh, and watch it go. I could give it to Cursor background agent. It, it sort of, like, makes it easy to pick these things up as, as individual tasks, uh, and, and make progress on them.

    11. CVClaire Vo32:38

        W- what I like about this approach as well is it's very- there's a lot of parallels to how you would approach something like this with an engineering team of, of human partners, right? You're gonna take a problem, somebody's gonna go investigate it, and identify priority tasks to do. You're gonna put those tasks in some sort of task tracking system, and, um, you and I both know all of our beloved, uh, task tracking and project management systems, and I am starting to see Cursor markdown files become the new task tracking system. So I'm seeing this, this trend of these check, check marked files in Cursor just being the source of truth for progress on initiatives. So you've created basically a list of epics and tasks here, if that's what we call it, and they're prioritized by how severe they are. And then what I like about how you're approaching this, instead of saying, like, "Rip through all 1,300 noisy lines," you're saying, "Prioritize them. Do them one by one." And then what I'm presuming you're doing is the work happens, whatever agent you decide to do the next task, closes it out, you review the PR, you make sure any changes work, you merge it, it gets marked off. The other thing I wanna call out is, while you are probably running this yourself, you could probably also get more people on the team to be aware that this test exists and just say, "Hey, if you have a few minutes and you're able to review a next set of noisy tests, like, tell Devin to pluck off one and do the, do the code review for me." And it's all set up, and it's ready to go. So, you know, I think this multiplayer aspect is very important when how, how you approach, um, some of these tools when you're working in a larger, larger team.

    12. ZDZach Davis34:22

        Yeah. I... Just today, I had a, a PR up to, to fix a few stray errors on this, on this one file, and one of the, one of the people from the team that works primarily on that, I included them in the review, and he said, "Hey, if there's any more, uh, stuff like this, feel free to kinda, like, throw it over the wall to us. You don't have to be the one that does all of this." He didn't know that I was just using, uh, you know, Cursor or Claude or whoever. And so now I can actually just point him at this file. I said, "Hey, you know, take a look at this file, and if there's any ones you wanna pick off in your ownership area, then, then just go ahead." Uh, and, and you're exactly right. Doing that, democratizing that, this is great for... Again, I f- I'm saying the same things, but it's, it's great for bots. It's also great for humans. Humans can come in here and understand this, and, and work against it.

    13. CVClaire Vo35:12

        Yeah, and if you're feeling like, you know, crafting your farm-to-table code, and you wanna pluck one of these off yourself and you [chuckles] wanna fix it, you can approach it the same way, right? Just open the file, mark the thing as done, do the PR. And so I really do think it's important that folks think of kind of these tools as an extension of the team, and the more the tools can operate the way the team would operate, and the more the team can operate in the same way the tools can operate, then, then we can kinda all collaborate together and be, be much more efficient. So I think this is a great, super great example. Um, I'm not gonna make all of us watch Cursor go through tests and lint errors, because I have lost enough of my life [chuckles] to doing that. But I think it was a really, really great example of tech debt. And then just to ask the question, you know, what's the end payoff for front-end developers? Like, the actual issues bubble up in your tests and, and these tests get less noisy?

    14. ZDZach Davis36:12

        Yeah, I, I think one is it's easier to find stuff when stuff is going wrong. Two is y- I think it said that the biggest problem was actually accessibility warnings, so that's, that's, like, a real problem that, that exists, but when there's 1,200 lines of, of that, and a lot of that's coming from, like, the same component, if it's tested a bunch of times, we'll, we'll spam the logs, and... But being able to sort of surface the actual signal through the noise, I think is, is one of the key benefits.

12. 36:40 – 40:34

    Demo: Using AI to improve hiring processes and interview feedback

    1. CVClaire Vo36:40

       Okay, and then for our last workflow, I know that, um, Zach, you're gonna impress everybody, and everybody's gonna think you are just an AI-enabled, you know, cutting-edge engineering leader who only works with his army of bot friends. But you're actually [chuckles] hiring at LaunchDarkly. We expand the team. Um, you're always bringing in great talent, and you've actually used AI, um, to solve another problem, which is making sure that you're doing a great job hiring. So do you mind spending a couple minutes on what that little workflow looks like?

    2. ZDZach Davis37:12

       ... Yeah, absolutely. So I am, you know me, I'm, I'm a, I'm a little bit of a conflict-avoidant person. I don't love giving people tough feedback. You know, it's something I've, I've grown to do over my career, but especially when it's someone I don't have a, a strong relationship with, it's not a direct report. I don't love just dropping in and being like, "Hey, this isn't great." Uh, but we were trying to improve, make our, our hiring more consistent, and I created a rubric for all of, all of the panels that we have. So there was really clear guidelines about how to score a candidate, but the other piece of it was we needed people to follow those guidelines, and I wanted to be able to give people feedback about whether-- basically, I wanted to raise the bar of the actual scorecards that we were creating. So I created this custom GPT, I gave it the rubrics, and I gave it examples of good, good scorecards, bad scorecards, um, and gave it as much kind of like... You helped me write the prompt, so thank you very much for that. And so what I'm gonna do is I'm actually just gonna paste in, uh, a scorecard. Uh, and so this is, you know, uh, a scorecard that we got, and I'm gonna click, click Go, and it's going to do a few things. One, the rating that it's giving me is, is the rating of the scorecard itself. So it's a little meta, um, but I wanna know... Basically, I-- one of the things I did in the prompt is I said, "Give it a rating. Is it excellent, good, fair, or poor in terms of scorecard? And then I want you to list out strengths, uh, and, and potential improvements." And so, and then the last thing that I had to do, which I also thank you helping with, so thank you very much, is what I-- the format I wanted to give this to people in was to send it to them over Slack, right? Like: "Hey," like, "thanks for doing this, uh, but also I had a little bit of feedback," and so it actually, it, it gives me the detailed feedback, but then it also crafts a short Slack message that I can, if I want, just copy and paste and, and send to the person who, uh, created the scorecard.

    3. CVClaire Vo39:18

       I love this 'cause so many managers and hiring managers can empathize with this. Because if you're running an interview panel, you're having everyone from your boss to your direct reports to people you've never really worked with directly interview candidates, right? So you have these cross-functional interviews, and while you can have all the rubrics in the world, interviewers sometimes write terrible notes and, you know, say-- you know, assess the wrong things or don't give you the right details or are really using the rubric incorrectly, and you're not sitting in every single one of those interviews to give live coaching. And so this is a really nice way to make sure that you're holding the kind of standards very high and then giving you some leverage as a manager to give your team coaching, and then as they get this coaching, they get better at doing the, the interview feedback, and then you can be more confident in, in your hiring decisions.

    4. ZDZach Davis40:09

       Yeah, and honestly, this helped me. Like, as what... In order to test this out, I was doing a bunch of interviewing, and I was writing scorecards, and I would paste it in and see what kind of feedback it gave me, and [chuckles] it was giving me very good feedback. And I learned very quickly the kinds of things, be, you know, be more specific, uh, you know, avoid, avoid certain kinds of things, and so it actually made me write better scorecards just through trying to create this

13. 40:34 – 42:08

    Summary of key approaches for integrating AI into engineering workflows

    1. ZDZach Davis40:34

       tool for other people.

    2. CVClaire Vo40:35

       Okay, Zach, you have given a masterclass in how engineering leaders at larger companies can really approach your i-i-integrating AI into not just their individual workflows, but their team workflows. So just to cover what we talked about, one, let the team experiment. Like, every tool, just let's see what works. So you seem pretty kind of, uh, generous with your experimentation mindset around what tools can use, um, can, can bring value to the team. I think, two, context is king, and so you are loading up your actual repository with docs and rules. Three, those rules are centralized, so you don't use agent or tool-specific rules. You create a central agent repo and then point all your specific tools toward that. You use your AI tools to actually create those rules. You, um, use, uh, Cursor and other tools to create plans to burn down tech debt and then have those AI tools burn down that tech debt. And then, um, since you have all this free time, [chuckles] now, now, uh, you're coaching yourself and your team to be better, uh, interviewers and better hirers. So, so just that. No big deal.

    3. ZDZach Davis41:40

       That's it.

    4. CVClaire Vo41:41

       That's all you have to do.

    5. ZDZach Davis41:41

       A few things, yeah.

    6. CVClaire Vo41:43

       And this is all- I mean, truly, from a, from a personal-professional development perspective, these skills were developed, what, in the last twelve months, right?

    7. ZDZach Davis41:50

       Oh, I think January is when I really started, like-

    8. CVClaire Vo41:53

       Okay

    9. ZDZach Davis41:54

       ... took on the mantle and playing with Devin and really going down this, this path full heart.

    10. CVClaire Vo41:58

        So, so six months, and we didn't give you-- I mean, we, I think we offered, but, like, formal L&D, none of that. We just pushed you into it and said, and said, "Go."

14. 42:08 – 44:56

    Lightning round and final thoughts

    1. ZDZach Davis42:08

       Yeah.

    2. CVClaire Vo42:08

       Okay, I'm going to wrap with two quick lightning-round questions, and I'll get you back to all your AI-assisted code. Question number one, you listed so many AI tools. Which one is your favorite, or which one has been most transformational?

    3. ZDZach Davis42:23

       Ooh, that's really hard. I would say Windsurf, actually. Everyone was hot on Cursor, and I... It just wasn't-- the UX at the time was not clicking for me, and I saw a video for Windsurf, and I was just like, "Whatever, I'll give it a try." And I had the free trial, and within an hour, I think I was paying for it because I just-- it really clicked for me, and the agent workflow just really quick- clicked, and I was hooked.

    4. CVClaire Vo42:49

       Amazing. And then when AI is not listening to you... You're such a-- you're conflict-avoidant, so I'm actually very interested in your answer here. AI is not listening to you. Um, you need to give it harsh feedback. What are your tactics? I know you don't yell. I know you're very polite, but what do you do?

    5. ZDZach Davis43:07

       I mean, some, sometimes I lose it, but I have to-- I, I... The thing that I actually do is sometimes I just feel like it's not the right task, right? So it depends. If I, if I think it's something that AI should be good at, then I, I, I get a little snippy with it. Maybe I don't yell, but I'm definitely, you know, um, getting, getting a little annoyed. But I also think that sometimes it's okay, right? Like, sometimes it's not gonna work, and you don't have to keep banging your head against it, and I think developing that sense of where it works and where it doesn't has been really powerful for me. And also, sometimes I just like getting in there and, and getting, getting dirty, uh, getting, getting my hands in the code. And so, uh, yeah, I think my, my technique is actually either I do it myself, or I go back and try and fix it. You know, am I providing the right context? You know, what, what, what is missing that it can't accomplish this effectively?

    6. CVClaire Vo44:04

       Yeah. You're, you're a very good manager, so I think it's from those skills. All right, Zach, this has been super informative. Where can we find you, and is there anything we can do to be helpful?

    7. ZDZach Davis44:13

       I'm on LinkedIn. Um, we are hiring at LaunchDarkly, and also, if, if you are a LaunchDarkly user and you have any feedback, I love user feedback, so please send it, send it my way.

    8. CVClaire Vo44:24

       Amazing. Well, thank you so much, Zach.

    9. ZDZach Davis44:26

       Thank you. [upbeat music]

    10. CVClaire Vo44:28

        Thanks so much for watching. If you enjoyed the show, please like and subscribe here on YouTube, or even better, leave us a comment with your thoughts. You can also find this podcast on Apple Podcasts, Spotify, or your favorite podcast app. Please consider leaving us a rating and review, which will help others find the show. You can see all our episodes and learn more about the show at howiai pod.com. See you next time!

Episode duration: 44:56