The Right Model for Each Coding Task

1. 0:00 – 4:02

   Optimizing AI Coding With Weave

   1. SPSpeaker0:00

      [upbeat music] I'm excited to welcome the founders of Weave here and to congratulate them on their recent Series A from our friends at Standard Capital, and also the launch of their new product, Weave Router, that they're going to talk about. Um, welcome, guys.

   2. ACAdam Cohen0:17

      Thanks.

   3. ACAndrew Churchill0:17

      Yeah, thank you.

   4. SPSpeaker0:18

      Yeah. So first off, tell us about Weave. What is Weave?

   5. ACAdam Cohen0:22

      Yeah, so Weave is a tool to turn any software company into a software factory. So we build models to measure engineering work, and we use that to optimize humans and agents and how humans and agents work together.

   6. SPSpeaker0:38

      And who's using Weave right now? And tell us a bit- a bit about where you guys are at as a company.

   7. ACAndrew Churchill0:42

      Yeah, absolutely. Um, so we're lucky to work with a bunch of great customers, uh, like Robinhood, uh, PostHog, Telnyx, uh, Extra Space Storage, Reducto, and a whole bunch more companies from, like, tiny startups with, like, a few people all the way up to, you know, Fortune 100 companies.

   8. SPSpeaker0:56

      You guys have had a blowout year-

   9. ACAndrew Churchill0:57

      [chuckles]

   10. SPSpeaker0:57

       ... 'cause the more that AI coding takes off, like, the m- um, the more critical Weave is to companies. Can you talk about how things have gone this year?

   11. ACAdam Cohen1:05

       Yeah, it's been awesome. So we recently raised our Series A, um, on the back of just our growth. The reason for this is everything we were doing for humans in terms of how we were measuring and understanding the work they did and how to optimize how humans are working, we realized this could easily be applied to coding agents. So this has just exploded, obviously, with the explosion of how much people are using these coding agents.

   12. SPSpeaker1:26

       And you guys recently a- announced Weave Router. Um, can you talk about Weave Router and how people can save money by using it?

   13. ACAndrew Churchill1:35

       So what Weave Router does is it sends every task to the best model for that task. You plug it into your coding agent, it sends everything to the router. The router decides the best model, uh, for any individual prompt or task to go to, um, and it routes it to that model. So you don't have to think about, should this go to Opus? Should this go to Astra? Should this go to DeepSeek? It all is handled for you, and we've, we've, we've trained a model, uh, that's basically better than any human or really anything out there at getting the most out of every single model, um, to the point where on, on recent benchmarks, we're able to hit Astra-level performance, uh, at half the cost and twice the speed.

   14. SPSpeaker2:07

       What coding agents can I use it with?

   15. ACAndrew Churchill2:09

       Claude Code, uh, Codex, OpenCode, Pi, uh, really anything where you're working in a CLI, you can plug it directly in with one command.

   16. SPSpeaker2:17

       And how do I know it doesn't make my coding agent worse-

   17. ACAndrew Churchill2:21

       Yeah, that's a great question

   18. SPSpeaker2:21

       ... by sending it to, like, a dumb model that's just gonna, like, write bugs?

   19. ACAndrew Churchill2:24

       Yeah, that's a great question. That's, that's, uh, one of the biggest potential problems with doing this. So we actually have a interesting ability to know that it's not doing this because this is-- You know, our entire product, uh, is focused on understanding how engineering teams are working and understanding, uh, both the velocity and the quality. And so I can talk about our own team directly. Um, you know, we're all using the router, and, like, our output only continues to accelerate. Uh, you know, we're in, like, the, the top 1% of all teams based on our own data, and there's no increase in, uh, the amount of bugs we're introducing, the amount of, like, quality issues that are, that are, like, making its way into the product. So that's the, the biggest testament I can think of to, uh, this working well.

   20. SPSpeaker3:00

       Okay. And who, who can use this right now? Like, do I have to get my whole team on board? Can anyone just go and install it? Is it open source? Yeah.

   21. ACAndrew Churchill3:07

       Yeah. So it's open source, uh, fully open source. Uh, anyone can go and try it out. Um, we make it really easy through our hosted version. It's a little bit more setup if you, uh, need a self-host. You gotta set up all your keys and stuff, um, but it's possible. And yeah, we, we have subscription plans as well, so if, if anybody out there wants, you know, to get the best out of every single model, um, they can sign up for a Max or a Boost plan.

   22. SPSpeaker3:27

       Yeah, yeah. Like, suppose I have subscription plans. The optimization is a little bit complicated because, like, I want to maximize the use of my, like, Claude plan and my, and my OpenAI plan before you start shipping stuff out to other stuff. Do you take that into account?

   23. ACAndrew Churchill3:41

       Yeah, absolutely. Um, th- this is a, this is a huge thing that I think, like, a ton of especially, like, early-stage startups are really into maximizing these subsidies.

   24. SPSpeaker3:48

       Yes.

   25. ACAndrew Churchill3:48

       Uh, you know, use it while you can, and we totally help with this as well. So you can, you can, you know, log in with a bunch of different accounts, and we'll route between those different subscriptions, uh, using all the subsidy while we can and then sprinkling in open source models where it's going to be the best performance even with the subsidy.

2. 4:02 – 8:39

   Getting More From Your AI Budget

   1. SPSpeaker4:02

      If, if I'm already paying for the $200 plans at Anthropic and OpenAI, and I'm, like, a hardcore token maxer where I'm already maxing out both of those plans, and I'm trying to push it even further, do you have a sense of, like, what will, what the model will do in practice if I'm using Weave and what it will do to my, like, end of month, like, total output?

   2. ACAndrew Churchill4:21

      Yeah. Yeah, absolutely. So first of all, for the same amount that you're spending on those, on those plans, you'll, you'll go, like, twice as far. Um, and so we do this by... If you're, if you're one of these people, you're probably sending everything to one of the biggest models, which is understandable. If it's all subsidized, you know, may as well just go to the smartest thing.

   3. SPSpeaker4:34

      Yeah.

   4. ACAndrew Churchill4:34

      The thing is, you know, you don't always need a bazooka where a handgun will do. Uh, and so maybe, for example, uh, you're exploring a code base, so you only need, like, Luna in order to figure out, like, oh, this is where Auth is implemented in this code base. Um, and so, you know, that sub-agent or this portion of the conversation will go off to that cheaper model. It doesn't affect you at all. You still get the same result at the end of the day, um, you know, even faster. Um, but it's using up less of that, less of that usage limit. Um, and so you don't hit your usage limits nearly as much. Um, and it can route between the different plans, uh, without you having to always, like, log out, log back in.

   5. SPSpeaker5:03

      Okay. And a big part of how you're getting these savings is you're routing not just the cheaper models from OpenAI and Anthropic, but to open source models.

   6. ACAndrew Churchill5:11

      Yeah.

   7. SPSpeaker5:11

      Um, can you talk about the sort of explosion of open source models and how this has changed the picture? What open source models is it going to? Like, what, what providers? W- how are you seeing in, in, in, in the open source model land?

   8. ACAndrew Churchill5:22

      Yeah. So the, the explosion of open source models has been, I think, reflective of the fact that they are, uh, not at the frontier, but they are not that far behind the frontier either. And, uh, the, like, cost performance, uh, is, is basically unparalleled. Um, so if you think of, if you think about, like, a Pareto curve, right, they're, they're on the, like, much cheaper side, but it's pretty incredible the level of performance they get for how cheap they are. So it's not-- You know, you don't want to give DeepSeek a task of, like, build an ML system to, you know, classify a, a billion images. Um, you probably want Astra looking at that type of a thing. But to do something like, you know, center this button, um, or even, like, you know, implement, implement this backend that returns this, like, you know, does this CRUD operation- DeepSeek is going to do great at that. And so as far as like what's actually being used, I think the, the biggest ones are, are DeepSeek, uh, Kimi and GLM models. Um, these are the ones that have like, uh, done the best on our own internal testing and benchmarking. Um, and so we route a lot of traffic to those for some of these simpler tasks. Yeah, it's, it's, it's pretty shocking. I think for pretty much any like hobby project, you don't really need to go beyond open source models. At least in my own experience, I've been, uh, you know, doing a bunch of like things on the side with this and the, uh, like open source models only will get you very, very far.

   9. SPSpeaker6:31

      And how does this compare to other model routers? Like how would this compare to me using OpenRouter, for example?

   10. ACAndrew Churchill6:36

       Yeah, great question. So, um, the key difference is, uh, we're doing the routing, uh, like intelligently. OpenRouter, uh, you know, maybe would be better referred to as OpenGateway, um, their, their, their whole game is they let you access as many different models as possible, and you choose which model you're using, and they just make it really easy and have all the fallbacks, which is great. It's super valuable. What we're doing is we're automatically choosing the model for you. Um, and specifically we're doing this in the coding agent use case, and these are-- this is something that you can't do at least, uh, according to benchmarks, you can't do well on OpenRouter. Um, they have an auto router, um, it doesn't work very well. It's fine. It's not really their focus.

   11. SPSpeaker7:09

       How does your auto router work? How do you, how do you know what model to route to?

   12. ACAndrew Churchill7:13

       So we have trained, uh, our routing model on hundreds of thousands of coding agent sessions, um, where we're looking not just at the coding agent session itself and what's working and what's not, um, but also, uh, all the downstream effects, um, because, you know, we, we can see all this, like how does this, uh, reflected in the PR that results? How is this reflected when it gets merged to production? How is it reflected a month down the line? And so we can see like which models are effective at which type of jobs. Um, and then we also do a bunch of additional training on top of that where we, you know, test out, you know, sending it down this model's path or this model's path. Um, and, you know, we, we basically, uh, give it a reward if it, uh, accomplishes a task successfully. Um, and so through this combination of like training off of, uh, lots of real sessions and then also doing a bunch of like, uh, you know, testing to see like where the different models excel, um, we're able to like have a much better picture of what each model's strong point is, um, and, and learn that automatically. Um, so it's, it's, it's not us saying like, "Hmm, we think this would be good for, for DeepSeek." It's like it's a, it's a machine learning model that we're training.

   13. SPSpeaker8:11

       The reason that you guys were able to build the smart model router and other people, including OpenRouter, haven't been able to, is because you guys have been like doing a different thing for the last year and a half. So I think this is a good time for us to talk about how you guys got to where you are. Um, let's rewind back to the batch. So you guys were in Winter '25, and when you were in Winter '25, you were doing only a part of what you're doing now. Can you talk about what Weave was like in the batch?

3. 8:39 – 12:30

   Measuring Work Beyond Lines of Code

   1. ACAdam Cohen8:39

      Yeah. So back in all the way January '25 feels like worlds ago, um, we were just focused on understanding and measuring engineering work. Uh, this was historically like impossible to do, to actually understand the work that engineers do, how productive they are, how many bugs they're creating, um, and-

   2. SPSpeaker8:55

      All you could do is like count the lines of code.

   3. ACAdam Cohen8:57

      Exactly.

   4. ACAndrew Churchill8:57

      Yeah.

   5. ACAdam Cohen8:57

      Lines of code-

   6. ACAndrew Churchill8:58

      That was state of the art [laughs] .

   7. ACAdam Cohen8:59

      Or the number of PRs-

   8. SPSpeaker9:00

      PRs

   9. ACAdam Cohen9:01

      ... door metrics-

   10. SPSpeaker9:02

       Number of tickets closed.

   11. ACAdam Cohen9:02

       Like these are all very fancy kind of feng shui metrics to try to make executives feel good, so we created a way to actually measure productivity, and we build models to actually do that. Um, during the batch, what we realized very quickly, it was the first batch where Cursor was just starting to take off. People were writing all of their code with Cursor, uh, for the first time, and people started asking us to show us what is the impact that Cursor is having on their productivity. So that was kind of the entrance point to, okay, we really need to figure out how to measure coding agents, not just the human engineers.

   12. SPSpeaker9:34

       I remember it being kind of a crazy time because there were like two things happening at the same time, both caused by AI. The first thing that was happening caused by AI was for the first time it was possible to measure human performance because you guys wouldn't just count the lines of code. You would have AI models look at the code and actually read the code and understand like what the person was actually doing, which enabled you to actually get, get, um, like real indicators of how productive different engineers were on the team for the first time ever. And at the same time, the same LLMs that enabled you to do that were also changing the way that those engineers were writing code. So these like two major shifts happening at the same time. And, um, 2025 was this crazy time, um, you, you guys were just reminding me, it was the first batch where we surveyed the batch and a significant number of people said that most of their code was being written by LLMs instead of by humans.

   13. ACAndrew Churchill10:23

       Which was shocking at the time, by the way. There was a lot of kerfuffle on Twitter about like, "What? They're doing what?" [laughs]

   14. SPSpeaker10:27

       Yeah, I, yeah, I, I remember we, we tweeted about this and there was like outrage on the internet of like, how dare the programmers like, you know, have AI, have, have AI write the code for them. Yeah. Um, what was it like selling early Weave in this environment of early 2025? This is before Claude Code. I think Claude Code had just launched and no one was using it, and everyone was using Cursor.

   15. ACAndrew Churchill10:50

       As a research preview.

   16. SPSpeaker10:51

       Yeah.

   17. ACAdam Cohen10:51

       Yeah, absolutely. So it, it was super interesting because there was clearly like a lot of, um, demand that people wanted to measure software engineering, but every single call we got on was pure skepticism. "I don't believe that you can actually do this." Um, so we had to actually prove it to them with their data. We would connect their data and show them, "Here's how it's actually working. Does this match your expectations?" Yeah, so that was like the kind of like huge liftoff moment when we started being able to prove it. Beyond that, what we realized is we were working with a lot of the fastest growing startups because they were the ones coding with coding agents, so that gave us this kind of always advantage of being ahead of the curve. We were always six months ahead of the curve, um, because we knew exactly what the people at the forefront were doing.

   18. SPSpeaker11:36

       That's interesting because like the, the early stage companies, like the batch companies, maybe they didn't care as much about measuring engineer performance because they didn't have any engineers [laughs] .

   19. ACAdam Cohen11:45

       Yeah.

   20. SPSpeaker11:46

       Like, but they, they were living at the edge in terms of using coding agents to write all their code versus the companies that you were selling to who were maybe like a couple years ahead, who had big engineering teams who really had the problem. They were six months or a year behind at using the coding agents.

   21. ACAdam Cohen12:02

       Exactly. And that's what we've seen- Honestly, till today, like we're always kind of been six months ahead of where people are typically needing to be because we continue to work with these fast-growing companies or these early, early startups.

   22. SPSpeaker12:13

       Yeah.

   23. ACAdam Cohen12:14

       Um, yeah, for them in particular, I think the biggest thing was just, uh, they want to measure their coding agent effectiveness, and was it actually faster than them themselves typing the code? And we were always able to prove that, so they kept investing in tokens and on their coding agents.

   24. SPSpeaker12:29

       Like

4. 12:30 – 16:47

   From AI Adoption to Optimization

   1. SPSpeaker12:30

      the, the core thing that Weave did is it helped engineering managers understand what was going on in their engineering teams. How have the questions that those engineering managers are asking you and asking your product changed from early 2025 to now?

   2. ACAdam Cohen12:44

      So it very much started off with just like how fast are we moving and how does that compare to the general benchmark? Then it was, uh, is AI actually making us faster? And then it's, okay, how can we start to use AI more and more efficiently? Uh, and that's kind of where we're at today, is this more efficiently. Everyone is writing all of their code with AI, maybe not everyone, but most people hopefully, um, that we work with at least, and now it's figuring out how do we start to optimize this? Um, this is things like this is exactly what brought us to the router, because that's one way to optimize is pick the right model for each task, but it's also just like how are you prompting, how are you doing agent parallelization, how are your agents task handling? Um, and that's what we help teams actually understand and see.

   3. SPSpeaker13:24

      Broadly, are teams more excited about how do I like increase the productivity of my team and get people to adopt AI more, to token max more, to like use more AI, write more code, or are they more concerned about other stuff? Like how do I make sure that people aren't just writing AI slop and like wrecking my code base? How do I make sure that they're not just like spending a gajillion dollars on like Fable and not actually like I'm, I'm not actually seeing the ROI?

   4. ACAdam Cohen13:52

      It's kind of this whole-

   5. SPSpeaker13:52

      How does it break down?

   6. ACAdam Cohen13:53

      Yeah. It's this transformation curve. So it's like at the start people very much care about just getting AI adoption. How much can we actually get? Uh, it goes into this token maxing phase, which is just let's spend, spend, spend as much as we possibly can, and then it goes into that optimization layer, which is, okay, now everyone's using AI, they're AI native, they're AI first. How do we fine-tune this to make sure we're actually not just burning unnecessary tokens on things like Fable? For us, like yeah, startups are typically in that kind of still token maxing phase. Um, they don't care too much about cost optimization, more about productivity optimization. We also have a lot of late stage companies, some private equity, uh, owned companies that are much more on the optimization layer of the curve.

   7. SPSpeaker14:35

      Can you throw out some ways, some concrete ways that Weave has helped people accomplish their goals? Like what's cool about Weave is that Weave can kind of help you achieve your goals, like whatever, like wherever you are on that adoption curve. So perhaps you can throw out a couple examples of people at different points of that adoption curve.

   8. ACAdam Cohen14:53

      One of my favorites would be a company, Uncountable actually. Um, they're using Weave right now to build out a software factory. Uh, and what they're doing is they're comparing the software factory that they're actually building and using to how is that doing versus the humans and, uh, which work can we continue to move into more of the like lights off factory as opposed to, um, humans doing that work.

   9. ACAndrew Churchill15:15

      Yeah. I think another, another pretty common one-- It's really interesting. A lot of times the first time people connect their data, uh, and we, and we take them through it, there will be some pretty like shocking outliers in the positive sense. Um-

   10. SPSpeaker15:26

       Interesting. Yeah. Well, what are the most common reactions people get the first time they open the fully populated Weave dashboard that has the full picture on what's been going on in their engineering team that they've been running?

   11. ACAndrew Churchill15:35

       Some of the really exciting ones is like, "I had no idea that that person was like that effective." If you're not feeling it, if you're not seeing every single PR, it can be hard to know. Um, but when you see it in numbers, uh, it becomes very, very clear. Um, and so for example, there will be people on a team who are, you know, maybe like they're spending like 10K a month or even more, but they're actually like they're-- the ROI is there. Like they're-

   12. ACAdam Cohen15:54

       Mm

   13. ACAndrew Churchill15:54

       ... they're also, you know, similarly like, uh, like shipping a lot more, like using that money. And so that's really exciting for people because it gives a blueprint. Um, this is what good looks like, and now we know that, and we can, uh, like one, use Weave to dive in and see like at the single like prompt level, like what is this person doing that's really effective? Um, and then, you know, at the kind of like system level, how can we spread this across the entire organization? So that's, that's I think the, the kind of like class of, uh, insight that people get when they connect all their data and see it for the first time that's most exciting.

   14. SPSpeaker16:21

       When people see people who are like huge outliers as like the, the biggest spenders of tokens and therefore the most expensive people on the team, is the, is the conclusion typically, "Wow, these are our best performers, and I'm so glad I have them on my team, and I wanna make everyone like them," or, "Whoa, this person just like blew $100,000, and I should have a talk to them about like, you know, how to be more intelligent about how they use these models"?

5. 16:47 – 20:16

   Measuring AI ROI and Coaching Engineers

   1. ACAdam Cohen16:47

      So one of the main metrics we show is like cost per output. So it's how many dollars are you spending on AI for unit of output. And when we talk about output, think of it as an hour of work for a human engineer to actually do. Um, the reason that's so important, it answers that question directly, is like someone could be spending $10,000, but if they're getting 10,000 hours of work out of it, you're only spending a dollar per hour of work. That's a really good deal. But there's the other cases where somebody's spending $10,000 and they're still getting 10 hours of work. Now you're spending $1,000 per, uh, hour of work. That's a really bad deal, and that's where somebody will interject and actually like course correct them and say, "Hey, you're using the wrong models," or, "You have this weird agent loop going on," and that's just burning tokens and giving us no real productivity. So you need those two data pieces to work together to figure out exactly like is this paying off or not.

   2. SPSpeaker17:35

      You were saying earlier that one of the ways that you've seen people, um, use the Weave analytics to make their teams more effective is by spreading the knowledge of the top performers amongst the teams. I know this isn't exactly what your product does, but I'm curious, have you seen best practices in teams for how people can actually do that and like upskill their team and make everybody as great as the like one or two top performers?

   3. ACAndrew Churchill17:58

      Yeah. Yeah. Absolutely. So I think the highest leverage, uh, like- single thing that we see is, uh, somebody's ability to parallelize. This is somewhat intuitive, right? If you are single-threading, then there's only so much you can get done. If you're multi-threading, there's a lot more you can get done. Um, the problem is that, like, parallelization is not just so simple as, like, let me kick off 10 Claude code sessions and just let them cook. What makes it hard to do it well, uh, is how do you actually create the right guardrails, the right frameworks, um, to make sure that those agents are, like, doing the right work and that you're getting pulled in at the right times and not at the wrong times? This is, this is, I would say, an evolving art, um, and it's interesting because it evolves every time a new model releases. Um, and the capabilities of the model change and, like, where the human attention is necessary changes. This is one of the most common thing is we see, like, somebody has really nailed this parallelization. Um, they're, they're quite good at, uh, you know, doing a bunch of things in parallel and u- using their attention as, like, w- in the highest leverage moments possible. Um, and what, what does spreading that look like? Um, often it looks like, uh, you know, sharing the, the kind of like guardrails, uh, the kind of... Maybe it's, maybe it's g- sharing a skill. Maybe it's like, you know, uh, kind of like refactoring some part of the code base to match a certain architecture that's much more familiar and, and, and comfortable for agents. That's where it kind of like varies from person to person, company to company. Um, but I'd say parallelization is like the, the kind of number one, uh, highest leverage thing.

   4. SPSpeaker19:16

      To add to that, um, our product actually does exactly this. Uh, it looks at all of your prompts, and it looks at what's landing in production, and it gives you specific skills that you could equip on, uh, how to use these tools better. Um-

   5. ACAndrew Churchill19:28

      Like what?

   6. SPSpeaker19:28

      ... it has like a coach-

   7. ACAndrew Churchill19:29

      Like what, for example?

   8. SPSpeaker19:29

      ... to actually do this.

   9. ACAndrew Churchill19:30

      Like, like give me some examples of things that, like, a coach might tell me.

   10. SPSpeaker19:33

       Yeah. So the... I mean, what it does is it's looking at literally every single session that you have, um, and it's giving you different little pointers. So this could be like all the ones Andrew was just talking about, how to, uh, use your agent to handle tasks or how to actually push back on your agent better, because that's something people don't do enough now, uh, is tell your agent that, "No, you're wrong." Or it might say, "Hey, actually, you should be restarting sessions more often," as opposed to just continuing to run with the current session because-

   11. ACAndrew Churchill19:59

       Big one

   12. SPSpeaker20:00

       ... o- often people don't-

   13. ACAndrew Churchill20:01

       Oh, because, because, because, like, the cost of the session just gets bigger and bigger as the context window gets, gets higher and higher, and, like, their engineers should be restarting the sessions more, more often.

   14. SPSpeaker20:10

       Exactly.

   15. ACAndrew Churchill20:11

       Yeah.

   16. SPSpeaker20:11

       So we have this whole analysis layer that goes on, and it's giving you these kind of insights every

6. 20:16 – 25:38

   Building Software Factories

   1. SPSpeaker20:16

      single day.

   2. ACAndrew Churchill20:16

      Hmm.

   3. SPSpeaker20:16

      Let's talk about software factories. So like the-- If the, if the base case of getting good at using AI coding is like learning how to run 10 Claude code se- sessions in, in parallel, maybe the next level is learning how to set up software factories that are coding while your engineering team is asleep. Um, what are you seeing the best teams do there? What are their software factories actually doing? And that's a place where you could easily spend a shit ton of tokens on like low ROI things, because the thing is just off and running autonomously without any human. How does Weave help me make sure that my software factories are actually like operating?

   4. ACAndrew Churchill20:48

      I think everyone who's ever tried to use a coding agent knows that they can spit out as much code as you ask them to. And so it's not really about measuring, uh, just the outputs. Um, it's also about, um, understanding like the validation. Um, I think this is the hardest part of, like, building a software factory is understanding how to like correctly validate this. Because again, an agent can put whatever you want into production, um, but i- if it's not good, then it's not worth anything. What, what people are normally doing is, uh, they're handing work over to like the agents, you know, the, the software factory, and then in Weave you can see like a very direct comparison. Okay, here's what our, you know, software factory agents, here's the outcomes that they're getting, here's the number of bugs that they're introducing, uh, here's the number of, you know, like customer reports coming from, uh, from whatever, whatever they did. Um, and you can like very directly compare that to, uh, the before or to like a, a parallel team that's maybe like not using such a kind of autonomous system. And then when you know what the problems are, that's when you can like build the systems that actually, uh, that avoid that entire like class of issue. Um, and so it's this kind of like constant iteration loop. All right, let's put this through the, through the system. Let's see where it fails. Let's fix that problem.

   5. SPSpeaker21:54

      What are the software factories you guys have running at Weave today? What's coding while you guys are here recording this podcast?

   6. ACAndrew Churchill22:00

      So for building product, I think the, the-- what we feel is like the highest leverage thing that we can do is, uh, picking the right things to work on. Um, and, uh, I guess like second to that is like building a system that can like build that product for us, um, as, as well as possible. You know, on the one hand, it's like, how do we, how do we, like, make sure that we are like prioritizing and getting to the right, uh, the right set of features for the, for the product? Um, and then on the other hand, it's how do we make sure that, like, it's not just slop, um, that this is actually something that is going to be useful to a human or an agent that's using the platform? The basics here are, um, we have, you know, it's, it's easy to run coding agents both locally and in the cloud. Um, we have like really good guidance for these coding agents. We have like all kinds of, like, you know, we have like a, a custom CLI that like will, uh, in, in a very token-efficient way, give the agents the context they need to, uh, build a new feature or fix a bug or like iterate based on feedback from, uh, CI or, or testing or whatever. The other side of this is like, uh, validating the things that humans, uh, like, you know, like a, a senior software engineer who's reviewing your PR might do but is otherwise not done. Um, so I think there's like two pieces to this. One is, uh, finding bugs. I think, um, a lot of the, like, off-the-shelf, uh, review tools out there are quite good at this. We use, we use four of them at Weave-

   7. SPSpeaker23:12

      Hmm

   8. ACAndrew Churchill23:12

      ... uh, two of which are YC companies, Greptile and, um, and Parameter. Um, and then we also use Cursor, BugBot, um... Oh, actually three of them, yeah, and Cubic. So [chuckles] three out of four YC companies on every single PR. Yeah, that's right.

   9. SPSpeaker23:23

      Wow, what's the dollar cost per PR?

   10. ACAndrew Churchill23:25

       Basically, it's worth it as long as each one is, is finding things the other ones aren't. Yeah, that's the calculation for us. And I, I would imagine over time these things will converge, and we'll have to choose the ones that are best, which, like, you know, we can use our own product to see. But we can still see in our own product each one is finding, um, unique things the other ones aren't. That's like one side of it. Then the other side is like, uh, okay, aside from just like the kind of single, like myopic goal of this one PR, how do we make sure that this is, uh, this is going to fit into the system well, that it's going to be like a scalable architecture, it's going to be easy for like future contributions? And I think this is the biggest place where the kind of like lights-off software factory approach fails, um, because every PR is locally logical, and then you put it all together, and it's something that's globally insane with like all kinds of like, you know, terrible one-off, uh, one-off workarounds and all that stuff.

   11. SPSpeaker24:09

       Mm-hmm.

   12. ACAndrew Churchill24:09

       And so actually for that, we've, we've built like our own kind of like checks system. You can think of this as like the QA for the software factory. Um, and so what, what-

   13. SPSpeaker24:16

       It's a software factory factory.

   14. ACAndrew Churchill24:17

       Yes. [chuckles] And so the way, the way, the way this works is, um, we define all the things that, you know, a, uh, like expert software engineer on the Weave team might really care about, the type of things that if we were doing human reviews on every single PR, they would commonly be calling out, and that, uh, engineers should over time be learning and not having to get called out as much. But agents will never learn because, um- You know, they, they don't have, like, memory like that. And then we put these into, we put these into these checks that are run on every single PR. Um, so for example, the type of thing that a coding agent will often do is over-abstraction. Um, you know, it, it ... There's like a, a single thing it needs to do, and it makes, like, this entire crazy, like, abstract system that will never be used. And so we have a check that, you know, prevents over-abstraction, um, with, with context on examples of, like, what good looks like, what bad looks like. These are the things that, like, you know, sometimes agents can do. Sometimes they can pull from the right patterns in the code base. Sometimes they pull from the wrong pattern in the code base, and then, like, the problem compounds. Um, and so this is, this is, like, the back pressure on that. You know, this is, this is ever-evolving. Like I said, every new model drop does, like, change the, change the parameters of the game a little bit, but these are the things that we've done so far. And I, I think the results are good. I mean, you know, we're, we're, we're top 1% on our own product in, uh, the most competitive segments, uh, of these benchmarks, which is the, you know, like, the s- the small startup segments. Um, so that's good. [laughs]

   15. SPSpeaker25:28

       So today, most people, the way they write code is they, like, set a model for Claude Code or Codex, and then that's the model that they use until they, like, go to the dropdown and they

7. 25:38 – 28:36

   The Future of Model Routing

   1. SPSpeaker25:38

      change it. How do you think it's gonna work in the future?

   2. ACAdam Cohen25:40

      Yeah. I think at this point it's getting exhausting to always stick up to date with, like, what every single model is good at, what they're bad at, what their strengths and weaknesses are. And that's why I think everyone's gonna be moving towards this router approach, where there's gonna be companies like ours, or hopefully you just use Weave, um, that takes all of this data and understands exactly what their strengths are so you never have to switch models. It's doing it always for you. The secondary part of this is that that same kind of concept leads towards, uh, if you actually understand what every single model's strengths are, you can actually start to build this router that is stronger than any individual model. Um, so you can pull the strengths of each of them and actually outperform whatever the frontier truly is.

   3. SPSpeaker26:22

      If you were giving advice to someone who's an engineer on a team, and they wanna be, like, the most cracked, most AI-pilled, like, person on their team, drag the rest of the team along with them, and just, like, really up-level everything about how they use AI coding, what, what advice would you give them? What should they do?

   4. ACAndrew Churchill26:39

      It's a great question, and my advice is maybe, like, a little ironic or strange. It's, it's, it's, uh, don't lose track of what you're actually doing. Um, I think what we constantly see is the engineers that are the best at using these tools are also the ones who understand the best, like, what they're actually doing, and therefore they can steer them and guide them the best. Something, uh, someone I really respect once told me is, uh, the best engineers understand everything two levels of abstraction below and two levels of abstraction above whatever they're currently working on.

   5. SPSpeaker27:07

      Mm-hmm.

   6. ACAndrew Churchill27:07

      And I think that's still 100% true in the world of AI. It's just that, like, the default level of abstraction you're working on has moved up, like, a level or two.

   7. SPSpeaker27:12

      Mm-hmm.

   8. ACAndrew Churchill27:12

      Um, but I think that when you, when you lose touch with what's actually happening, um, that is where it, it becomes hard to avoid being swept up in the tidal wave of slop.

   9. SPSpeaker27:20

      AI coding has changed dramatically in the year and a half since you guys started Weave. You, you know, when you guys started, Cursor was the big thing. Everyone was using that, and now we're living in a whole new universe. Um, where do you think things are going? What is AI coding going to look like in another year and a half?

   10. ACAndrew Churchill27:36

       I think we think that broadly, uh, running an engineering team will be an exercise in token allocation, um, where you find the people who are the best at looking out in the world, sensing the opportunity, and, uh, y- you know, like deploying their swarm of AIs to take advantage of that opportunity. They have, they have taste, they have sense, whatever it is, um, and they're able to, like, harness these agents really effectively. I think that's, you know, in the limit where you, you know, kind of like trace out the curve of how the models continue to improve, um, it's, it's, it's probably not going to be, uh, you know, necessarily the people who are, uh, the best at inverting a binary tree. Um, it probably is going to be the people who are the best at, uh, looking out into the world and seeing that here's a problem that needs a, a solution, and here's how I'm gonna build this solution, um, and here's how I'm going to make sure that this is actually, like, delivering on that.

   11. SPSpeaker28:23

       I think that's a great note to end on. Um, guys, thank you so much for joining us today.

   12. ACAdam Cohen28:27

       Yeah, thank you very much.

   13. ACAndrew Churchill28:27

       Thank you. [upbeat music]

Episode duration: 28:36