How Spotify runs agents across 20M+ lines of code, with Niklas Gustavsson

1. 0:00 – 0:10

   Intro

   1. NGNiklas Gustavsson0:00

      [on-hold music]

2. 0:10 – 0:57

   From “no one will use an IDE” skepticism to a new default workflow

   1. NGNiklas Gustavsson0:10

      I actually remember talking to you back in, I think, September last year, and you said something like, "Yeah, I don't think at the end of the year no one is gonna be using an ID." And in my head, I was thinking like, "That's crazy. That's never gonna happen." Like I, I could imagine that happening on a two-year timeframe, something like that, but two months seemed extreme, and then two months later, I found myself not using an ID anymore. And like the, the way that I was working had completely changed. It changed that I had not seen in the 30 years that I've been doing this type of work.

   2. SPSpeaker0:42

      It's funny. I-internally, it felt exactly the same way that it did externally.

   3. NGNiklas Gustavsson0:46

      Okay. [chuckles]

   4. SPSpeaker0:47

      But, you know, we had a head start of like a few weeks.

   5. NGNiklas Gustavsson0:49

      Yeah.

   6. SPSpeaker0:49

      But that, that was it, but it felt exactly the same way. So, okay, here I, I wanted to start with how did you get into coding?

3. 0:57 – 1:26

   Biology to software: entering programming through early “big data” genomics

   1. NGNiklas Gustavsson0:57

      My formal background is actually in biology, so I'm a molecular biologist by training. And in that area, when I was doing my PhD studies, um, we started having what was then considered big data. So we had a lot of data from, um, genome sequencing, so I felt that I needed to improve my ability to do programming essentially. So I switched over what was intended to be a sabbatical year, ended up being, I guess now close to 30 years of-

   2. SPSpeaker1:24

      [chuckles]

   3. NGNiklas Gustavsson1:25

      -being in this, in this industry.

4. 1:26 – 3:04

   Early “AGI moments”: automating code changes before Claude, then the Opus breakthrough

   1. SPSpeaker1:26

      So fast-forward to today with, with all the change right now with, with agents and LLMs, I feel like your personal usage and Spotify usage is on the frontier of what I see in the industry. What was, what was your first feel the AGI moment personally?

   2. NGNiklas Gustavsson1:44

      I think I have a-- I've had a few, depending on a little bit of the problem that we were trying to solve. We started pretty early as LLMs came about to try to use them to automate code changes, and that was a real struggle to begin with. But after a while, as we, as we started figuring out like how we can use LLMs and judges and whatnot, we started getting some pretty, uh, inspiring results from that. Uh-

   3. SPSpeaker2:09

      And this was, this was like a few years ago.

   4. NGNiklas Gustavsson2:10

      Yeah, it was pre- pre-Claude and pre... It was like early GPT days, something like that. And again, like the results we got then wasn't like we can fix all our problems, but it, it was giving an insight of like where this is heading in the future.

   5. SPSpeaker2:24

      Mm-hmm.

   6. NGNiklas Gustavsson2:24

      So that was certainly one. For-- I have to say for my own personal coding, the real breakthrough moment was probably Opus 4.5 back in November, December. It went from being this like smart autocomplete to something that I can actually throw real problems at, and I didn't have to do all that much prompt engineering.

   7. SPSpeaker2:44

      The biggest thing for me was also just not having to edit code anymore, 'cause my workflow up to then was I have the model write, you know, like maybe 80% of the code or 70% of the code, depending on the model, and then I always had to go into the IDE to do the last-

   8. NGNiklas Gustavsson2:59

      Yeah

   9. SPSpeaker2:59

      -two of my old edits.

   10. NGNiklas Gustavsson3:00

       Yeah.

   11. SPSpeaker3:00

       And I just stopped having to do that.

   12. NGNiklas Gustavsson3:02

       Right.

   13. SPSpeaker3:02

       And that, that was, that was crazy.

   14. NGNiklas Gustavsson3:04

       Yeah.

5. 3:04 – 4:18

   Niklas’s day-to-day agent setup: tmux, many sessions, and worktrees

   1. SPSpeaker3:04

      Um, but yeah, that-- I think that's a big part of the reason that it felt like such a leap. What's your-- So what's your workflow like today? Like how, how do you use QuadCode? How does Spotify use QuadCode?

   2. NGNiklas Gustavsson3:14

      Yes, I use it in a, I'm gonna say fairly vanilla way, I think. I run it in a bunch of Tmux, uh, sessions in a, in a terminal. Um, usually have a bunch of agents running in the background whenever I do some, some work. Um-

   3. SPSpeaker3:29

      How many terminal tabs?

   4. NGNiklas Gustavsson3:31

      So I will have anything in between five and ten tabs, uh, and then I use some panes because I like to have a terminal that where I can actually like get diff and whatnot. Um, so I have this set up with a matrix of Claude sessions and term- and matching terminals in a, in a set of, uh, work trees that I work in. The way that we're set up is that we have a f- uh, a few sl- very large monorepos, which we're gradually moving towards, but we still have thousands of small polyrepos for that, that remains. So I'm-- Most of my work happens in those, uh, monorepos. So I usually have a few Claudes and terminals going on there at, in a, any given point in time, and then when I need to dip into one of our polyrepos, I will open up a more temporary Claude session there.

6. 4:18 – 5:27

   Agents in a 20M+ LOC monorepo: surprising effectiveness at scale

   1. SPSpeaker4:18

      Do, do you feel like one like mo- monorepo or polyrepo is a better fit for, for Claude or?

   2. NGNiklas Gustavsson4:23

      I was a bit worried, to be honest, about the monorepo setup and agents originally because, um, I think with some of the prior tools we've been using, we've been seeing issues with indexing and things like that. Um, and this, these are fairly large repositories that our backend monorepo is more than 20 million lines of code. But turns out it-- Claude works amazingly well in those repositories and, um, I think one of the things we found is how good Claude is looking at other code, uh, in the repository to get, I guess, inspiration for the problem you're trying to solve.

   3. SPSpeaker4:59

      Um, I, I wanted to ask about some of the infra that, that you built.

   4. NGNiklas Gustavsson5:03

      Mm-hmm.

   5. SPSpeaker5:03

      So, you know, at, at Spotify, obviously you, you built Honk.

   6. NGNiklas Gustavsson5:06

      Yep.

   7. SPSpeaker5:06

      I feel like from the earliest days of experimenting with models to building Honk and building background agents on, you know, on, on the Agents SDK-

   8. NGNiklas Gustavsson5:16

      Yeah

   9. SPSpeaker5:16

      ... you see the future before other people do. What, what is it about the, the culture or the people working on it that kind of leads to this? And just tell me that story and how, how has it been going.

7. 5:27 – 7:12

   Why Spotify built “fleet management”: maintenance automation at company scale

   1. NGNiklas Gustavsson5:27

      Five, six years ago now, um, we identified that our code base was growing much, much faster than the number of engineers we had to support it, like seven times faster. So that meant that we, over time, we just had more and more code that we needed to maintain, uh, and Spotify is a company that has an endless source of ideas of things we want to ship to our users. So being bogged down by our maintenance was not a good place to be in. So we started automating, trying to automate as much of that maintenance as possible. A lot of that was pretty dull work, like migrating to the latest Java version or library update or whatever. Uh, a lot of it was moving from some API to some other API across, uh, all our code. Um, so we built out this infrastructure that we call fleet management, which is all about like instead of imagining Before that, when we were doing a migration, we would send out the migration description or like, um, tutorial to all our teams and ask them to do that migration manually for all of their components. And instead of doing that, we imagine like, can we find ways where we can do mutations towards our entire code base instead, living in thousands of repositories.

   2. SPSpeaker6:39

      Be-because every, every team was kinda doing the same thing.

   3. NGNiklas Gustavsson6:42

      Yeah, yeah. Hundreds of teams doing the same operation manually over thousands of components. So each of these migrations took months and months and months to complete. We could maybe do 10 of them a year, so we were barely keeping up with, um, being on the supported version of the frameworks that we're on. So again, we started automating this. We built out all of this infrastructure to do this. We've merged millions and millions of those types of PRs and, but they all relied on these like deterministic scripts that you would apply,

8. 7:12 – 8:43

   Hitting the ceiling of deterministic refactors—and turning to LLMs

   1. NGNiklas Gustavsson7:12

      and that would make those code changes or configuration changes. And one of the things we found pretty early was code has an enormous API surface, so trying to make changes to code gets very complicated very quickly. So we pretty quickly ran into a ceiling of how complex changes we can do. Even, even switching out the method and API becomes pretty complicated when you can call that in five different ways.

   2. SPSpeaker7:39

      So, so doing with this, with just traditional like static analysis, like AST transformation.

   3. NGNiklas Gustavsson7:43

      Exactly.

   4. SPSpeaker7:43

      Because like, let's say there's an API, you just like, you alias it to a variable or something, now you, now you need kinda like variable and state tracking.

   5. NGNiklas Gustavsson7:49

      That's exactly right.

   6. SPSpeaker7:50

      That's messy.

   7. NGNiklas Gustavsson7:50

      Yeah. So each script that we had to migrate code turned into thousands of lines of taking care of every edge case in that code. So that inspired us, as I mentioned before, as pretty much as soon as the early LLMs came along of like, "Hey, these things, can we apply them to this problem?" And early on, it didn't work at all, all that well.

   8. SPSpeaker8:13

      [laughs]

   9. NGNiklas Gustavsson8:13

      Uh, partially because the models weren't good enough, partially because we just, we were very naive in how we were trying to do it. We would basically just put the code in front of the model and try to get it one-shot that, that change. So that didn't work. Over time, models improved, and our thinking about how to do this improved. We started applying LLM sets judge to make sure that the output was as intended. We started breaking down the problem, decomposing the problem in various ways.

9. 8:43 – 9:45

   Honk evolves from migration bot to ubiquitous internal agent platform

   1. NGNiklas Gustavsson8:43

      So many, many, many iterations of this, uh, and many internal hacks to try to take on this problem in different ways. Uh, we started consolidating that, and that then became what we now call Honk. Um, it was a very different beast. Originally, it was not on top of Claude. Um, it was more a bunch of homegrown type of things in there, but it was the first sort of light in the tunnel of like, yeah, this is actually a problem that we can solve. And then we've done many, many iterations on, on Honk. Yeah, so today we re-we released what we call V2, but I think in reality it's V8 or something like that.

   2. SPSpeaker9:20

      [laughs]

   3. NGNiklas Gustavsson9:20

      We just didn't keep track of the, of the iterations we did on it. And it started out as this like automate these code changes, schedule that and orchestrate over all, all our repositories. But pretty quickly, engineers figured out that, hey, this is useful for other things as well. I want to mention this thing on Slack and have it do a task for me or, or all of those types of things. So today, Honk is, has grown into being a much more ubiquitous tool for us.

10. 9:45 – 10:01

    Honk architecture today: Agent SDK on Kubernetes, extensible tools, and CI verification

    1. SPSpeaker9:45

       Tell me about the architecture of Honk. Like how, what are the big pieces? So you talked about having, uh, uh, there, there's, uh, there's the agent that codes and this, this is just built on the Claude Agent SDK.

    2. NGNiklas Gustavsson9:56

       Yes.

    3. SPSpeaker9:57

       Um, and then you also have a, you have a verification step, like a agentic verifier. Tell me more about that.

11. 10:01 – 12:01

    From “judge agents” to stronger models: raising success rates and simplifying the loop

    1. NGNiklas Gustavsson10:01

       So we used to have a judge in Honk, but we actually have removed that because we found that the, um, uh, agent and models just, again, going back to four or five, got good enough that we don't, didn't need the judge anymore.

    2. SPSpeaker10:14

       Mm.

    3. NGNiklas Gustavsson10:14

       The judge was very important in the first iterations of Honk. So it, it made us go from, if I remember the numbers correctly, like roughly like 20, 30% success rate on PRs to like 80% success rate, so-

    4. SPSpeaker10:27

       Wow.

    5. NGNiklas Gustavsson10:27

       So it's a big, big change. But then again, as we talked about, the models caught up and, and, uh, the agent hardness caught up. So we have now eliminated that judge from, from Honk. So Honk architecturally is fairly simple, so it's the Agent SDK running in a Kubernetes pod. Um, it has access to a set of, uh, tools. Um, it used to be prior to V2 that those tools were a predefined allow-listed set of tools that we trusted to give to that agent. Now in V2, um, users can add their own tools, just auth those tools, so now the agent can use any of our internal tools. And one of the most important tools that it has access to is that it can run verification, like basically run CI builds, um, and it can run those both on Linux and macOS. So macOS is particularly important to us because any iOS development, for example, needs macOS builds.

    6. SPSpeaker11:30

       Mm-hmm. And is, is this just building or are you doing like a full like open up the iOS simulator, have the model like start the app? Kinda how, how deep does it go?

    7. NGNiklas Gustavsson11:39

       It, it can do those types of tests. We definitely have cases where we integrate the simulator and Claude to automate things like going directly from, uh, designs in Figma to UI implementations.

    8. SPSpeaker11:53

       Mm.

    9. NGNiklas Gustavsson11:54

       And we've been using that for porting, for example, our TV apps-

    10. SPSpeaker11:59

        Mm

    11. NGNiklas Gustavsson11:59

        ... from, from our iOS apps.

12. 12:01 – 14:45

    Verification as a forcing function: better tests enable auto-merge and autonomy

    1. SPSpeaker12:01

       I, I feel like verification is, it, it's one of these things that we talk about a lot.

    2. NGNiklas Gustavsson12:05

       Yeah.

    3. SPSpeaker12:05

       But, but I think when you're doing this kind of closed loop development where it, it's an agent that i-it's given a task and then it has to maybe like fan out and break down the task, and it, it just needs to do a lot of work without a human in the loop.

    4. NGNiklas Gustavsson12:18

       Yes.

    5. SPSpeaker12:19

       It-it's just the single most important thing.

    6. NGNiklas Gustavsson12:21

       Yeah.

    7. SPSpeaker12:21

       And I, I feel like one of the common mistakes I see is companies under-invest in how well that verification loop works.

    8. NGNiklas Gustavsson12:28

       I think that's very true, and I think it's true for us as well. One of the- Major change that we did in our, in our engineering practices as part of that was to strengthen our test automation. We have di-divided our co-code base into many thousands of components. Each of those components have a, a well-defined ownership, so it's owned by a particular team, and that team is fully responsible for that. They probably designed it originally, they implemented it, and they operate it. And part of that, prior to the investments we did in fleet management, was around, like, the, that team was in the loop for every change that got merged to their co- their code base. Uh, and that mean that, that meant that in some case we could be a bit sloppy on post test automation because that team could always check every PR if they needed to. But with starting to automate PRs towards our source code, one of the things was we needed to change the expectations for teams. Like, you might not no longer be in the loop for, for these changes. We're gonna be auto-merging most of these changes, uh, without you ever seeing the PR. So that meant then having to build out much better test automation to make sure that, uh, all our software could sort of s-survive those types of automated changes. Now s-zooming into where we are today, that's been very, very helpful for us because now we can throw agents at that and use the same, uh, verification that we had in place before.

    9. SPSpeaker13:52

       There's one of these trade-offs that people talk about all the time in engineering of, uh, reliability and quality on one side and speed on, on the other side.

    10. NGNiklas Gustavsson13:59

        Yep.

    11. SPSpeaker14:00

        And to, to me, it feels kind of like a false dichotomy because if you wanna go faster, the thing that you need to do is you need to automate your quality practices so that it's better encoded. It's not in someone's head, it's, it's actually like in a skill or in a Claude MD or in some set of MCPs. It's something that Claude can do.

    12. NGNiklas Gustavsson14:17

        Yep.

    13. SPSpeaker14:17

        And that's ultimately what, what lets you go faster. And th-this is just another example of how in engineering, productivity is always about investing in infrastructure. It's not about working more hours. It's about just making the infrastructure better and better.

    14. NGNiklas Gustavsson14:29

        Yep.

    15. SPSpeaker14:29

        And that sounds like what you're talking about.

    16. NGNiklas Gustavsson14:31

        We're seeing that we're keeping our quality metrics neutral while significantly improving our, our speed. Um, but that has not come for free. We've, we've needed to m- to make these investments into-

    17. SPSpeaker14:44

        Mm-hmm

13. 14:45 – 16:38

    Reliability at extreme deployment velocity: 4,500 production deploys/day

    1. NGNiklas Gustavsson14:45

       ... into test automation that we, as we talked about. Um, I think we're g- we're gonna have to continue our investments into, uh, our reliability practices as well. Some of those are changing as part of this-

    2. SPSpeaker14:56

       Mm-hmm

    3. NGNiklas Gustavsson14:56

       ... this transition as well.

    4. SPSpeaker14:58

       And, and I guess as you try to go kind of faster and faster and faster, you have to invest even more in reliability-

    5. NGNiklas Gustavsson15:03

       Yeah

    6. SPSpeaker15:03

       ... just in case.

    7. NGNiklas Gustavsson15:04

       Yes, that's exactly right. So yeah, so we make something like 4,500 production deployments every day.

    8. SPSpeaker15:10

       Whoa.

    9. NGNiklas Gustavsson15:11

       Um, so there's a lot [chuckles] of opportunity for things to go wrong. Uh, so yeah, we need to have good practices around making sure that everything that ships into production has the, the quality that we want.

    10. SPSpeaker15:21

        What's the idea with doing this many deployments? Is it kind of i-in the past it was just continuous deployment, and now maybe it's faster signal for the agent? Or how, how are you thinking about it?

    11. NGNiklas Gustavsson15:30

        This is something we've always been optimizing for, for as long as Spotify's existed, I think. We, we want to be able to basically have an idea and, uh, for a developer to have an idea and be able to ship that into production as quickly as possible. That used to be weeks or months back, back, um, back a few years, and we've just, uh, continuously tried to optimize that, and now it's, you know, an hour or something like that. Like, as I mentioned before, we have lots of ideas. We want to validate and explore those ideas, and the faster we can get feedback on that, and in some cases, that might be feedback from our internal users, in some cases it might be feedback from our, uh, external users. But in both of those cases, the faster we can iterate, we've found that we, um, we both build better products, and we're able to ship them faster to our users. Not every idea ship in an hour. Many ideas takes, you know, lots of exploration before we're able to ship them, but, but the notion of being able to, um, get that quick validation is super important to us.

    12. SPSpeaker16:35

        Mm-hmm.

    13. NGNiklas Gustavsson16:35

        And yeah, agents are certainly part of that loop as well.

14. 16:38 – 19:20

    Measuring ROI: PR attribution, cost/benefit, and linking work to user value

    1. SPSpeaker16:38

       And so for Spotify, the, the engineering org is fairly big. It's like, uh, thousands of engineers, right?

    2. NGNiklas Gustavsson16:42

       Yeah, it's 2,900 engineers, something like that.

    3. SPSpeaker16:46

       2,900 engineers. How, how do you think about, uh, as, as you do all this stuff, how do you think about ROI? Uh, like measurements, just making sure you're moving in the right direction.

    4. NGNiklas Gustavsson16:55

       In terms of measuring ROI, like we've been... It's been easy, and we've seen very, um, clear signals in that space. We're seeing a 75%-plus improvement in PR frequency, for example, uh, that we can directly attribute to AI tooling, and I think by now 73-ish percent of PRs are directly attributed to being AI authored. Um, so those types of metrics we're doing pretty well on. But then, of course, we want to connect that to user value and revenue.

    5. SPSpeaker17:28

       Mm-hmm. And how do you, how do you measure something like that? Is it sort of a, like A/B tests or some kind of holdout, like case studies? Like, how, how are you thinking about that?

    6. NGNiklas Gustavsson17:36

       Yeah, so we want to connect, basically be able to connect the deliverables that the engineering, engineers do, so PR's deployments, into, we call them work items, so basically like the, the planned work that we have.

    7. SPSpeaker17:49

       Mm-hmm.

    8. NGNiklas Gustavsson17:49

       And then that connects to, uh, A/B tests and rollouts, and then we're able to, from that, see like basically attribute back to, say, this PR contributed to this, uh, uh, DOD that we have, and that contributed to this user value. That's the idea, and we're trying to build those connections right now.

    9. SPSpeaker18:08

       Yeah. I, I feel like back in the day, you know, like y- we, we've worked in developer productivity for a while.

    10. NGNiklas Gustavsson18:13

        Mm-hmm.

    11. SPSpeaker18:13

        Like, when you have a big team, you wanna make them more productive.

    12. NGNiklas Gustavsson18:15

        Yep.

    13. SPSpeaker18:16

        And I, I feel like back in the day, a big win [chuckles] was, you know, like a few percentage points.

    14. NGNiklas Gustavsson18:21

        A single percentage point.

    15. SPSpeaker18:22

        Exactly. Exactly.

    16. NGNiklas Gustavsson18:22

        Yeah, yeah.

    17. SPSpeaker18:23

        If you were lucky enough to be able to measure that.

    18. NGNiklas Gustavsson18:25

        Yes.

    19. SPSpeaker18:25

        And like with, with the improvements nowadays, it's just so obvious to everyone, yet, you know, as engineers, we still want to measure it.

    20. NGNiklas Gustavsson18:33

        Yeah, I'm gonna say like the ROI discussion initially was fairly easy because we could see such large improvements. And, um, but as the maturity is getting there and the costs have been improving, I think the precision around those ROI estimates, the expectations on the precision is going up as well.

    21. SPSpeaker18:56

        Mm.

    22. NGNiklas Gustavsson18:56

        So that's why we're trying to improve how we can, how we can do that type of measurement.

    23. SPSpeaker19:00

        Part of it is about the improvement in productivity, and then part of it is how much does it cost to get that improvement.

    24. NGNiklas Gustavsson19:06

        That's exactly right.

    25. SPSpeaker19:06

        And now, you know, people are seeing these, like many dozens or hundreds of percentage points of improvement, and now you really want to attribute it to, to figure out like how many tokens did it take? How many hours did it take?

    26. NGNiklas Gustavsson19:17

        Yep.

    27. SPSpeaker19:18

        What was the productive output?

    28. NGNiklas Gustavsson19:19

        Yeah, that's exactly right.

15. 19:20 – 26:09

    Advice for leaders and engineers: standardize, embrace new roles, and prototype faster

    1. SPSpeaker19:20

       Um, I wanna end on, uh, maybe one question. What, what advice would you give your peers? What advice would you give to, to other CTOs and, you know, eng- engineering leaders like VPs of engineering at, at other companies?

    2. NGNiklas Gustavsson19:33

       What we've found is that these investments in foundational capabilities, we talked about test automation and verification. I'm gonna say the same is true for, uh, or another aspect that we've seen is, uh, standardization. So we've been driving, you know, more consistent code bases, more alignment on the tools that we use, the, um, frameworks that we use. And we've seen that this was originally investment we did to simplify things for humans and make humans more productive, but we've seen the same thing transition really well to agents as well. So if you have, uh, I mentioned this before on Claude being able to find inspiration from other pieces of code in our monorepos. If they look in 10 different ways, Claude is gonna be more confused. So we've been seeing the more consistency we have, the more, the better our agents work. So I think if, if there's one advice I would give would be to not, not ignore those types of investments. You need to have the same, the, the same, same engineering practices that we had before still applies in this new world. Might look different. The, there's a new actor being in your code base, but the fundamental seems to apply equally well, at least that's been the case in our, in our environment.

    3. SPSpeaker20:53

       What's your advice for engineers that, you know, maybe have been doing engineering work for a while? And I know Spotify has talked about engineers, you know, like shipping PRs on the subway-

    4. NGNiklas Gustavsson21:02

       Mm-hmm

    5. SPSpeaker21:02

       ... which is, which is really cool. So, you know, obviously engineering is changing. What, what's your advice to everyone that's, that's in the middle of it and, you know, trying to figure it out?

    6. NGNiklas Gustavsson21:10

       Yeah, let me talk about this from a more personal angle, I think. So I'm someone who's alwa- always have truly enjoyed the problem-solving part of coding. This is gonna sound as nerdy as it is, but like in my spare time, I will do like competitive programming at times because it's just like fun mental exercise. In the back of my head, I was always a bit worried, like we were talking about before, of like how this was change, completely changing the way we were working, and I was pretty worried about that from just my personal point of view of like, am I gonna miss that part of like the hard mental challenge of solving problems? And now I find myself having, you know, five agents working in the background, and my way of interacting with them is very different from the way that I was working a year or two ago. And for me, that's been turned out that I was wrong, and I like the, the thing that I like to do is solving problems, and the way that I solve those problems turn out to not be the most critical piece for me. This is always gonna be personal for, for different people are gonna have to make that transition in, in different ways. But I think focus on the types of problems that you're able to solve. Um, I'm, I find myself both to be more productive in that I can bring more value from the work that I did, can do. I can also solve problems that I really couldn't solve before. I can jump into code bases that I, that would've taken me days or weeks to get into before, and be, be contributing things that I just could not do before. So for me, that's been amazing. Um, and again, it's gonna look different for different people, but I think give it a shot and find the way that you, you can use those tools in the way that you like.

    7. SPSpeaker22:58

       I feel like for me, I've seen this big shift from implementation time, 'cause now, you know, Claude does it in the, in the background-

    8. NGNiklas Gustavsson23:04

       Yeah

    9. SPSpeaker23:04

       ... while I do other stuff.

    10. NGNiklas Gustavsson23:05

        Yeah.

    11. SPSpeaker23:05

        And instead, what's filled up that time for me is thinking about what's next, talking to customers, and also, like, actually much more prototyping than I expected, and some of it is for external products, some of it is for internal automations. How, how has that shift, how, how has that change looked for you?

    12. NGNiklas Gustavsson23:22

        I think it's been similar for me, and yeah, we didn't talk about this, but one thing that we're making a big investment in is, is prototyping in particular. Um, and this is targeted both towards, I'm gonna say engineers, but also the non-engineering cohort. One of the things that Claude and similar tools has unlocked is to allow anyone to take their idea, whatever that idea is, express that in natural language, and have Claude then go implement that. So as we, as folks started figuring this out, including, again, non-engineers, um, they started trying to do this in our real, uh, apps, and they're pretty complex beasts of code. Uh, but they were s- starting to see, again, like signs of light that they could do it. So we started a few months ago, we, um, basically built out the infrastructure to make that simple.

    13. SPSpeaker24:17

        Mm.

    14. NGNiklas Gustavsson24:18

        So today, we have a very simple way of getting going to build an end-to-end prototype in our, uh, mobile apps and our backend, and we have an internal app store for those prototypes where you can share them and like take a look at someone else, try out someone else's prototype in your, um, in your app.

    15. SPSpeaker24:35

        Mm.

    16. NGNiklas Gustavsson24:35

        And that's been a real unlock for folks that maybe before, and again, including engineers that maybe weren't super familiar with how to build something in our mobile apps-

    17. SPSpeaker24:44

        Mm

    18. NGNiklas Gustavsson24:44

        ... to be able to express ideas that used to take, you know, motivating a bunch of engineers to try to build that for you, and now you can go in and with the a- within an hour or two, you have a working prototype that you can start sharing with people to show what that actual idea looks like in real life with users' real data and, and so on. So yeah, those types of things are, were unimaginable a year ago, and now we're doing them every day.

    19. SPSpeaker25:14

        Yeah. I, I, I love that. Have, have you seen a, have you seen a shift in who's producing this? Is it, is it like-

    20. NGNiklas Gustavsson25:20

        Yes

    21. SPSpeaker25:20

        ... engineers doing it? Is it mostly coming, coming from designers and product managers? How has that changed?

    22. NGNiklas Gustavsson25:25

        It's everyone, up to our, uh, one of our co-CEOs, uh, have, uh, prototypes in that app store at the moment. So-

    23. SPSpeaker25:33

        Wow

    24. NGNiklas Gustavsson25:33

        ... it's actually been-

    25. SPSpeaker25:34

        Is it good?

    26. NGNiklas Gustavsson25:35

        Uh, yeah. Yeah, yeah. There's a bunch of, uh, like our senior execs have, have built prototypes that are good. Like, again, like ideas that they alway- always had in the back of their head. They have an entire engineering team that could build that out, but that team is focused on other things.

    27. SPSpeaker25:51

        Mm.

    28. NGNiklas Gustavsson25:51

        So for them to then be able to try something out more quickly than they could before and, you know, get a touch and feel for what this thing is gonna look like, yeah, allows you to test out an idea in, in a day instead of-

    29. SPSpeaker26:03

        Mm

    30. NGNiklas Gustavsson26:04

        ... weeks or months.

Episode duration: 26:10