How Intercom 2X'd engineering velocity with Claude Code | Brian Scanlan

1. 0:00 – 2:40

   Introduction to Brian Scanlan

   1. BSBrian Scanlan0:00

      Suddenly you started realizing that you have to think bigger about things, or that your imagination is now the barrier, not the tool.

   2. CVClaire Vo0:07

      How is this not happening in your organization? Like literally the physical limits of my ability to type code are unlocked by AI.

   3. BSBrian Scanlan0:14

      Today we are seeing twice the number of throughput as we did compared to nine months ago on our engineering team. Now it's like, why can't it be 10X?

   4. CVClaire Vo0:21

      This is a little bit more of what my instinct tells me is possible, which is if you go all in, if you prepare your team, if you prepare your code base, I think your overall product quality is gonna go up, I think your overall developer experience is going up. There's just so many good things that come out of using these tools and using them correctly.

   5. BSBrian Scanlan0:38

      Backlog zero is a realistic thing for teams to be able to go after. All the things that you wish you'd ever wanted to do, it's now just achievable.

   6. CVClaire Vo0:46

      I often advise a lot of CTOs and VPs of engineering when figuring out how to get their engineering team AI pilled, say, "Everything you hate about the code base, go spend a month fixing and see how fast we can speed run that. That's gonna feel really good."

   7. BSBrian Scanlan0:59

      I've been having the most amount of fun in my career over the last three months.

   8. CVClaire Vo1:04

      [upbeat music] Welcome back to How I AI. I'm Claire Vo, product leader and AI obsessive, here on a mission to help you build better with these new tools. Today, I am showing how Intercom 2X'd the number of PRs that their R&D department is shipping in just a few months. Brian Scanlan is a senior principal engineer at Intercom, and he is gonna show us truly all of their secrets to getting a large product and engineering organization cooking on Claude Code. Let's get to it. This episode is brought to you by Celigo. Every company today wants AI to improve how work gets done. The fastest way is building it directly into everyday business processes, automating employee onboarding, keeping customer data accurate, managing orders and inventory, or resolving finance and operations issues. When AI lives inside the flow of work, it can update records, trigger approvals, route work, and kick off the next step across systems. That's how teams operationalize AI and deliver measurable results. Celigo makes this possible, and now with Celigo Aura, it's never been easier. Celigo Aura gives you access to the entire platform through natural language, connecting your systems and turning intent into action, all of it under your control. Companies like Databricks, PayPal, and Olipop rely on Celigo to run critical business operations at scale. Ready to operationalize AI? Visit celigo.com/howiai. That's C-E-L-I-G-O.com/howiai.

2. 2:40 – 5:01

   Why Intercom went all-in on AI for both product and engineering

   1. CVClaire Vo2:44

      Brian, welcome to How I AI. Why I am so thrilled that you agreed to join the podcast is I think Intercom has done it, which is you all have met the moment in sort of two ways. One, clearly met the moment from a product perspective. We're one of the first companies that had... Sorry, I don't wanna say legacy business, but had a, a going concern business that saw AI coming and really transformed how your product worked for customers, and I'm a happy Fin customer. They did not tell me to say that. And then second, what we're gonna talk about is the team met the moment in terms of really understanding AI was gonna change how, in particular, product engineering and design orgs and engineering organizations were going to work, and you just went full speed at changing how the team works. What, what drove sort of the urgency around meeting the moment? How did that come to be? Was it a single person? Was it everybody? What was your experience?

   2. BSBrian Scanlan3:45

      I think it's... In some ways, it's been the easiest place to be driving out the adoption of AI in engineering and product, um, because we've focused the company so much on, or focused on product on adopting AI and being AI first in how we think about the product, future customer support, and all that. And we also had very clear expectations. Like, we, you know, we, we've seen what's possible in the product space, and it's just very clear and obvious to us as, like, connoisseurs of AI, it's like this is clearly gonna be huge in engineering and product and building. Um, and honestly, there's been a lot of impatience for, like, why, why isn't this happening today? You know, if we go back a few years and Cursor's picking up a bit of business, and the models are getting better and... But it still wasn't transformative. It still wasn't like the whole business was changed, and we're seeing vast amounts of extra productivity. We knew there was potential, but it still felt like we needed to have some sort of breakthrough moment or, or something needed to g- to big had to happen for us to get to the kind of huge velocity wins that I think now we're starting to achieve. That said, we still want more. You know, we're, we're proud of where we're, we're at, um, but we're, we're, we're, we're not content with, uh, with what we've

3. 5:01 – 7:02

   The breakthrough moment with Opus 4.6 and Christmas break 2025

   1. BSBrian Scanlan5:01

      achieved so far.

   2. CVClaire Vo5:02

      I feel like every three months I have a, a breakthrough moment, and in fact, I feel like Opus 4.6... I, I don't know, something just, like, really inflected in what was possible when that particular model came out. Now I think the GPT-5.4, uh, models are also exceptional, and so it's something about that one moment with models that really inflected my own personal use of AI in engineering. Did you all see the same sort of inflection around that model point?

   3. BSBrian Scanlan5:32

      Totally. I think you can go back to, it was like November, December last year. Uh, and suddenly you started realizing that you're, you have to think bigger about things or, or that your imagination is now the barrier, not the tool. You're spending less time massaging the tool to get it to the right place, um, and it's less about autocompletes and more about just literally giving us your, your ideas and seeing what happens. Uh, I think the Christmas break happened as well. I remember we, we had pretty much decided before Christmas like, "Hey, we're gonna go all in on Claude Code," 'cause up, up to that point there was a bit of Cursor here and there and Augment and different tools. Um, the Christmas break really helped. Like, we, uh... But I just saw everybody go wild on Twitter, X, you know? That, uh, people were, uh, talking about how this was... Like, they were getting so much done, and they were building all these things. I just come back to work after Christmas break going like, "Okay, everything's changed." [laughs] Like, we, we knew that there was something here and that we were starting to see the signs of it, but now the whole world is convinced, or at least all of the p- influencers on Twitter now like-

   4. CVClaire Vo6:32

      That, that would be me. Yeah, I'm, I'm actually kind of convinced that companies should increase their PTO and parental leave policies because everybody I know right now in tech that is, quote-unquote, "taking time off" goes on their vacation and pops open Claude Code and comes back, like, 10 times more skilled than they were befine their... before their time off. And so if anybody wants a, a little minor hack to AI literacy in your org, give people time off [laughs] to hack, and they will come back with more information

4. 7:02 – 12:50

   Demo: Intercom’s merged PRs per R&D head

   1. CVClaire Vo7:02

      than you expected. Okay, I think we're gonna skip to the punchline, which I love, which is we're gonna see how AI has actually changed how you all ship at Intercom. So can you just show us a little bit of how this has changed inside the org? And I think you all are measuring a lot of this.

   2. BSBrian Scanlan7:17

      Yeah. So I think we've been diligent as, you know, product owners inside of Intercom, uh, that we've been trying to, uh, get feedback from people and see how they're using the, uh, the tools and, uh, really, like, just doing everything, uh, that we would normally do with a regular product. Um, and so, uh, we've spent a lot of time hooking up Claude Code with telemetry, both into things like Honeycomb, um, and li- data also going into, uh, Snowflake, where we have our data warehouse. Um, we also store session data in S3. Um, and we mine this stuff for, for useful, uh, insights. And, um, one, one... But one of the main things that we use to drive adoption, uh, of the tool was, uh, our CTO, Dara, setting a goal of us two X-ing, like doubling the throughput of R&D. Uh, and we use pull requests as a crude, a simple measure. But, you know, there's, uh... And, you know, you can argue back and forth about what's a good measure, what's a bad measure, and whether measuring anything's appropriate or whatever, but I think it's reasonable to just have the expectation that if you can get a lot more done and it's so fast and fun, then why aren't, why isn't everyone just shipping more stuff? Uh, and so it's a basic measure that, like, the tools are being adopted, um, and that they're being used well. And, you know, of course, we don't tolerate lowering quality, and we're a high-trust environment, so we don't expect people not to game these stats or whatever. But our metrics, and just what I'm showing on the screen here is, you know, it's a classic number-goes-up, uh, kind of thing that where we ha- we started tracking this back at, like, how many, uh, PRs and, and what percentage of them were, uh, generated by either, uh, Claude or Cursor or whatever. And, um, yeah, since our major investment in Claude Code the platform, and going all in on it, and really pushing out, like, enablement and, uh, giving people, uh, freedom to explore and start to build skills and everything, um, but also pushing 'em on, on, on we expect kind of throughput, uh, increase. We've seen a big, big increase in, in the throughput, uh, of pull requests through our system. And, you know, like last year, like, our CI system completely broke. It melted. It, you know... B- but I don't mean it got like 10 times ex- as expensive and, you know, we did the work, we fixed the bottlenecks, we improved the performance of our CI system, and that stopped being the bottleneck. Um, now, uh, code review is our bottleneck. But, um, but like we're still... But today we are seeing, uh, twice the number of throughput as we did compared to nine months ago on our engineering team. Um, and, like, we're very proud of that and, you know, now it's like, "Why can't it be 10X?" [laughs]

   3. CVClaire Vo9:52

      So what I love about this chart, just for a moment, is I had spent the last two decades of my career in product and engineering, last decade of my career as a CPTO, and it's so funny. I wanna go back to a couple things you said, which is, one, you have to treat your org like a product. And I always thought that my job was not just the product strategy and the capital P product that we were delivering to customers. It was to design our organization to, I would say, like, output innovation on demand, which is... That was the job. And less romantically, or, well, then less romantically put, my job is to invest R&D for positive enterprise value. That was, like, fundamentally my job as a CPTO. And so what I love about this is it's merged PRs per R&D head. I'm presuming that includes... Does that include product managers and non-engineering R&D, or is that purely software engineers?

   4. BSBrian Scanlan10:47

      Yeah. This is all of R&D, and it's definitely the case that our designers and product managers and, uh, TPMs, like every role in Intercom is really actively using Claude Code on certain... And shipping code and all that. Um, and also, we've been hiring. Like, this number has not been static. So, uh-

   5. CVClaire Vo11:04

      Yeah

   6. BSBrian Scanlan11:04

      ... the number of PRs, uh, the raw number is, is dramatically higher than just 2X what it was a good while ago. So this is everything from your newest hire to, uh, your pro, uh, product manager who's, like, adding, uh, some copy or shipping, like, small changes or whatever. Um, you know, that's all, uh, based in this number.

   7. CVClaire Vo11:24

      The other thing I wanna call out for folks is every board meeting I have been in for the last three years have said, "How are we getting..." Well, f- actually, every board meeting I've ever been, period, has been, "How can we get more velocity out of R&D?" Certainly in the last three years it's been, "How is AI inflecting our velocity?"

   8. BSBrian Scanlan11:42

      Mm-hmm.

   9. CVClaire Vo11:42

      And it's so funny. I talk to so many people that are like, "It doesn't really inflect velocity. We're not actually becoming that more efficient." And I'm like, "Is that true?" Because I look at a chart like this, and I say, "This is a little bit more of what my instinct tells me is possible," which is if you go all in, if you prepare your team, if you prepare your code base, if you have, as you said, I think, a high-trust culture, people are gonna look at this and say, "Oh, they're shipping these smaller PRs," or, like, "Engineers are gaming the system." I just... I have not worked at a place that has such kind of, like, bad culture that that would actually come as an outcome of setting some sort of ambitious, fun target like this. And so I, I take this as at face value, and I think, how is this not happening in your organization? [laughs] In your org... Like, literally the physical limits of my ability to type code are unlocked by, by AI. You should get some inflection there. And so-You know, for VPs of engineering, CTO, even people that are on these R&D teams look at this and think, "You know, this is possible." And it may be a crude measurement, but it's, I think, an appropriate one as a leading indicator of, of what's happening

5. 12:50 – 14:27

   Agent-first work as a fundamental reimagining of technical workflows

   1. CVClaire Vo12:50

      in your org around AI.

   2. BSBrian Scanlan12:51

      Yeah. And we support this with not just telling people to move faster. Like, that's... But, uh, you know, we're, we're really looking from first principles of how to, how to do the work. Like, we believe that, like, all technical work will become agent-first. Um, and I'd like to set, like, a deadline for that, that, you know, at the end of the month we're just gonna go all in, and, uh, it's never gonna be the first thing that happens, uh, say, in response to an alarm or, uh, in a planning meeting that a- there isn't, like, an agent in there kind of doing the, the basic work. And I, I think that's a realistic expectation, but it, it involves not just... We're not just moving faster for the sake of it, it's m- we're seeing that we're moving faster by looking at the fundamentals of where we're spending our time and reimagining how that work could be done in an agentic world. And honestly, if, like, the, if the agents didn't get better, if the models didn't get better, the harnesses didn't get better, uh, we've got the building blocks just today to be able to just continue going, moving around, looking at how we do our technical work today. Uh, on a... By technical work, I mean everything in delivery of product. And, and move it to entirely be agents first, and l- allow us to move up to a higher level, to be able to, like, work on higher level concerns, or just getting more stuff built, more stuff out there, or higher quality. That's all within every org's grasp today. But you, you have to be very open to change. And I guess what's been fortunate at Intercom over the last while is that we have been extremely open for change, both in the product side of things and adapting the, the company to how, uh, I think companies need to work now, uh, with AI, and

6. 14:27 – 16:47

   The cost tradeoff: treating AI spend as an investment

   1. BSBrian Scanlan14:27

      we're starting to see results.

   2. CVClaire Vo14:29

      Yeah. The other, uh, r- other reflection I have upon looking at this chart is, uh, we're recording this in kind of the spring of 2026, and Anthropic just said that they crossed 30 billion in, [chuckles] in revenue, I think up from 19 a couple months ago. And I, I, I suspect their revenue chart looks a little bit like your merged PRs per R&D chart. So, how are you all thinking about the trade-off on cost here, right? Like, we're all consuming Claude tokens. Yes, you know, efficiency or output is going up, throughput's going up, but is cost scaling proportionately? Are you all worried about... Is that the problem right now? Are you even worried about it? How do you think about that?

   3. BSBrian Scanlan15:09

      Yeah. We're definitely worried, uh, in that the build is, looks exactly like this.

   4. CVClaire Vo15:13

      [laughs]

   5. BSBrian Scanlan15:14

      Um, and, um, you know, I've, I spent a lot of my career worrying about AWS costs and-

   6. CVClaire Vo15:19

      Yeah

   7. BSBrian Scanlan15:20

      ... uh, worrying about our margins and stuff. And then suddenly you've got these costs showing up, uh, a- and they're disproportionate to the growth that we've seen anywhere before. It, it's like hiring whole new offices of people.

   8. CVClaire Vo15:32

      Yeah.

   9. BSBrian Scanlan15:32

      Um, and, but at the moment our attitude has been, "Look, everyone just turn on Opus for everything, 1 million with, uh, context window." You know, we, uh, we just use the API plan, so it's all just on demand. And we, we think that there's enough, uh, alpha or benefit in, at this point, going as fast as possible and caring about the bill later because of the later benefits we'll get. And maybe that's the position of where Intercom is. I don't think it's realistic or feasible for absolutely every single business to do it. And honestly, I do kind of respect when you have to actually think about your token use-

   10. CVClaire Vo16:09

       [laughs]

   11. BSBrian Scanlan16:10

       ... and how that can kind of force you to be more considerate. Or it sometimes even gets you better results. You know, you don't need Opus for everything. Like, uh, there's, there's faster models out there. Um, and so we're, we've, we're just kind of avoiding that optimization phase until a point of where we're, we... Until, you know, we've gotten serious benefits from investments in this platform. And so I think this investment, and I think it's, it... We are treating it as, like, an investment at this point, um, is worthwhile. Uh, but, you know, if this keeps going at this rate, yeah, we should all work for Anthropic, you know? [laughs]

   12. CVClaire Vo16:42

       [laughs] Um, I think the way they're hiring, we're all gonna end up working for Anthropic. So

7. 16:47 – 21:22

   Measuring quality

   1. CVClaire Vo16:47

      okay. And then one other thing, because I think, you know, folks are gonna look at this, certainly engineers, and they're going, "Okay, like, you're shipping more PRs, but it's all slop, it's all garbage." Uh, you know, I know you all are measuring quality on the outside of this, on the other side of shipping all this stuff. So how have you seen this inflect your measurements around quality or customer value or what you're trying to achieve at the end, not just lines of code?

   2. BSBrian Scanlan17:11

      Yeah. I have a standalone graph that I can share, which is kind of interesting. Um, and so we've started to look at the, uh, the time it takes from the first line of code written in a feature to the time it gets posted on our news channel, like our updates. Um, and that's, uh, that, that has decreased consistently over the last few months. Now, we're not optimizing for this, uh, but we're interested in it. And the other thing is, like, the sheer volume of things we have shipped also appears to have kind of just rapidly increased in the last few months as well, and that's, that should be a bit of a trailing metric. So we believe that these numbers, this, like, increase in volume, is being borne out in real features, real products that our customers are using. And even we've been running some experiments, like how far can one person get on their own building, um, something that's plausibly a whole entire product area, um, feature to be able to sell. Uh, so this is something we're taking seriously. Um, and it's, uh, we also care a lot about quality. We've been working with a research group in Stanford. We've been giving them our data and, uh, you know, mostly just looking for any kind of insights to make sure we're not blind. You know, I join absolutely every single incident. I'm a ambulance chaser, and I make, like, uh... And I'm not seeing any increase in kind of regular kind of incidents or outages or customer-facing problems. Um, we've had a few kind of weird problems, but not related to production. Um, and, uh, but also, uh, the, the interesting thing from the Stanford data when we checked back in on it last week was that their measures of-Code quality reckons that the code quality was improving. Um, and, you know, the models are improving, the agents are improving. We're adding more and more guidance and skills, and all these kind of things, which I think do craft, uh, or do force people down a road which should result in higher, uh, quality output. But, um, it's great to see when t- when, when tools kind of can independently pull that out. Uh, now, devils in details. You've gotta go into the weeds. You've gotta actually really have a strong sense for what quality means in your own environment. But, you know, we're not seeing some of the things that people are worried about out there. Um, but that said, we've got a mature environment. We're a 15-year-old SaaS company. We've been doing this for years. Uh, you know, AI and speeding up your f- velocity will, will magnify all of your strengths and weaknesses. And thankfully, I think we've got a lot of strengths on the software delivery side of things that we've been able to take advantage of.

   3. CVClaire Vo19:41

      One thing that I wanna kinda call out here, which is you said that you've seen your code quality increase, which again, intuitively, I've always believed to be the ultimate endgame of this. And every engineer, not every, many engineers that I've talked to just don't believe it to be true. But when you have the capacity to take on tech debt, when you have the capacity to take on the dragons in, in your code base, you actually can do those things, whether it's developer experience, security and compliance, just general maintainability of your code base, flaky test, improving your CI/CD. All those things become very tractable, not just technically, not just can an engineer execute on it, but actually the business, and I feel like people don't appreciate this, the business, capital T, capital B, only has so much capacity for internal projects, meaning we can only allocate so much of R&D towards improving code quality. Just, just how we live. We don't generate ARR on code quality, unfortunately. But when the, the, um, when the cost of doing that compresses, then you're able to say, "Yes, as a business, we should invest there," one, because we can, and two, because it'll unlock velocity on the outside for our agents and for our product managers and for engineers. And so I think this is actually a really important moment for folks to invest in code quality, and I often advise a lot of CTOs and VPs of engineering when figuring out how to get their engineering team AI pilled, say, "Everything you hate about the code base, go spend a month fixing and see how fast we can speed run that." That's

8. 21:22 – 24:03

   Demo: Shipping a redirect in the Rails monolith with Claude Code

   1. CVClaire Vo21:22

      gonna feel really good. Okay. We've chit-chatted. We've shown graphs. The point of How I AI is to actually ship some code. So, let's switch over to that. We can probably come back to all these topics. I think they're so interesting. But you're gonna show us how you all, again, in your mature code base, mature organization, are actually getting things live, and some stuff you've done in the repo to make that possible.

   2. BSBrian Scanlan21:43

      Yeah, sure. So I'm going to do a fairly trivial change in our majestic Ruby on Rails monolith. So this is-

   3. CVClaire Vo21:51

      I love it

   4. BSBrian Scanlan21:52

      ... you know, millions of lines of code, all the tests. Uh, yeah, it's, it... The code base is older than Intercom. It, it was, uh, created before Intercom was incorporated. Um, and, you know, it's got, it's got its problems, but we love it, and we, we tend to it. Um, and so, uh, I'm just gonna do a relatively simple change of adding a, uh, a lobster emoji, Rails redirect to ChatPRD.ai.

   5. CVClaire Vo22:17

      Yeah.

   6. BSBrian Scanlan22:17

      So, uh, also, I, I try and give hints to Claude when I'm actually demoing something. Um, I don't know if it actually helps-

   7. CVClaire Vo22:25

      [laughs]

   8. BSBrian Scanlan22:25

      ... but it makes me feel better. Uh, just trying to add a bit of urgency here, you know?

   9. CVClaire Vo22:30

      I think that's everybody's prompting strategy, which is, "I don't know if it helps, but it makes me feel better." [laughs]

   10. BSBrian Scanlan22:36

       Totally. Um, so that's a nice way to, uh, interact with the agents, you know? Um, and so what we're seeing here is, I mean, it's already kind of figured out, [chuckles] uh, where to put a redirect. It's got the nice lobster emoji. Um, and it's asking me if I want to open a PR. Uh, so obviously I do. And, uh, I think it's actually gotten the URL wrong. I... It's app.intercom.com wh- which will have the URL, but we can tell, uh, Claude Code later on about that. So what we're seeing here is, first of all, an important point, so I'm just gonna scroll back up. One of the things we noticed early on when we started getting Claude Code to write all of our code, um, and you know, we're up well above 90% now, is that it would create pull request descriptions that were kinda terrible. They... It would describe the code, and that's the least interesting part of a pull request. You actually, as a human, or even as a, uh, an agent reviewing code, you want to know the intent behind the pull request. You want to know the interesting bits. What's kind of related to this? And, uh, you know, LLMs are very good at just regurgitating or rewriting code into English. That's fine, but it's not what we need. And so one of the things... And we noticed as well, when people were using Claude Code, we, we, we created an LLM judge to evaluate, uh, because we had suspicions that the quality of the pull request descriptions was going downhill. So we created an LLM judge to evaluate what does a good pull request... Well,

9. 24:03 – 26:33

   Creating a custom PR skill

   1. BSBrian Scanlan24:03

      we decided what a good pull request description, uh, should look like, and then got an LLM judge to go through, uh, all, like, months and months of data. And yeah, the trend was awful. The trend was going in one direction. Um, and this is bad. Um, and you know, look, humans aren't, uh, perfect at creating pull re- uh, pull request descriptions. Sometimes they're just blank and whatever. Um, but-I think with, uh, our use of tools like Cloud Code and setting up these kind of platforms around us, you really have to be pushing for, like, higher standards. You want as close to perfection as possible, and this was clearly something that we're just not gonna tolerate a lowering of standards our, in our environment. So we created a skill called Create PR, and what it does is it uses whatever context it can from the session to describe the pull request. Uh, so it's not quite rocket science, [laughs] um, but often the session knows exactly why it's doing the thing. And so, uh, but then we had to kind of force it in. You know, we, we started... We told people, like, "Oh, just use the Create PR skill," and then people would and wouldn't use it. You don't really actually wanna be, have people remembering things, so we added it as a hook. So if Claude decides to, uh, you know, use the GitHub CLI to open a, a pull request, uh, we just block it, and we say, "Yeah, tough. You need to use the Create PR skill, and also, you're probably gonna have to, uh, like, figure out a different text description. Uh, and then I might interview you if it's just not enough context there. Hopefully, uh, there's enough context in this." But the point being that, you know, this is a platform. We want great outcomes, and we measure the inputs and outputs. And after we, we put this in place, the LLM judge reckoned we're doing a great job now. And so we're, we're at higher quality pull request, um, descriptions now. Now, this is not the most important thing in the world. Like, this is not gonna get Intercom to 2X or to 10X revenue or anything like that, but it's the... all of the composite little jobs that, like, are... when you assemble, means you have an extremely competent engineer who works appropriately in our environment, and that's where we're putting our investment for each little skill and hook to do these things. So they look almost look inconsequential, but, you know, they result in better outcomes. And so if we look through here, it's, uh, it's creating a PR. I'm gonna have to check on what it's going. This probably will be automatically approved as well, which is pretty cool, and we might even see some pull request feedback as well in action. Um, ah, it's still building. We'll come back to it in a couple minutes.

10. 26:33 – 30:15

    Building a software factory with predictable quality standards

    1. CVClaire Vo26:33

       One thing I wanna call out for folks, as, as you were describing sort of why you put in this skill to improve the PR, and for those who don't know, um, a skill is basically just, like, a set of instructions and sometimes scripts that a LLM or a agent harness can invoke at a certain step in your flow. One of the things that I was thinking as you were describing why you put this skill together and got really opinionated about PR descriptions is in engineering, we have been able to architect really opinionated CI/CD pipelines, so how written code goes from being written to deployed in production. And we have... I mean, you saw it in GitHub. We have all these checks and lints and pre-deploy, you know, pre-flight things and preview branches, all these things once the code is written. But what I think is really interesting about skills is you can bring some of that determinism to as you write the code, how you want that process to go. And we used to not be able to do it because it used to flow through the hearts and minds and hands of humans, which are much harder to put in these structured guardrails. And we would do this by writing wikis or having, you know, SOPs where it said, "Can you please follow step A, B, C, D, E?" And now you can just make it really easy to enforce those standards across a team, which I don't think is micromanaging. It's actually just making everybody's golden path much smoother to production. And so I think there's this just very interesting, uh, parallel to how we've approached CI/CD to how we approach things more upstream, even from the product management perspective.

    2. BSBrian Scanlan28:10

       Totally. We're on this movement towards a software factory, and, uh, what factories are great at is, uh, you know, like an Ikea factory or something. It's all the same furniture, all the different bits, and you know how to assemble it. And, uh, look, it's not your artisan stuff. It's not, uh, or it's not cutting edge or whatever, but it's very predictable and, uh, you know, has a certain quality and meets certain standards when it comes out the other side of the factory. And so while pull request descriptions, again, they're not, they're not make or break for the factory or the, or the pull request or whatever, um, it's one of those qualities of just good quality work that's reliable, predictable, and then when assembled together, you've got your Ikea factory.

    3. CVClaire Vo28:51

       Well, and people don't wanna feel... Certainly engineers don't wanna feel like they're part of a slot factory, right?

    4. BSBrian Scanlan28:55

       Mm-hmm.

    5. CVClaire Vo28:56

       And so these things that you can add into the flow that actually uplevel and meet the standards of the engineering team really help your human engineers on the team feel like they're working in a place that values quality. And so I appreciate that you've put the, that effort into, um, into these behind-the-scenes hooks and skills because I'm sure it reinforces to a culture that's being asked to move very fast, to ship h- you know, ship things differently than they have before, that you still do care about their experience reading pull, pull, you know, pull request descriptions. Um, their, their, um, you meet their bar for quality, and I just think it makes everybody happier.

    6. BSBrian Scanlan29:38

       Yeah. Well, it's great when the robots just produce the work that you'd expect of your best engineers, you know?

    7. CVClaire Vo29:44

       Yeah. And I, uh, you know, maybe as you get this live, I also think there are just still such more interesting problems to solve in software engineering, and we can talk a little bit later in the episode about some of the interesting problems that you all are solving on the product-

    8. BSBrian Scanlan29:58

       Mm

    9. CVClaire Vo29:58

       ... side, on the technical side. I think there is no lack of hard, intellectually stimulating, creative problems to solve for customers, and coding redirects is just 100% not one of them.

    10. BSBrian Scanlan30:13

        Yeah.

    11. CVClaire Vo30:13

        Um, so did we get, do we get my redirect live,

11. 30:15 – 32:10

    Telemetry infrastructure: Honeycomb for skill usage tracking

    1. CVClaire Vo30:15

       or are we close?

    2. BSBrian Scanlan30:16

       It's still there. I'm waiting for an automatic review to kick in, but, uh, we can come back to it. So one of the things I would like to show next might be some of the telemetry that we have in place. So we saw that, uh, you know, there was different skills getting invoked and, um, and-We don't like flying blind. Uh, to run a system like this, you need to know how well people are using us. Uh, are people using these skills at all? Uh, you know, the kind of basic information that you'd expect of, like, when you ship a product to your customers. Uh, like, you know, where can I see the usage? How can I fight for the usage? What's, what's going wrong, or what's not going, uh, wrong? And so we collect a bunch of telemetry using different mechanisms and have different homes for it. Um, the most open one that we have is we collect, uh, basic usage information for skills and, and the like, uh, and we send it to Honeycomb. So we just have a shared key that's deployed to all of our laptops, um, and, uh, anyone can go in and kinda look through this data. So if you're developing a skill internally in Intercom, and, like, hundreds of people do this, um, it's very easy for you to go in to discover, like, "Hey, how, where are... Who's actually using this? Uh, when are they using it?" And you can kinda use this as a kickoff to, like, follow up on, um, uh, just, like, basic discovery of usage of your skills and all. And, like, unsurprisingly, the kind of main skills that we have are things like creating PRs. Admin tools is our admin, like, internal tooling APIs or, um, where we have an MCP in front of it. Buildki is our CI system. Snowflake logs is where we put Snowflake. So you can see from this, like, a lot of work, uh, or a lot of the skills that are being invoked are all around the building, and then seeing where my stuff is, and maybe some troubleshooting-type information as well. Um, and so this is the first kinda step. It's like if you don't have this, it's hard to have a large system, uh, like all these hundreds of skills and, uh, hundreds of creators working in this area without having decent telemetry.

12. 32:10 – 36:08

    Session data collection and personalized usage insights

    1. BSBrian Scanlan32:10

       The, the next thing we do as well is we also collect all of the session data and put it into, uh, S3. And so we anonymize it. We do a few things to make sure we're not doing anything too private. Uh, you know, people put all sorts of stuff in their sessions.

    2. CVClaire Vo32:22

       [laughs]

    3. BSBrian Scanlan32:22

       Um-

    4. CVClaire Vo32:23

       They yell at their sessions.

    5. BSBrian Scanlan32:24

       Yeah. Uh, uh, yeah, people have personal relationships [laughs] at times with, uh, with Claude. And, like, we don't really want to know about that, but we do want to be able to dive deeper into, uh, how things are going. You know, I think, um, understanding, like, how... What the dropout rate of sessions, like, did. How quickly people got to something useful, like, whether it was a PO or, or something like that. Uh, this kind of information's pretty interesting, and so we're harvesting a lot of session data, and we're doing different things. This is... What I'm showing here on the, on the screen is, like, a very simple tool that we put together, which just gives you some personalized insights. And, you know, you can do this inside Claude these days as well. There's... And there's plenty of skills out there on GitHub where you can do session analysis. But I think we've, we just built a little tool on top of our session collection, uh, system s- to give people feedback. And it's feedback that we're interested in giving feedback about how their sessions are going and, and how they're kind of fitting in, how you should think about your own, I guess, use of Claude Code compared to everybody else in the org. And, you know, I'm not doing too bad here. It's, like, 79th percentile.

    6. CVClaire Vo33:30

       [laughs]

    7. BSBrian Scanlan33:30

       Um, you know, someone has to be down at the bottom of every percentile. And, um, there's... And there's some interesting feedback here. Like, uh, it's tell- it's, it's kinda getting annoyed at me. Or rather, I was getting annoyed at Claude a few weeks ago because I'd set up Gog to interact with, um, all of our Google stuff internally. Um, and, uh, but it kept on trying to do the wrong thing, and I was kinda getting edge with, and ended up adding stuff to Claude and MD and stuff. Um, I... It's, it's kinda giving out to me here, or it's reminding me that this wasn't a very effective way to interact with, uh, Claude Code.

    8. CVClaire Vo34:04

       [laughs]

    9. BSBrian Scanlan34:04

       So, you know, it's a good prompt for me to actually go and fix up my memory or whatever. And, like, we all... Like, people are at different levels. Even at Intercom, um, people are at different levels of adoption. Uh, people are joining Intercom. They may not have, like, seen a system like this before, and they want to know how things are going and get feedback. And so, uh, this is one example of how we're just trying to pull together this information to give useful, uh, actionable insights, uh, to people so that they can... They feel supported, and that we're not just throwing them an API key and saying, "Best of luck." It's like, no, we've got... We understand what growth looks like and the progression that people go through, um, when they're using these tooling, um, and getting better and kinda self-improving, and we want to support all that. So this is one of the things that we're doing with the session data. There's loads of other things that's work in progress, um, like being able to... Like, we want to get insights to which skills are, are, are the highest quality, which gets you... Which, which skills get you to your results as quickly as possible, and then which ones need work. You know, which ones, uh, aren't working out so well and might need a bit of attention to improve.

    10. CVClaire Vo35:08

        This episode is brought to you by Cursor. If you all have been watching How I AI, you already know this. Cursor is my favorite way to code with AI. Whether I'm using plan mode to build out an ambitious feature, reviewing AI-generated diffs right in my editor, or kicking off cloud agents to multithread our roadmap, I reach for Cursor as my favorite multi-model coding platform. Even better than building myself in Cursor, I love collaborating with BugBot to fix PRs for code security and quality, and have begun relying on Cursor's automated agents to keep our code base clean. It's not just me. The most ambitious teams love Cursor too, including engineers at Stripe, OpenAI, and Figma. Ready to build more? We're giving $50 in Cursor credit to How I AI listeners. Claim your credits at chatprd.ai/howiai. That's $50 in Cursor credits by going to chatprd.ai/howiai.

13. 36:08 – 39:20

    Quick overview

    1. CVClaire Vo36:09

       I have to pause before we look at your list of skills, because I'm so excited about that part. But if folks aren't watching, it... They may have missed how amazing what you just showed is. So I'm gonna reiterate it, which is, one, you've instrumented all your internal skills with telemetry so that, and, and you're using Honeycomb. Um, love the Honeycomb team.

    2. BSBrian Scanlan36:30

       Amazing.

    3. CVClaire Vo36:30

       You're using Honeycomb to see how often those skills are invoked over time. So this is just a tip for anybody building out a skills repository internally, or even somebody who is maybe trying to get some visibility into their impact across the org. Let's say you build-A skill, and you wanna go to your boss and be like, "Boss, my skill is being used by literally everybody every day." Um, find a way to put event-level tele- telemetry invoked in the skill, a little dashboard, and you can track those over time. Again, treating your org like a product, treating your repo like a product, treating your AI setup as a team like a product. And all products, all good products, have tracking plans. And so figuring out how you put that telemetry in I think is really smart. And then the second thing for v- for those that missed it or how to do it, is you're taking all the raw session, I'm presuming JSON files.

    4. BSBrian Scanlan37:25

       Mm-hmm.

    5. CVClaire Vo37:25

       So for folks that don't know, Claude Code stores all your chats with Claude Code, um, in, on your computer in JSON, and you can go look at those or query those at any time. It sounds like you all-

    6. BSBrian Scanlan37:37

       Mm-hmm

    7. CVClaire Vo37:37

       ... are uploading those files to S3, and then layering on top of it some anonymization, some user-level views. And then you're essentially building sort of what I would call, like, an internal eval of how people are using Claude Code and what problems they are having over time, so that individuals, one, can triage their own implementation. As you said, "Oh, it looks like I need to do this or that, or improve my agent's MD." But then if you're seeing consistent themes over the organization on it's never invoking this MCP when we need it to invoke this MCP, or people are yelling no every time the create PR, um, skill gets queued up, you can fix that at a systems level. But you can't do that if you don't have the visibility. So again, my VPs of engineering, my CTOs, my friends out there, put some telemetry in your skills, and then do some meta-analysis on your Claude Code sessions across the org, and you'll be able to identify places where some, probably some high-leverage fixes are gonna get your team unblocked over time.

    8. BSBrian Scanlan38:42

       I do hope and expect that this stuff will get easier over time, you know. Um, uh, I, I'm happy to kind of invest the work, uh, so that we can move fast and kind of be on the bleeding edge. But there's something to be said also for being, for having, like, last mover advantage, and just getting all this stuff for free whenever Antropic ship it or whoever ship it. Um, I mean, maybe this is a product just that people should buy, uh, or build. Um, but for us right now, we've no choice. We just gotta build it. We're, w- we, we're, we, like, we're fascinated with the insights that are locked away in these sessions, uh, and so we just gotta build, uh, stuff so that we can see what's going on.

14. 39:20 – 42:16

    Walking through Intercom’s skills repository

    1. CVClaire Vo39:20

       I, I love it. Okay, can we see some of these skills?

    2. BSBrian Scanlan39:24

       Yes. Uh, so it's a very exciting GitHub repo. [laughs] Um-

    3. CVClaire Vo39:31

       Our lives are all GitHub repos and Markdown files.

    4. BSBrian Scanlan39:34

       Totally. Um, and we have, we have a lot of activity at the moment. We, we ran an AI day last week, kinda getting more people, uh, contributing to it. And so, well, what... So what this is, is it's a plugin, uh, repo, and, uh, we have a series of plugins, and they've been, they're growing daily at the moment. Um, kind of every team will have their own kinda specific plugins. And actually, in general, though, we're very liberal. We want stuff to end up in here, even if it's not great, and, but we do sweat the details on the core plugins, things that we think are fundamentals, foundational ones that go out to everybody. And so where we start off was we have, like, this base plugin which gets installed. Oh yeah, so we distribute this not via the Claude Clo- Code plugin mechanism. Uh, we found it was just a bit flaky. It was, you know, sometimes it'd update, sometimes it wouldn't. And it would, ended up kind of like trying to manage a Python install on, on hundreds of different laptops. You know, it's, you just don't wanna do it. And so we ended up using our internal IT systems to synchronize all of the plugins to the disks of everyone's laptops. Uh, so this is a great cheat code, and yeah, strongly recommend getting very close with your IT team to be able to deliver things like this reliably and not have to rely entirely on the Claude Clode Code plugins mechanism. Just in our experience, it's a bit flaky, and it just gives us a lot of reassurance. We don't have to do certain types of debugging once it's all on disk. So, so this is... We know this stuff works anywhere, uh, because we've got our IT team pushing it out to disk. Uh, and so we've got some safety hooks. We have some, uh, some of the ba- foundational things like, yeah, merging PRs. We don't want our agents going off into AWS. And then just different settings, and the telemetry things as well. So these are, uh, the core things that absolutely everybody gets, and, um, but we, you know, these are minimalist. We, we don't want anything that could be inappropriate in, say, a non-technical person's laptop or whatever. So, uh, that's, this is, like, the, the basic p- uh, building block. The n- the next main bit for us is, like, what we call developer tools. Again, like, this would be things that we then do all of engineering and beyond at this point. Uh, and these would be generally skills that would be appropriate to be used by any engineer in the course of their work day to day. And again, we would have a high-quality bar again for all of these. These would all require evals. These would all require to pass different kind of tests or analysis that we do on the quality of skills. Uh, and so we, we try and maintain these and make sure that they're well updated and well used, and we pay a lot of attention to, too.

15. 42:16 – 46:44

    Deep dive: The flaky spec skill and how it reached 100x capability

    1. BSBrian Scanlan42:16

       I can maybe go through one of these skills in a bit of detail. Uh, this one's near and dear to my heart. It's flaky specs. And I think the interesting part here is not the skill itself. The skill does reliably fix flaky specs. And I'll, I can pull up, um, in the meantime, like, here is a list of flaky specs that we have at the moment. I'm gonna open up, uh, the skill and just start to run it on this issue. And so while this is running, just walk through what's in the flaky spec skill. And so there's a checklist here. And-The, the fun part about how I built this was not that I b- b- was a world-class expert of fixing flaky specs. I roughly know the problem a- and, you know, have fixed a few of them in my time. But there's different classifi- in a, in a large test pe- te- test environments like ours, we have hundreds of thousands of tests. And if you're not super careful about, like, data poisoning or race conditions and all these kind of things that can kind of kick in when you're running millions and millions of tests a day, you know, you end up with these tests that end up slowing down your ability to deliver code to production fast and reliably, and not confuse developers by things randomly breaking. And there's kind of known patterns and known, known ways you would go about this. Um, but I knew my goal, which was to have a skill fixing all of these flaky specs. And it was something that agents are pretty good at when you give them a, a kind of testable goal. You know, this wasn't quite open-ended. And I also had this huge backlog. Or, yeah, there was a backlog of probably a few hundred, but then also all of this historical flaky spec information. And so you can just harvest all of this data in your environment to go, "Hey, Claude, I'm gonna build a skill. First of all, go and research every single flaky spec we've ever had, and then we're gonna build a checklist. We're gonna build a mechanism, and then we're just going to g- crunch through them over and over and over." And you get to this, like, 1X kind of, you know, it's doing a good job, probably as good as a job as I would do. But then as you keep building up all of these, like, little, teeny steps, which are the kind of things that, you know, our best Rails coders kind of do. They've got all the stuff in their head and all the different classifications of flaky specs and, you know, verifying against real data. Um, and, uh, and then, uh, but the f- the really fun part is then you get... So you get something that's starting to be, like, 10X. It's fixing flaky specs that I'm not even sure if I could do. Um, it might take me a day or something. Um, and I probably wouldn't do it. But then you start to add in, uh, stuff into the, the skill along the lines of, like, okay, when you fix something and it's novel, you need to update yourself as well. So in that session, it's updating the skill, so the skill itself is kind of learning as it goes along. And we also fan out. So it's like, okay, I'm very happy that you fixed that flaky spec. Now find every flaky spec that got impacted by that nature of it. And so I went from zero to, like, 100X in terms of this skill now is, like, you know, senior distinguished engineer [laughs]

    2. CVClaire Vo45:14

       [laughs]

    3. BSBrian Scanlan45:14

       Uh, level or being able to fix, uh, these specs. But it was more, like, the process that got there, um, and sort of, like, working with a feedback loop, working with, like, a very clear goal, and then giving it the freedom to do it. You know, giving it access to the systems where it needed to pull in metadata, being able to run builds itself, um, and having that feedback loop where it's learning. And, and then, you know, designing the skill as well so that it's... You have to edit it every so often. It get, it ends up taking up too much information that might confuse things. But then you break things out into, uh, like, reference guides, so you're doing this, like, progressive discovery thing. And, and I've even accidentally pointed this skill at, like, a Python code base, and Claude has just gone, "Eh." Like, "It's just Python. I'll give it a go." Uh, and it kind of uses the knowledge that's applicable to it. And so the, again, this skill is not going to make Intercom's revenue go 100X. Um, but it's now this, like, perfectly reliable thing that we really no longer have to think about. Now we can expand out into many, many different areas, and we're, we just have to maintain this, and the maintenance work for a skill like this just isn't much. And we have evals and stuff so that when we're upgrading models or maybe even moving to cheaper models or whatever, that we can make sure, yeah, this thing isn't regressing. It's still working as well as we think it is, and we've got confidence and certainty that this is still a reliable building block. And again, the constituent parts put, when put together, you've got, like, a very senior engineer who's able to get any work done in your environment. And so, yeah, uh, uh, we

16. 46:44 – 52:31

    The “and then” workflow for building comprehensive skills

    1. BSBrian Scanlan46:44

       can take a look at what it's doing. Oh, it's asking me for permissions.

    2. CVClaire Vo46:48

       [laughs]

    3. BSBrian Scanlan46:48

       Um, should've checked.

    4. CVClaire Vo46:49

       You forgot to date... Make no mistakes, dangerously skip permissions. That's the rule on How I AI.

    5. BSBrian Scanlan46:54

       Yeah. I-

    6. CVClaire Vo46:54

       One thing while it's running I wanted to say is, you know, this skill is a perfect example of what I call the, like, and then-

    7. BSBrian Scanlan47:02

       Mm-hmm

    8. CVClaire Vo47:02

       ... AI workflow. Which is, I tell everybody, like, pull your skills and pull your workflows through a bunch of and thens. So I wanna fix flaky, flaky tests. So I go to GitHub, I find a flaky test, I run through the skit. Let's say you fix it, and then what would you do? Well, I would document how I fixed it. And then what would you do? Well, I would go find all the other ones that are just like this and fix them. And then what would you do? I would go from, you know, a Rails code base to a Python code base and apply the same... The, you can just do that over and over. And because the cost of running these is so low-

    9. BSBrian Scanlan47:39

       Mm-hmm

    10. CVClaire Vo47:39

        ... you can actually pull the thread of a bunch of stuff any reasonable human would've quit at step one, because you're not limited, again, by head count or coordination cost. You're limited by the technical capacity to solve the problem. Which I think is, is a really interesting way to think about how you get from, like, the, you know, engineering intern that, whose job is to go through and take a first, you know, gentle pass at all these flaky tests, through to the distinguished engineer who has just speed run through 300 of them, and has thought of a completely different way to architect your, your testing overall in your repo. So I think that's a really great model for, for things. And then the other thing is, like, again, engineers, go speed run your tech debt. Fix your flaky te- like, these are all things that as somebody who has run engineering organizations, I have heard over and over, "We can't because our code base," blah, blah, blah, blah, blah. Like, "Can we pretty please allocate this amount of time to just fixing this really annoying front-end flaky test?" Like, uh, you don't have to ask permission [laughs] for that stuff anymore, because there's just a new way to solve it. And I think, again, just going back to some of the stuff we were talking about earlier, I think your overall product quality is gonna go up. I think your overall developer experience is going up. There's just so many good things that come out of using these tools and using them correctly.

    11. BSBrian Scanlan49:05

        Yeah, I think backlog zero is a realistic thing for teams to be able to go after. You know, all the things that you wish you'd ever wanted to do, you know, it's, it's now just achievable. I mean, of course you gotta balance it with, you know, all of the extra stuff that you can just deliver at the same time, but it's so sweet to be able to think that, hey, we actually have a path to getting rid of our, all of our backlogs, and all of the kinda architecture changes or whatever. You know, we, we can... Recently I was taking a Go microservice and re-implementing it in Ruby. And it was a single Claude Code session. Before November, this was something that I would've had to advocate for on a roadmap, and like, you know, plant some seeds in different engineers' heads, and kinda nudge people towards it, and kinda blame a lot of problems on the existence of this microservice.

    12. CVClaire Vo49:52

        [laughs]

    13. BSBrian Scanlan49:52

        Um, but now we're-

    14. CVClaire Vo49:53

        Wait, trigger warning first before you talk about that process. [laughs]

    15. BSBrian Scanlan49:58

        Uh, I'm sorry. I'm giving the secret sauce here of how to influence an org. Uh, but yeah, um, but now it's like, well, I don't even have to think about this now. It's a single session.

    16. CVClaire Vo50:07

        Oh.

    17. BSBrian Scanlan50:07

        And in fact, I can, I can get Claude to implement it five times, and compare the styles or compare the... You know, get us to review them and figure out what the best way of, of, of implementing the thing is. And this is just, like, this level of kind of creativity and freedom that where, like, your imagination is the blocker, not, not the, the time it takes to actually knock out one of these things, which was months in the past, you know?

    18. CVClaire Vo50:31

        I-

    19. BSBrian Scanlan50:31

        Yeah

    20. CVClaire Vo50:31

        ... I completely agree, and I, I feel this at ChatPRD, where people are like, "What are your..." I mean, I'm a product tool for product people. They're always asking what my roadmap is. Well, like, I literally don't have a roadmap. We burn down the roadmap every week, and then we figure out what we're going to ship next. And of course, we have thematic ideas we wanna pursue, and things that are larger. And one of the things that I do to keep myself from over-shipping absent product market fit is literally constrain the ideas to what I can do in my brain, [laughs] which is, there's, like, a natural throttle on not getting slop out, because it's not engineering throttling me, it's actually just good commercializable ideas. And I think that's where we're gonna see some of the limits start to come into play. I, again, referring to Anthropic, y- another big news piece came out is that they're hiring a bunch of PMs because they have so much engineering capacity, they're actually limited at the PM capacity. And so it'll be interesting to see where the bottlenecks in your business, you know, end up, and which bottlenecks are appropriate. It's probably good to have a product bottleneck a little bit, because then you're not shipping anything, um, which customers can't absorb. And so I just, I, I think it's gonna... And it's gonna evolve over time, and then, you know, product is gonna have a whole set of skills, and then, I don't know what we're gonna do with our time. Hang out on the beach. But I think it's, it's a pretty interesting time to, to run orgs.

    21. BSBrian Scanlan51:57

        Yeah. You know, I think engineers, designers, product managers, maybe it's just all gonna be one blob of builders-

    22. CVClaire Vo52:04

        Yeah

    23. BSBrian Scanlan52:04

        ... or something like that. Uh-

    24. CVClaire Vo52:05

        Let's do it

    25. BSBrian Scanlan52:05

        ... and everyone, everyone just does things. Everyone just does things. [laughs]

    26. CVClaire Vo52:11

        Yeah.

    27. BSBrian Scanlan52:12

        Um, and, uh, you know, that... It's great. It's, uh, it's, it's lowering the barriers to, like, just getting a lot of stuff done. And it's, like, so much fun when you can... When you don't have to ask somebody or pr- get something on a backlog or whatever. You can just get it done yourself, uh, or even just get it done very fast in a small group. Doesn't matter what your discipline is. It's just, like, a great

17. 52:31 – 53:32

    The live website and overview of workflows

    1. BSBrian Scanlan52:31

       leveler at the moment. So yeah, so we're live. I think our lobster is live, and-

    2. CVClaire Vo52:35

       Yeah

    3. BSBrian Scanlan52:35

       ... it should be on app.intercom lobster emoji. [laughs]

    4. CVClaire Vo52:38

       Look at that.

    5. BSBrian Scanlan52:40

       It's amazing.

    6. CVClaire Vo52:40

       I need to get you all an affiliate code, you know? [laughs]

    7. BSBrian Scanlan52:44

       [laughs] Uh, yeah, I mean, lobster emojis, they're, they're the new thing. They're the new, um, growth hack.

    8. CVClaire Vo52:50

       They are the new growth hack. Okay, so we have seen your PR per R&D employee go up. We've seen how you can get from kinda Claude Code to production very, very fast with a bunch of guardrails. We've seen your list of, it looks like hundreds of skills, but at least dozens of skills that you're invoking via hooks. You're using that to not only ship customer-facing product, but you're also using that just to make developer experience better, burn down tech debt, all those things we wanna see. You all are... You're measuring it, both from a telemetry perspective, um, both, like, quantitative and qualitatively. You're measuring your Claude Code sessions. And, you know, 2X isn't enough. You're gonna get to, to 10X.

18. 53:32 – 56:18

    How internal AI experience informs customer product decisions

    1. CVClaire Vo53:32

       So you all are on the edge, at least for, for folks that I talk to, and I'm sure you're like me, where you're like, "Sure, you think we're on the edge," but then I see people, and they're really on the edge.

    2. BSBrian Scanlan53:44

       Yeah.

    3. CVClaire Vo53:44

       So we always have ambitions to move forward. But my question now to you is, how has this impacted how you think about your customers' product? You know, I'm an Intercom customer. I'm a Fin customer. I interact with Intercom code and Intercom UI literally every day. My OpenClaw-

    4. BSBrian Scanlan54:02

       Mm-hmm

    5. CVClaire Vo54:02

       ... has an Intercom API key. How do you, how do you think about, you know, now that you have this experience with Claude Code internally, how do you think about what that customer experience is gonna look like?

    6. BSBrian Scanlan54:12

       Yeah. There's a few things going on. One is that people are outsourcing a lot of decisions to their agents. And, and like, this is a good thing in many cases, but, you know, there, there was good research done recently about what does Claude Code pick? And certainly I've had the experience in the distant past where I'd ask an agent to add something, except do it behind a feature flag, and then it would start to go and implement its own feature flag system. And this was-

    7. CVClaire Vo54:39

       No, no, no. [laughs]

    8. BSBrian Scanlan54:40

       [laughs] Um, this was in our code base, which has a pretty sophisticated old-school, uh, home-rolled feature flag system. So, you know, nowadays, it mostly will stick to whatever's in the code base, and that's fine. Um, but, you know, SaaS products, they're really good at their jobs. They're actually worth paying money for. And, uh, getting back to the feature flag, uh, situation, you know, uh, if you're building a new business, you're, you're, um, relying on your agent to make decisions, uh-Often an agent will, when prompted, it's like, "Hey, how should I solve a feature flag problem? I want to make sure I'm doing all these safe deploys and that." Uh, the agent will just go, "Yeah, I'll do it myself." And the kind of build over buy decision, uh, and you can see why the agents do it this way, because they can achieve this. They can get it done. They don't have to rely on the human. Okay, like, OpenClaw changes things here a little bit, and maybe computer use does as well, but still we're not re- we haven't really adapted, um, SaaS businesses to be agent-friendly. And that means, well, all sorts of things around, uh, how do we position our websites and content, and how do you get updated in their, in their knowledge, and how do they discover it? Um, but also, can they actually just get it done? Like, can you ask an agent, "Hey, could you just sign me up to Intercom and get me, uh, Fin working on my website?" And so, uh, like, this goes alongside just w- having to make more, more APIs for things. I think, I think, uh, I'm, I'm kinda like omni-channel as such. I think, like, there's a future for CLIs and MCP and, like, REST APIs. I think

19. 56:18 – 1:03:49

    Making SaaS products agent-friendly with CLIs and helpful hints

    1. BSBrian Scanlan56:18

       I'd, I'd like us to get more comfortable around things like ephemeral APIs or multi-step APIs. I think CLIs are good at wrapping these kind of things. Um, but the whole point of all this, where I'm getting at is, like, you know, you wanna be able to just help agents out at the, at the time, uh, when they're interacting, they're in discovery mode, and you want to give them clues, you want to give them hints, you wanna give them help to be able to do things like sign up for something fully without having to go back to the user and say, "Yeah, sorry, can't help you there. You gotta go away and, like, figure out how to sign up for something." Um, so, uh, uh, I've been working on something over the last few weeks which hopefully should solve that problem. And I can, I can paste in a, uh, a prompt and then see how far it gets.

    2. CVClaire Vo56:56

       I also, just while we're running this, I have to go back to your feature flag example, because it... You know where I used to work. It broke my heart that build it yourself was at the top of [laughs] the feature flagging list. But I do think y- I have, I have a, a paranoia moment about this, which is model providers and harness providers are highly incentivized to build it yourself. Consumes lots of tokens, versus [laughs] buy it, maybe consumes less. So I'm, I'm just really interesting to see how this all shakes out. You know, people, people are very anti-SaaS is dead.

    3. BSBrian Scanlan57:37

       Mm-hmm.

    4. CVClaire Vo57:37

       Um, and I'm a little bit more like, yeah, but, like, the current form factor of SaaS really is, has something coming for it, and a particular dev tools, because these models are so good at writing code, I think you're in a real, um, pickle to try to figure out how to find the right value edge at the right moment, how you can allow agents to not just sign up and set up things, but purchase it. You know, like, what does your trial experience look like if your first user is an agent? I think all of that is super important. And then, you know, to your point earlier where you said, you know, are we APIs, ephemeral APIs, CLIs, MCP? I think the answer is yes right now.

    5. BSBrian Scanlan58:22

       Mm-hmm.

    6. CVClaire Vo58:22

       Which is you cannot predict the, the medium by which a user is gonna come to your site. They could come through a search and hit your website and download things and look through your docs. They could come through Claude Code. They could come through an OpenClaw. You just really don't know. And so you sort of have to meet your customers and your non-human customers where they're at. And, um, I think it's really smart for teams that have any part of their product that needs to be implemented via code to be thinking about this problem yesterday, because you will be left behind, I think, if your agent experience isn't there.

    7. BSBrian Scanlan58:58

       Yeah. Agree entirely, and I think there's a whole craft in how to make, say, a CLI, like, agent-friendly. I think, like, MCPs obviously get that right, uh, a lot, a lot, a lot of the time. But, you know, for example, uh, one of the things that we do in, in the help is, like, kinda just give a hint to the, uh, the agent. It's almost like p- prompt injection to a certain extent, except it's not malicious. You're just trying to get it along to what it's trying to achieve. It's like, well, maybe you could check email, and if, if an agent has access to your email-

    8. CVClaire Vo59:27

       That's what I was looking at.

    9. BSBrian Scanlan59:29

       Yeah. [laughs] So it's, it's just there going, "Oh, you know, I can probably get this done." Uh, or, like, you can hint to them, like, uh, I've kinda cheated with this. So this is on my own personal website hosted in, in Vercel, and it is, uh, I've kind of pre-populated a few articles so they can upload, and Fin has some content to answer questions with. But you can also just, uh, you know, return in the, the help going, like, "Hey, you know, you should probably think about creating some articles if you want Fin to actually start cr- uh, answering questions," and that can be extracted from, you know, the code base or whatever. Well, uh, yeah, been like, like, I, I've been also think, like, a lot of interfaces, like CLI interfaces, like, I use GOG, uh, you know, as part of the OpenClaw, uh, universe. And, uh, I think it's, uh, a lot better than, uh, the official Google DWS one. And, uh, but I think if you w- start to use it, it's, it's actually just more human, um, as in it's, the interface just kind of makes more sense to a human. I think the Google one is like, I kinda get what they're getting at, and there's kind of JSON in there and stuff like that. Uh, it's not that, uh, uh, but it feels more human-friendly or something. Things that are effective for agents can often be things that are more human-friendly, because they're discoverable, all these verbs and words, and not just kind of inscrutable weird stuff going on, um, in command line options. I think I've c- confused Claude here. I'm not sure what, where this is-

    10. CVClaire Vo1:00:51

        That's, that's, that's okay. I'm gonna, I'm gonna narrate for folks what's happening here, which is you basically said, like, install Intercom on this site. There's an Intercom CLI that's like, cool, I can access the Intercom APIs and do a lot of this. My favorite part of it, though, is signing up, getting a verification email in your email address, invoking via, like, this hint basically of, like, if the user has email access set up in however you're accessing it, go check for this verification email, because we have, we have a code in there that we gotta snag, and because you're using GOG-Um, which is a command line tool to access Google Workspace. I- you, you can go do that, pull that code in. And what I think is so interesting about that particular flow is, you know, th- I, I think AI is creating sort of race conditions in shipping across the org, which is, like, you can YOLO a CLI probably faster than whatever team that manages email authentication can change how email verification works. And so you're like, "I'm not gonna let that break my product. What I'm gonna do is create a flow that I can, I can use that sort of sticky part in the flow, AI brains, and, and get through it." And so again, your product doesn't have to be perfect for an agent to traverse it. And this is one of the things I'm actually really excited about SaaS, is all those things that are just so complicated to do as a human, multi-step forms, and, like, nested fields on nested fields, and finding, you know, categories, and just those things that I would say UX designers and product managers have written their most tedious PRDs on and done their most detailed specs on. Like, you don't actually have to worry about making that, quote unquote, usable 'cause you can just brute force intelligence [laughs] against it and, and solve the problem. And so I think that's interesting because the core value proposition can get bigger and bigger without being constrained by the surface area of a website or a UI or any of those things. Um, and so I think if you're not thinking about what does that CLI look like for you and what adjacent systems does your product butt up against, it may be email, it may, may be at some other dependency, um, and how an agent might traverse those systems, you're just gonna get less and less adoption 'cause this is gonna be more how people install products.

    11. BSBrian Scanlan1:03:20

        Yeah, and if I don't poke holes, and if I don't re- make a CLI that kind of bypasses some of the ways-

    12. CVClaire Vo1:03:27

        Yeah

    13. BSBrian Scanlan1:03:27

        ... that product works, somebody else will. You know, they'll just put their own agents on it and they'll burn more tokens, they might get frustrated.

    14. CVClaire Vo1:03:33

        Mm-hmm.

    15. BSBrian Scanlan1:03:33

        And so you may as well shortcut them and give them an interface which just works. May not be the perfect interface, but that's the beauty of these things. You can get updated over time, you can... Agents can just pull down the latest version. Um, and yeah, like, hopefully I have something to show here, though.

20. 1:03:49 – 1:05:28

    Why conversion drop-off is invisible in agent-driven workflows

    1. CVClaire Vo1:03:49

       Well, the, the other thing that I, I wanna call out while you're talking about that, which is as I'm watching this, and it's taking some time to build, your conversion rate drop-off point is somebody pressing the escape button.

    2. BSBrian Scanlan1:04:02

       Yeah.

    3. CVClaire Vo1:04:02

       And just saying, "Forget it." Like, "This is clearly not working. What if we built it ourselves?" And so I think it's a really interesting moment for product managers who right now are not getting the visibility of the drop-off, right? When you were going through a website, you could put telemetry in it. You could say, "Okay, user's going to the sign-up page, drop-off. Email verification drop-off. Going to the docs drop-off." You could build this nice little funnel that identifies where your users are having problems. You can put some telemetry in your CLI, but at the end of the day, some of that drop-off and the alternatives is very invisible to you here, and the, the switching cost, quote, quote-unquote, is like pressing escape and saying, "Do it a different way." And so again, how quickly you can speed run to a zero to one installation in an agent, I think is something that everybody should be running right now. Um, and it doesn't just have to be a code product. Like, I think more and more people are doing non-technical tasks and interacting with non-technical SaaS in Claude Code, in Claude Cowork. And so i- you know, even if you're not dev tools, if you're not thinking about how can a user do this quickly in, in, in a third party harness or, or system, or an agent can do this quickly, um, you're really missing out on customer growth.

    4. BSBrian Scanlan1:05:26

       Totally.

    5. CVClaire Vo1:05:27

       Okay. How are

21. 1:05:28 – 1:18:43

    Lightning round and final thoughts

    1. CVClaire Vo1:05:28

       we doing?

    2. BSBrian Scanlan1:05:28

       It's on its fourth attempt. [laughs]

    3. CVClaire Vo1:05:30

       That's fine. And you know what? Let's, let's press, let's press the escape.

    4. BSBrian Scanlan1:05:34

       Yeah.

    5. CVClaire Vo1:05:34

       Because you know what? Let me tell you how cheap that exercise was.

    6. BSBrian Scanlan1:05:38

       [laughs]

    7. CVClaire Vo1:05:38

       It was, like, five minutes-

    8. BSBrian Scanlan1:05:40

       Yeah

    9. CVClaire Vo1:05:40

       ... and some tokens, and you're gonna spin up a fresh Claude code. You're... I don't know if you put make no mistakes. That was probably what we missed. [laughs] Make no mistakes. Um, and it, and it could have done it. And again, this is just learning. Like, why, why aren't e- e- why isn't every engineer, every PM doing this once a week or once a month just to figure out ho- how it can work? Um, I think this is great. So Brian, you've shown us everything. You've given us all, all the secrets. Let's get out of the terminal and let's do some lightning round questions. So my first question for you is how does it feel? Because what, what I observe from our conversation is it feels fun. Like, culture has in fact gotten better, not worse, because of this investment. And so, you know, as a company that has really put in the effort, both on the, on the customer side and internally, how do you think it's shifted culture? Has it at all? Um, what have you observed?

    10. BSBrian Scanlan1:06:43

        Yeah. Everything is just faster and more exciting. You know, I've mentioned feedback loops a good few times and, you know, you can just get stuff out there so fast now. And, uh, I've been having the most amount of fun in my career over the last three months or something like that. And like, it's, it's fun in many ways. It's fun because I can do stuff that, again, I would've had to convince other people to do, or I... They were just things on my wishlist and I could never get around to them, or I would just kinda complain about them. Um, but now they're just realizable. But also the fun aspect of, like, making other people productive, like leveling people up, getting... Like, removing work. I had, like, uh... Intercom has pretty good culture around resisting, like, the kind of slow...... movement towards being a large company and all this process and stuff like that. So we're kind of in denial that we're like a large company.

    11. CVClaire Vo1:07:35

        [laughs]

    12. BSBrian Scanlan1:07:35

        Um, but I think it's a healthy way to work in many ways. And, uh, uh, but this has kind of got us back to our roots in, in a lot that, you know, you, you can make fast decisions and get them delivered and get that feedback super fast. Um, and I've been able to, like, ship actual features, like not just the CLI, but I ship, shipped some web host features and, um, it's been a long time since I've done that. I'm just... I've been in the weeds in platform space for a long time. Um, and, but it wasn't even a big deal. It was like just a couple of hours, just kind of get something done. It was like something a customer asked for. So my job has become more varied. Um, I'm able to kind of see more and get more done, and help other people get a lot more done. So you get this ex- kind of excitement and velocity increases and, you know, we have all those measurements and that's all kind of good stuff, but just the excitement of waking up in the morning going like, "I'm gonna get a lot done today." Like, that is a fun way to go about your day.

    13. CVClaire Vo1:08:28

        I, I completely agree, and I hear this over and over and over again. I certainly feel it myself, which is this is the m- it brings me back to why I learned to, learned to code. It's like that same moment of, I didn't learn to code because I like to type code. I learned to code because of the magic of you running, like, hello world, and it, it shows up somewhere, and that feels so... It's just a very creative experience. Which leads us to my second question, which is, I see all the time that one of the most impactful change agents inside an engineering organization can be a senior principal engineer saying, "Let's go ham on some AI code." And the single most blocking person in the organization can be a senior principal engineer going, "I don't believe it. Absolutely not. Not me, not here, not... No way." And in fact, last week I heard a story of somebody who had their most senior staff engineer quit, says, and I quote, "I do not believe in AI. I will not work at a place that does this." So what is your appeal, sort of engineer to engineer, of, of why to invest in this, why, why you think it's the way that engineer organizations are moving, and how you kind of come to meet skeptics where they are, um, and hopefully see things a little bit from, more from where kind of Intercom is approaching them?

    14. BSBrian Scanlan1:09:44

        I mentioned that Intercom kind of had it on easy mode.

    15. CVClaire Vo1:09:47

        Mm-hmm.

    16. BSBrian Scanlan1:09:47

        Um, we didn't have to convince leadership that there's something to this AI stuff.

    17. CVClaire Vo1:09:51

        Mm-hmm.

    18. BSBrian Scanlan1:09:51

        Like we, we were pretty much... had decided the direction of the company the weekend that ChatGPT came out. So-

    19. CVClaire Vo1:09:57

        Yep. [laughs]

    20. BSBrian Scanlan1:09:57

        Uh, so we already had this expectation that this would be transformative across many parts of our work, including all of building products and engineering. We were just kind of mostly annoyed about how long it took. Um, um, but I think, uh, for sure it does need strong advocates, and you need to push, uh, boundaries. Like, one of the biggest things that I've been able to do successfully was kind of push through the barrier of, like, should we let, uh, an agent connect to Snowflake? Like, what, like... And there's, all these things can go wrong, or should we let our agent run real production code in, in our Rails console over API? And the easiest thing to answer there is like, "Well, you know, I'm not sure," or like, "This, this is risky," or, "We, we should think about this." But we've been largely pushing through it, and, uh, like not recklessly. Uh, like we've lots of good controls, and we're a mature business, and, uh, we have a, like, I've been on our security team. Definitely, uh-

    21. CVClaire Vo1:10:54

        Mm-hmm

    22. BSBrian Scanlan1:10:54

        ... not trying to do anything, uh, too wild. But there's still, even then I have apprehension. There's like, is this... Like, I think I, I think we should do this, but it seems weird or it seems hard. Uh, but then I just have to give myself permission, and then I realize if I have to give myself permission, there's loads of people out there who just need, need permission. And, um, honestly, like one of the biggest things I do at Intercom is just telling people they can do things. Um, this is, this is pre-a- pre-A-I, uh, post-AI, and, uh, or telling them like, "Look, whatever you do, just blame me if it all goes wrong." And I guess maybe we can blame, blame C- Claude now, but, but ultimately it's that, like, permission and just, like, th- there's a level of ambition which comes from it as well is like, if you, if you're out there saying, "I'm not sure if AI is going to take or have a big role to play in all of our work," and if you keep on saying that, that kind of will permeate through the com- cul- culture and people t- say that. But if you're very clear, you say, you're saying that like, "Look, all work is gonna be agent first, like at some stage in the near future, uh, and so we're gonna figure out the path there, and so we're gonna break down every barrier as we come across them. And look, it's your job, it's my job, and if anything goes wrong, blame me." Like, that's largely been how I've been approaching it. But not just me, like this has been a very large collective effort. But giving that kind of permission thing, but also the kind of, uh, like freedom to like explore or push things or whatever, it's kind of necessary. And look, it, it might be a, a, a less stressful way to go about it to like just take a nap for a few years and come back, and then when all the problems have been solved, uh, and we've got these perfect agents, uh, running amok in our environments, then, um, then that, that would avoid some of this. But like, I think all places have to get through that kind of apprehension and initial kind of issues that some of these can, uh, some of the introduction of agents in these environments can have. And I think our job as leaders, whether it's en- as an engineer or as a manager or whatever, just has to be on that, like, enablement and giving people space to, to, to go deep on the work, enjoy it, and, like, have that moment where things click and you start realizing like, "Oh my God, this is, uh, something that will transform how much I can get done."

    23. CVClaire Vo1:13:03

        Say it again for the people in the back. I lo- I was like, "Oh my gosh, I love this so much." And you know, I, it, it is absolutely those two things, which is like, give permission. You, you can. Please just go. Please, by all means, go ahead. Designer, hit me with a PR. No one's gonna get mad at you. Like, go ahead. And then the second thing of just accountability can roll to the top, and not in a scary way. Let's not do irresponsible things. But I, you know, we've seen a, we've seen a couple incidents in the past month, some big ones. S- And what you see is CEOs or big leaders coming out and saying like-The team's shipping, and we wanna keep shipping, and we're gonna be careful with our customer data, and we care for the customer experience. And stuff happens. We've learned from it. It's ultimately on me. I'm gonna call the customers, and we're gonna, we're gonna move on and deliver great innovation for you. And you know what I tell people to, you know, to get them over that hump, which is, like, you really gotta know what your existential problem is. And I love what you said is the second that ChatGPT came out, Intercom changed, 'cause that is an ex- existential problem. Who writes the code in your code base, agents or humans? Not an existential problem. Like, will you be fundamentally disrupted by a new technology? That is the real problem in your business. So I always tell people, like, let's differentiate the real problems in our business from problems that we can tolerate, and then go, go, go use the problems we can tolerate to move fast. Um, and so it sounds like you have a really good co- I mean, I think at the end of the day, the results speak for themselves. And again, you all are not asking me to say this. Intercom has met the moment. You went all in on AI-assisted, you know, customer support and experience. You're now building models. And so it's not just a one and done, ChatGPT is here, we need to change how our product works, or AI-assisted coding's here, so we need to change how our engineering team works. It's, you know, models are gonna be how people differentiate. We need to go there. CLIs are gonna be how people use products. We need to go there. And so I think this sort of, like, fearlessness and what I would suspect is, like, just a fun, nice, high-trust culture, good people. Uh, you actually see the business results on, on the other side. So I'm gonna hype you up. I see a lot of teams. I see [chuckles] a lot of leaders. Um, and I think people can take a lot of inspiration from this. But let's uninspire them really quickly before I get you outta here, which is my last question.

    24. BSBrian Scanlan1:15:28

        Yep.

    25. CVClaire Vo1:15:28

        Which is when, um, Finn takes 15 solid minutes on a li- live podcast to do a very basic task that you know it can do. Or not Finn, when, when Claude Code.

    26. BSBrian Scanlan1:15:39

        Yep.

    27. CVClaire Vo1:15:40

        What do you do? Do you yell? Are you a yeller?

    28. BSBrian Scanlan1:15:43

        Um-

    29. CVClaire Vo1:15:44

        What does your, what does your meta-analysis on this internal dashboard say the human needs to improve on?

    30. BSBrian Scanlan1:15:49

        I, I, I do lapse into giving Claude Code, like, uh, just, like, smiley faces or unhappy faces or, you know, not over the top. I, I certainly haven't cursed at it. Uh-

Episode duration: 1:18:45