How Ramp engineers work with AI agents at every step

1. 0:00 – 0:32

   "Fix all our import cycles"

   1. RSRahul Sengottuvelu0:00

      I told Fable to fix all our import cycles. I also told it to make our app lazy, so the app boots up, and it's an enormous amount of Python modules. And Fable did-- made a lot of progress in both of these. A lot of this code was merged. Really understanding what the boundary of where these models break is very important because those sets of problems are the problems that we would want our people to try the next Fable comes out or the next release comes out. [upbeat music]

2. 0:32 – 1:33

   Stress-testing Fable on Ramp's Python modules

   1. RSRahul Sengottuvelu0:32

      One of the first things I tried when I got my hands on Fable was I wanted to find a use case in Ramp that we can empirically verify, so it must be a large piece of code that I haven't read, that Fable wrote. It must be in a non-product place, so something in CI. So this is for us in our testing suite, and I wanted to empirically verify it, so be able to produce a lot of data and check if the code is acting like it should. It's been working really well. We've been running it for a few weeks in shadow, and it's, it's been consistently faster than where our current implementation is. But I, I, through the deep critical problems in our code base, so we have this large monolith Python code base, and I told Fable to fix all our import cycles. I also told it to make our app lazy. So again, we-- the app boots up, and it's an enormous amount of Python modules. And Fable did-- made a lot of progress in both of these. A lot of this code was merged. Really understanding what the boundary of where these models break is very important because those sets of problems are the problems that we would want our people to try the next Fable comes out or the next release

3. 1:33 – 3:36

   Fable and dynamic workflows cut CI time 66%

   1. RSRahul Sengottuvelu1:33

      comes out.

   2. BOBoris1:33

      I have these tests that I give every model when they come out, and I just sort of... You know, no, no model ever just was able to do all of these things, but this is the first model that just did all the things. And when Fable struggled, I just used a dynamic workflow. You, you guys have used it. This is, like, where Claude has a bunch of sub-agents that it orchestrates in using this sort of, like, algebra in the sandbox. But essentially, it's like a new form of test-time compute. And so you just tell Claude, you know, "Use a workflow," and Fable does it. Just yesterday, it actually reduced our, uh, CI time from, I think, eighteen-minute P fifty to six-minute P fifty.

   3. ARAustin Ray2:08

      Wow.

   4. BOBoris2:09

      And it just, like, optimization after optimization after optimization, and they just kept profiling it. The code landed, then it waited a day and used a routine to schedule itself to run a day later to get that real production data, and then it just, like, repeated this for days on end until it landed all these wins, and then it showed me a chart when it was done.

   5. ARAustin Ray2:25

      Rahul loves that. He's a huge CI time m-minimization advocate.

   6. RSRahul Sengottuvelu2:31

      It's all I think about.

   7. ARAustin Ray2:32

      I just have a follow-up question for you. So when, when you're doing these, uh, dynamic workflows, I, I would have to expect that you've already built up the familiarity with Fable as a main agent, like a foreground agent, right? Because there's, like, a lot of learning from what the forty or a hundred different agents are doing in the background that maybe you're not seeing, but you, you do have to have comfort by that point because that's a big workload.

   8. BOBoris2:59

      Yeah, 'cause the, the agents are, you know, like, Fable orchestrates it, so the agents might have, like, a work trace solution. They might not. They're executed, you know, in parallel or serial. You might add more rounds of, like, adversarial verification or whatever it feels this task needs. So yeah, I just trust it. Essentially, my mental model for dynamic workflows is the test-time compute dial goes from, like, low to medium to high to extra high to max, and essentially the way to think about this is this is the maximum amount of thinking the model can use. It won't always use it, but it's the maximum.

   9. ARAustin Ray3:27

      That's, that's why you have it on the thinking dial at the very end-

   10. BOBoris3:30

       Yes

   11. ARAustin Ray3:30

       ... right?

   12. BOBoris3:31

       Exactly. And for, and for me, dynamic workflows are just like it's the next level. It's like another form of test-time compute. It's

4. 3:36 – 5:15

   Loops vs. dynamic workflows for long-horizon tasks

   1. BOBoris3:36

      new.

   2. ARAustin Ray3:36

      How do you think about loops versus dynamic workflows for achieving some sort of long horizon thing?

   3. BOBoris3:43

      So loops are kind of like repetitive work, and dynamic workflows are like dynamic work. Like, you don't, you don't exactly know what the steps are ahead of time. Like, I, I use a loop, for example, for babysitting my pull requests to, like, fix CI and rebase them automatically.

   4. ARAustin Ray3:57

      Right.

   5. BOBoris3:57

      But then I use dynamic workflows for things like system optimization, where you don't necessarily know what the next optimization is.

   6. ARAustin Ray4:03

      It's a total mental model shift. It, it, it does feel like using Claude Code for the first time, where you have to start relinquishing a lot of the software engineering workflow, including running commands to this agent, right? [chuckles]

   7. BOBoris4:16

      I, I sort of feel like for me loops is, um, sort of if you have a bunch of engineers doing work, loops are kind of slicing a horizontal off of it. Like, if there's one task every engineer does every day, you can maybe take that and put it in a loop or in a routine, and this is something like a code review, babysitting a PR, addressing, uh, feedback. Uh, you know, like, we just have dozens of these. Uh, like I have one, for example, for deleting dead code. This is a, this is a routine that runs every day. And then on the flip side, you can do this vertical slice, and for us, this is like Claude Tag.

   8. ARAustin Ray4:46

      Mm.

   9. BOBoris4:46

      And, you know, an example of that is I'll, I'll have Tag, uh, ship an experiment. It'll make the experiment. It'll land the PR, and then it'll set a reminder for itself using a routine to monitor and check in the next day. It'll make sure the expo- the exposures are balanced.

   10. ARAustin Ray5:00

       Yeah.

   11. BOBoris5:00

       It'll crank up the exposure, make sure the experiment's running, and maybe a couple weeks later, it'll be like, "All right, I'm gonna ship this variant," and it puts up another PR for that, and I wasn't in the loop at all. Like, at the beginning, I asked Claude to do this. I stamped the pull request, but the rest was just Claude. I

5. 5:15 – 6:49

   Claude Code setups: vanilla vs. background-heavy

   1. BOBoris5:15

      wanna, like, detour a little bit. I wanna hear about your coding setups.

   2. ARAustin Ray5:18

      So iTerm2, pretty vanilla. Um, no IDE these days, uh, and as many panes as my monitors can handle. Pretty bare bones Claude Code setup, so not a lot of plugins or skills or MCPs. Pretty simple claude.md inspired by your, you know, your Twitter post for the vanilla setup. I think it's the best way to learn the models. Um, and then good amount of sub-agent use, adversarial review.

   3. RSRahul Sengottuvelu5:48

      Yeah, my mind's gone increasingly background-heavy, and almost most of my sessions are information-gathering. It's like, why is the memory spiking on this service? Or how can we get this project done faster? And it's, it's a great way to, like, fan out a lot of sessions and gather a lot of context. When I do loc- things local with Claude Code, it's usually maybe more hands-on programming where it's closer debugging or I need more services or, uh, context on my, on my computer.

   4. BOBoris6:14

      Has that changed a lot over time? Like, did you, did you start with a sort of- You know, like an Austin setup of like just terminal, terminal, terminal, terminal, and then you kind of move to this

   5. RSRahul Sengottuvelu6:22

      I mean, we, we have so many services, a lot going on, databases and, and message queues and Redis and all that. And so having multiple instances of local dev running can become a constraint very quickly. Uh, especially with the, the latest models, I, I think they, they require much less hand-holding, and sometimes you just gotta let them cook and get out of the way. And so I, I found myself carrying my laptop with the lid open a little bit too much, and so then we, we decided to move.

   6. BOBoris6:45

      It's funny, I know exactly what you mean. [laughs]

   7. RSRahul Sengottuvelu6:46

      [laughs]

   8. ARAustin Ray6:47

      Yep. Make sure Caffeinate is

6. 6:49 – 7:23

   AI agents across the engineering lifecycle

   1. ARAustin Ray6:49

      running, right? Yeah.

   2. RSRahul Sengottuvelu6:49

      Yeah. So we've implemented agents in pretty much every part of our business, but especially in the engineering lifecycle. So if you take the process of building and shipping software, everything from coming up with ideas, figuring out what the bugs are, getting notified, uh, when there's problems in our logs, in our systems, to writing the code, uh, to, uh, reviewing them, and sometimes when they're, after they're deployed, uh, looking for how they're doing in production, seeing if they're doing the thing we want. We've tried to build systems that, uh, along this whole stack. We've also thought about it from the lens of security, trying to find bugs and other issues.

7. 7:23 – 9:11

   Building for future models, not today's

   1. BOBoris7:23

      Okay. So, so now, now you're at the point where you're using, you know, the model kinda everywhere throughout, like, the whole lifecycle. How did, how did you get there? What was like ... What was the first place where you started using Claude Code, and then how did agents kinda expand out of that?

   2. RSRahul Sengottuvelu7:38

      We were seeing and slowly realized, like, clearly this thing's gonna, gonna continue to improve, and maybe we shouldn't build for 2.7, maybe we should build for whatever's coming next or the model after that. And over time, we've, w- as we built these harnesses, we've learned to step back and just wait it out because a lotta time we end up removing the scaffolding over and over again because the model's just, like, outgrown the harness. At any given point when there's a shortcoming with the harness or the model, we've tried and have ... We're not perfect. We, we tr- we als- also need to make the product work today because otherwise we won't have a business, but we've tried our best to go the other direction and give the model more tools, more context, more agency, with the goal of almost, uh, being able to treat our agents like a coworker. So, "Hey, can you go figure this out, this out? Like, there seems to be some sort of exception that's popping up," or maybe this customer is complaining of, like, a sm- a certain issue. Uh, and we want the models to be able to access the right systems, the l- right level of access, and, and produce the right amount of co- right code. And so just wanting that simple goal allows us to, uh, like allows us to, like, figure out what we need to do to give the model enough access to do these things.

   3. ARAustin Ray8:48

      I think it's a, it's a velocity bet in a lot of ways, right? 'Cause you're, you're, you're basically saying, "I think the stuff we would put in place to make this work now really well is, is gonna become technical debt really quickly and, and that's gonna slow us down. If we aim a little further in the future or sometimes a lot further in the future, you know, we'll actually make it further with, with the resources we'd have."

8. 9:11 – 12:00

   AI agent guardrails and least privilege

   1. ARAustin Ray9:11

      Yeah.

   2. BOBoris9:12

      So many questions, but maybe, like, one direction we can take it is how do you make sure they have the right guardrails? Like, you know, like they, they can access this data but not this data, or how do you make sure the cost is under control? How do you make sure the code quality is good? And how, how have you guys thought about this, like, as you scale up the systems?

   3. RSRahul Sengottuvelu9:29

      Yeah. So we've also at various levels on the stack tried to implement safeguards. We also study the trace a lot. One of the things that I think we've tried to focus more on is studying individual traces and less on aggregate level benchmarks. Benchmarks do give us a lot of information cross-model, but a lot of the time there's a usually a correct trace. It's like, what is a command the model should have run in this scenario, and why did it not get there? Is this a context issue? Maybe it does not have access to the right tool. And just following these simple traces for workflows that should work allows us to, like, get, get there in the right way. We've implemented a lot of layers of defense across ... I mean, we're, we're continuing to do that. It's how, how many layers we have also allows us to move faster and give it more agency and more access. Uh, and so again, at every part of the stack, we've done everything we can to give the model what it needs, but nothing more.

   4. BOBoris10:18

      So you're essentially like you go to BigQuery or Datadog or whatever and you, you, you give it like a read-only service key. This is essentially how you think about it?

   5. RSRahul Sengottuvelu10:26

      That's right. Yeah, exactly. So I just wanna be able to say, like I, I talk to my agent like I talk to my coworker, and so we're almost focusing on the default experience, the iPhone experience, where you just, you open it up, there's a text box, you just say what you need to get done, not how to do it. The prompts must be declarative. Uh, so we- we're not, w- we don't want people to instruct the agent to do it in a certain way. We just want people to say, "Implement this feature," or, "Fix this bug," or, uh, "Help this person out." And over time, especially when you focus on the correct trace ... Okay, so what's must the agent do? It must first query this source, and then it must query these other sources and read the code in this, in these repos. Uh, just by focusing on what the correct trace in your head is, you can then shape the, the agent trace purely through prompts and tools and skills to get there. And thankfully, we're also on this exponential increase in model capabilities. So maybe if it's not working right now, just gotta trust that it will get there, and just with that belief alone, just, just ship it and just wait.

   6. ARAustin Ray11:26

      Uh, the one thing I'll add is g- good old-fashioned hard con- controls on top of that, like you said, like principle of least privilege stuff, the, the basics of just not even giving it the opportunity to be able to do certain things.

   7. BOBoris11:38

      Yeah. And, and how do you think about, like, enforcing it? Is it, is it, like, the security team's job to do this, or are you, like, federating out the design of these sort of systems? How, how do you think about that?

   8. RSRahul Sengottuvelu11:47

      Yeah, the really exciting part about this is the infrastructure has been built by the security team, and it's been ... And security team's very closely, uh, related to this. So they helped us set up the network access policies. They help us get the keys, and they're also

9. 12:00 – 13:08

   Cost controls and AI code review

   1. RSRahul Sengottuvelu12:00

      regular users of these agents.

   2. BOBoris12:01

      How do you think about cost controls? How do you think about code quality? Um, what else do, do you think about as, as you scale it?

   3. RSRahul Sengottuvelu12:08

      We're continuing to find and look for cases where, again, we can guarantee, we know for sure the worst thing that could happen if, for example, this code has a bug or something like that, the, the effects are extremely constrained, and we do have an upside, and we're, we're finding more problems like that, and we're trying to use this handler for that. We're also expecting a massive increase in the amount of productivity with, especially with the ne- next few models coming. Uh, and so we're, we're readying our, our verification loops, especially with CI and CD.

   4. ARAustin Ray12:34

      I, I think also changing what our reviewers look at over time because as the models get smarter, they stop making certain classes of mistakes And so you, you ha- it's not worth spending your reviewer tokens on that anymore, right?

   5. RSRahul Sengottuvelu12:46

      Yeah, I mean, we've, we've invested in our own, uh, code review bot as well, which is also built on Inspect or, or our background agents API. We pull from some memories of things that we especially want to look for. We have s- certain teams that write their own, uh, skill files that look for certain things so, so that they can codify the knowledge that they have built up over the years into these, these files that allow people to

10. 13:08 – 13:52

    Ramp's culture of experimentation

    1. RSRahul Sengottuvelu13:08

       move a little bit faster.

    2. BOBoris13:09

       It sounds like it's, it's not like Austin and Rahul that are going in and just, like, breaking down every bottleneck, although I'm sure you're doing a lot of this. Like, how do you create a culture where engineers feel empowered and have, like, the visibility and the tools, whatever you need, to find the bottleneck and to break it down?

    3. ARAustin Ray13:24

       It's just Ramp, right?

    4. RSRahul Sengottuvelu13:25

       Yeah, I mean, I think a lot of it is, is, uh, the culture that the company has built is a culture of experimentation, a culture of, like, building something that maybe didn't pan out, and that's okay. We've tried something, we move quickly. I think one of the things that has been helpful is because we've had free access to all the tools, to all our engineers, we don't really like to l- impose a certain token budget or tool budget or tell people that they should use this thing or not, or that thing. A- a- and in general, it, it m- it becomes

11. 13:52 – 16:05

    Glass and Inspect: Ramp's AI coworkers

    1. RSRahul Sengottuvelu13:52

       a lot easier to speak the same language.

    2. BOBoris13:53

       I mean, it sounds like you guys just built, like, a huge number of these, like, background agents, like various APIs and systems internally. So you mentioned Project Glass, you mentioned, uh, Inspect. Like, walk, walk me through these. What are these tools? How do you use them? How are they built?

    3. ARAustin Ray14:07

       Yeah, so Glass is where our... It's the home base for our non-technical folks. It's where they interact with the coding agent on, on a daily basis, and it's, it's been our belief since, since the beginning that everybody should have access to this, this power and, and this velocity increaser. But you got to meet people where they are. The, you know, they don't want to be looking at code. Uh, all the technical detail is not gonna help them go faster, and some things need to be set up ahead of time.

    4. RSRahul Sengottuvelu14:34

       Yeah, Inspect is our... At this point, it's basically our digital coworker. We've tried to give Inspect all the tools that a Ramp, uh, builder, so product engineering design person would have. So this includes access to GitHub and Linear and Slack and Datadog and Sentry and various other tools. Um, and at this point, you can ask Inspect to solve a support ticket or fix a GitHub issue or look at a Sentry error, uh, or a Linear ticket or a Zendesk ticket, whatever it may be. It, it runs on modal in the background. Um-

    5. ARAustin Ray15:06

       You access it via web.

    6. RSRahul Sengottuvelu15:07

       Yeah.

    7. ARAustin Ray15:07

       And, and a lot of people kick off stuff from Slack. So you're in a conversation with someone about something, and you go, "@Inspect, can you go handle this," or, "Can you put up a PR to fix this or investigate this?" And that actually ended up being the main way that adoption was spread 'cause you would hop into someone else's thread and go, "@Inspect, can you help them with this?" And they'd see it and go, "Oh, you can just do that? Oh, great."

    8. RSRahul Sengottuvelu15:29

       Yeah, j- and every PR now comes with its own VM, and it's, like, running for, for a little while. So people can take over sessions, collaborative. It's, it's all link based.

    9. ARAustin Ray15:38

       Multiplayer. Yeah.

    10. RSRahul Sengottuvelu15:40

        It's all multiplayer. It just works out of the box. And again, we've tried to focus on the correct trace, like, what should this agent have done, and try to shape it, uh, to that way, uh, so that it can, it can do a lot. At any given point, sometimes people feel the urge to move back to local dev. We haven't, we haven't f- fully finished this, this project, but we've tried to give Inspect that additional tool, that additional se- repo or dependency that allows people to

12. 16:05 – 17:13

    On-call assistant: an AI SRE on Claude Code

    1. RSRahul Sengottuvelu16:05

       see a little bit further in the background.

    2. ARAustin Ray16:07

       I also do want to mention On-call Assistant. So On-call Assistant, uh, has been running on and always ran on, uh, Claude Code. Uh, and that's another instance of just taking what works really well locally, proving it out locally, building up the skills and MCPs and prompts that make, uh, essentially like an AI SRE run really well on incidents to root cause them and put up PRs and fixes, and then just packaging that and, and having it run in a, in a container with safeguards and guardrails, right? So On-call Assistant runs on every incident, uh, that gets assigned to our engineers. So that includes customer tickets, customer support tickets that require an engineer. It also includes system level incidents, and then, uh, comes back in with a really solid root cause analysis, uh, in the Slack channel that we have for every incident, and then the incident responders interact with it. And we've had that running since, uh, late February or March. Um, the, the stuff you can build on the primitives on this sort of Unix philosophy Claude Code executable,

13. 17:13 – 18:44

    More agent sessions from automations than humans

    1. ARAustin Ray17:13

       it's just wild.

    2. BOBoris17:14

       Yeah. We, we have a bunch of really similar tools internally and, um, now Claude Tag-

    3. ARAustin Ray17:20

       Yep

    4. BOBoris17:20

       ... which it sounds like pretty similar to Inspect, it sounds like.

    5. ARAustin Ray17:23

       In a lot of ways.

    6. BOBoris17:23

       Yeah. It's, it's also multiplayer. It's also kind of proactive. Uh, you know, it's in Slack. It's sort of taken over a lot of these special purpose bots.

    7. ARAustin Ray17:30

       Right.

    8. BOBoris17:30

       I wonder if you guys are seeing the same thing.

    9. RSRahul Sengottuvelu17:32

       Yeah, we're seeing something similar. Some more Inspect sessions are coming from automations than humans at this point.

    10. ARAustin Ray17:37

        Yep.

    11. RSRahul Sengottuvelu17:38

        And so every time something... there's some sort of trigger, uh, sometimes they're scheduled a certain time of the day, sometimes they're from other external systems, uh, the Inspect session kicks off and sometimes notifies people in channels or, or by DM.

    12. BOBoris17:50

        And, and I guess for this one also, organizationally, culturally, how do you do it? Is it like each of these automations is built by different teams that's closest to it? Or do you have like a central dev infra or like AI team that is responsible for all of these?

    13. RSRahul Sengottuvelu18:03

        I think it's been very surprisingly, very decentralized, and we're very happy about that. Uh, I mean, there are teams that maintain certain abstractions, uh, and the Inspect team is maintaining the Inst- Inspect abstraction, and that is a lot... a bedrock for a lot of these automations. If we let everybody build what they, what they would like to build, again, we were okay with that, and we want that. We want more of it. Uh, all we can do is pro- build a great product so other teams are incentivized to build on top of us.

    14. ARAustin Ray18:26

        It's, it's also a mix of, like, desire paths of people wanting the same-- expressing the want for the same sort of thing or building the same thing separately, and then the sort of platform team going, "Okay, let's make a solid thing for this," and vision from the platform team of, "We're gonna need this when,

14. 18:44 – 20:48

    No token budgets for engineers

    1. ARAustin Ray18:44

       you know, the model gets smarter."

    2. RSRahul Sengottuvelu18:45

       Yeah, I mean, so taking a step back, again, one of the things that we've tried to do is- not impose limits on how m- how many tokens or dollars each individual spends. We want them to be able to access any level of intelligence without limits. So because of that, we've tried to do everything else in our power to make sure that people can step up and get that intelligence when they want. So that includes things like defaults. It's using batch and flex APIs. It's using cheaper models for automations when they're not human-controlled. So we always expect to stay on the latest frontier, and so we don't want features or people to overfit on a certain model's behavior.

    3. ARAustin Ray19:22

       And then there's a good amount of just talking to people, too. I think you and I have both done this, where we, we see someone suddenly get to the, become a top spender on a certain month, like way above what they normally do. And, and we reach out to them and say, "Hey, what are you working on? [laughs] You know, it looks like you're, you're spending a lot. I'm, I'm curious. And if it's, if it's something that you're not planning on platformizing but is platformizable, like, let's work together. Let's do that. Let's, let's expand the impact. And if it's a mistake, then le- I'll, I'll help, I'll help you with that. And, and then, uh, we can work on getting the cost down later if it's something you do want to platformize."

    4. BOBoris19:59

       So essentially it's like this, this culture of experimentation and innovation.

    5. RSRahul Sengottuvelu20:04

       Yes.

    6. BOBoris20:04

       I- it's letting you just totally automate like big swaths of work that used to be manual before. So obviously it works. And so then kinda your job is to support people and optimize the use case after it takes off.

    7. RSRahul Sengottuvelu20:17

       Yeah. And so the, the other way to, to look at it almost is like if you are in the positive ROI section, where you know that every do- every dollar you spend on tokens, you're actually making more than $1, you actually don't want to be minimizing costs anymore. We also expect the level of intelligence that Fable has, the, the cost of that to decrease over time, as it has for the last few years, so it's not anything new. And we would rather have everybody at Ramp be familiar and really good at pushing the frontier and pushing this intelligence, making it sweat, um, on, on hard problems

15. 20:48 – 21:58

    Advice for CTOs adopting AI agents

    1. RSRahul Sengottuvelu20:48

       sooner than later.

    2. BOBoris20:49

       What is your advice to your peers, to other CTOs that are trying to figure out, like, what do you do? How do you adopt agents? How do you, how do you make your way through this thing that's happening in the industry?

    3. RSRahul Sengottuvelu20:59

       We've made a lot of progress in the models today. We've had... We have great tools at our disposal. But I think the thing that people don't pay as much attention to is also the rate of change and how much things are changing, uh, over the last, uh, few years. And if you pay more attention to that as opposed to the current snapshot, then you begin to see the pattern of, like, rising intelligence and agency and the ability of models, for models to do more things. And I think we've tried to build for what comes three to six months later down the line, because sometimes when you're playing catch-up, uh, and you're building for what's available today, it might already be too late by the time you, you ship. And so, um, paying attention to the scaling itself has, has been very helpful for us.

    4. BOBoris21:43

       All right. So with that, uh, Austin, Rahul, thank you guys so much for taking the time and for hosting us in this beautiful space. Thank you guys so much.

    5. ARAustin Ray21:50

       Thank you so much.

    6. RSRahul Sengottuvelu21:51

       Thank you. [outro jingle]

Episode duration: 21:58