By playing, you agree to uListen's Terms and Privacy Policy and YouTube's Terms of Service.
The Pragmatic EngineerSoftware factories and code reviews with Harjot Gill
Summaries, chapters, transcripts, and FAQs on this page are made by uListen, not YouTube.
Great conversations run long. Their best ideas don't have to.
uListen is free on iPhone, iPad, Android, Chrome, and Claude.
See howTranscript
100 min read · 20,131 wordsIntro
Harjot Gill 0:00 Our AI cost, mine alone, would be 250K a month, 275K, something like that.
Gergely Orosz 0:05 Wow. You're really spending it.
Harjot Gill 0:07 Yes. I mean, that's just me alone. Humans can make a lot more decisions in a day to drive millions of dollars of spend. This might be a hot take, but humans were writing slop, not AI. That's what I'm realizing. AI is actually writing less slop at this point if it can validate against something.
Gergely Orosz 0:19 So you're saying that in the past, most teams would ship kind of slop or low quality stuff when we didn't validate it, right?
Harjot Gill 0:25 Yes. We are seeing less defects, not more actually, as the velocity is going up.
Gergely Orosz 0:29 So you are hiring and you said there's a caveat. The person who applies must meet one of the following two criterias to join: either has tastefully shipped at least 1,000 PRs in the last 30 days, or has played at least 500 hours of simulation games such as Civilization, Factorio, et cetera.
Harjot Gill 0:44 If people have grown playing strategy games, I think they will be very successful with these agents right now.
Gergely Orosz 0:50 [upbeat music] What does it look like when a startup CEO merges 5,000 pull requests in a single month thanks to going all in on AI agents and software factories? This CEO is Harjot Gill, heading up CodeRabbit, the AI code review startup. Harjot had been writing code for 30 years, is on his third startup, and never shipped as much as he does now. Today we cover why Harjot says human code review is dead, but AI code reviews are more important than ever, the concept of driving license for engineers, how devs at his company lose their right to ship to prod if they ship too many bugs, his spicy take that $8 million seed rounds are no longer enough to build a software startup and me pushing back on it, and many more. If you want an unfiltered look on how a truly AI-pilled founder/software engineer operates who is extremely hands-on with AI, this episode is for you. And while I am nowhere near as AI-pilled as Harjot, I found it educational to understand his thinking and his views on the dev tools industry, where competition feels more intense than ever before. This episode is presented by Turbopuffer, and I want to show you the new and improved The Pragmatic Engineer search at pragmaticengineer.com/search that I built on top of it. I was frustrated with Substack Search, how their search was keyword only, and I couldn't do a search like the article where me and Ken Beck made fun of a big consultancy. I'm obviously looking for the piece about McKinsey, but Substack cannot find it for the life of it, so I decided to build a search that I needed. I did the whole thing from idea to deployment in three hours with Codex and Turbopuffer, with TPuff doing the heavy lifting. I dumped all my articles into Turbopuffer. It handles creating embeddings and getting full tech search working. The result is a much better search than what Substack has. Check it out for yourself at pragmaticengineer.com/search. But let's think bigger. What would it take for Substack to give every single publication this kind of advanced search? With most other search systems, you would need to create a bunch of shards, and it would be expensive to operate the search indexes. But with Turbopuffer, each publication is simply a namespace. Idle namespaces live on object storage, which has no compute cost. Active ones are pulled into NVMe and memory caching tiers, so they are served very fast. And this is how Turbopuffer can scale to hundreds of millions of namespaces and why multi-tenant AI products at companies like Ramp, Harvey, and Notion are built on top of it. With Turbopuffer, you really do get the best of all worlds: ridiculous scale, low latency, and cheap operating costs. Whether you're building a side project like I did or a platform with millions of accounts, check it out at turbopuffer.com/pragmatic. Harjot,
How Harjot shipped over 5,000 pull requests in a month
Gergely Orosz 3:17 welcome to the podcast.
Harjot Gill 3:18 Hey, thanks for having me here.
Gergely Orosz 3:20 It's great to have you here.
Harjot Gill 3:22 [laughs]
Gergely Orosz 3:22 This is your pull requests per month. Like-
Harjot Gill 3:25 Yeah
Gergely Orosz 3:25 ... the last month, it was more than 5,000 pull requests.
Harjot Gill 3:30 That's right. That's right.
Gergely Orosz 3:31 How? How? [laughs]
Harjot Gill 3:34 [laughs] At this point, like, it's like really, um, machines are building machines, like, I mean, just good at keeping the agents busy at this time, I would say.
Gergely Orosz 3:41 Uh-huh. What are these pull requests do- When are you building it? Because, uh, you know, like you will, you will have a lot of responsibility and time spent just running the company, I guess coordinating a few people, pulled into fires, the usual stuff, right?
Harjot Gill 3:55 Yeah, I mean, it's all started, like, I think when the models, um, I would say Py3 Codex was kind of the first inflection where even we started seeing these agents getting good enough to do, like, real world work, like we can even ship to production. And around that time, like, it was very clear to me that, okay, we have been around for, like, two years. Like, I mean, so generative AI has been going on for a while, but last three or four months were, like, a big inflection point.
Gergely Orosz 4:18 I also see it here. It was up, up, up to July, you, you had like, I don't know, maybe 100 or 200 max, and then it went to more than 1,000 in July.
Harjot Gill 4:25 And it was just something else, and you wouldn't believe, like, everything just breaks when you start shipping at that scale. So as a leader, like, one of the key things that I keep doing is, like, how do I make my team productive? And a lot of my pull requests go in just fixing some bottleneck in our SDLC at this time. I mean, of course, I'm doing like maybe one or two tactical features where I'm driving the agents with each message to craft the UI or something, but most of my volume of the code will be, like, believe it or not, fixing things like janitorial work, like fixing CI or, uh, flaky tests or, um, building some, like, distributor system, like fixing something in caching or something, right? So all of that hill climbing is what I'm doing with these pull requests. Um, right now it's very, not very optimal. I would say that number should probably like be 10X of where it is. We have-
Gergely Orosz 5:12 Really?
Harjot Gill 5:12 We're not even hitting the, the peak here with what these agents can do.
Gergely Orosz 5:16 And what agent or what tool really kicked this off around, I see it was July, August. Was, was, was this Opus? Was this Codex? Was it something else?
Harjot Gill 5:27 Yeah, I think, yeah, both the model providers have been, like, in lockstep in terms of capabilities. So a lot of our team members use Opus 5.5 with Claude Code, but I personally use Codex. I mean, I've been a big fan of, uh, the Codex desktop app. Like, I've been, like, coding, like, 30 years. I've done three startups, and most of the time I've also been, like, kind of the CTO hat on every company I've been. And I have been in a Neovim person. I've been, been a code editor person-
Gergely Orosz 5:52 Oh, yeah
Harjot Gill 5:52 ... all my life. Even, like, had put out my dot files, which a lot of people use. Um, um, but now we are in a different zone altogether. I've not been in a code editor since I sh-shifted to Codex desktop app. It's pretty awesome. And ever since then, like, it's been always about, like, how many agents can I keep busy at a time. So I'm actually not sleeping much. That's the other [laughs] side of-
Gergely Orosz 6:13 Mm.
Harjot Gill 6:13 Like, you get all this anxiety that you have to really, um, because the kind of value you are creating, I mean, if you know really what to build, then the kind of value you're creating is, is phenomenal. Like, I mean, I've never seen things move in such a compressed timeframe quickly within the companies. So it's not just the volume. Like, these are not like just some silly PR. They're, like, actually moving the needle forward for our company.
Gergely Orosz 6:33 Okay, you're not sleeping much, so it's eating into your, your nights, clearly.
Harjot Gill 6:36 [laughs]
Gergely Orosz 6:37 Uh, and, and you're running ED agents there with mostly with Codex and, and some other, uh, tools. But when you're commuting, you know, a lot of people listen to podcasts or, or, or music or just drive, and you have a cheat code there, right?
Harjot Gill 6:49 Yeah. So I have, um... So I kind of live in the East Bay, um, and I take the Bay Bridge on my way to work here. And I have a Tesla FSD, and I love FSD. I'm, like, roughly 97%, um, self-driving on FSD. Like, pretty much it's, like, driving on its own. And what I do is, like... And, and FSD has gotten really good. Like, earlier it used to, like, nudge you that you have to hold the steering wheel or something, but now you can, like, as long as you're, like, looking at the r- road every few minutes, it's okay. So as a cheat code, what I'm doing on the side is, like, I have got my laptop with the Codex session, and I'm just, like, sending the instructions while I'm driving. So it's kind of funny, like the AI is driving the car while, like, the other AI is driving the company.
Gergely Orosz 7:28 Wow.
Harjot Gill 7:28 [laughs] Which I'm steering.
Gergely Orosz 7:30 As well that we're there, right? And, and also I assume, you know, you can now talk as well or, or, or w- like, you know, there, there's, it's, it's Simon Wilson was telling me how I think a year or two ago he was already talking. It's, it's just crazy how different hands off or, or in different areas you can, you can just use AI to, like, build stuff.
Harjot Gill 7:46 Yeah, that's right. Like, one of the things with a company like CodeRabbit is, like, as you start shipping faster, um, every part of your business also have to move at agentic speed. Like, what we are finding is we're shipping so much that the attention, the marketing has become a bottleneck. So I had to go back and also, like, refactor the marketing, uh, foundation to be very agentic, which means that we build, like, a separate repository. We call it the content studio. All our slide decks are now in code, HTML, CSS. We built a whole portal to, like, share those, download PDFs and so on. We actually are running even ads through code. Like, to s- like, we have millions of dollars of Google ad spend, and the way we are going after all these keywords and the different copies on that is, well, we have a loop where we look at the ads' performance, and then some agent comes up with the code changes, and we actually create pull requests to deploy ads. So when you merge that pull request, it deploys it on Google Ads.
Gergely Orosz 8:41 Wow.
Harjot Gill 8:41 So it's like, like everything is code at, at, inside CodeRabbit at this point.
Gergely Orosz 8:45 Okay. So you've been
Harjot’s path into tech and his first startups
Gergely Orosz 8:46 in the industry for closer to 30 years. I, I just want to go back before we get more into, like, how we're building now, but how did you get started? How did you get into the industry? Where did you first work?
Harjot Gill 8:56 Yeah. I mean, I started my career in very early stage in a telecom company, which was actually a startup acquired by Alcatel-Lucent back in the day.
Gergely Orosz 9:03 Mm-hmm.
Harjot Gill 9:04 That's where I first got a whiff, okay, people could start companies and build, um, deep tech. I mean, there was, like, carrier grade systems, high availability systems. So learned a lot about distributed systems in my first job. At that point, I kind of like felt that, okay, I want to do my own startup, and that was the reason I came to the US. Um, and the route I took was, I always wanted to do a startup. I mean, that, that's very clear. It's not like something-
Gergely Orosz 9:26 Very clear. And, and now you're on your third one anyway.
Harjot Gill 9:28 [laughs] So the route I took is I came into my PhD at, uh, University of Pennsylvania in the iSchool at Philadelphia, and the goal over there was very clear that, I mean, I'll incubate some good idea that I'll take out and then commercialize, and that's exactly what I did. So we did a technology transfer from the university. My first startup was, uh, kind of also like, um, an inflection point, right when, like, microservices and Kubernetes were taking off. We saw that the observability is going to become a bottleneck.
Gergely Orosz 9:55 Yeah.
Harjot Gill 9:56 So that was my first startup, was called Netsil. Listen spelled backwards. [laughs]
Gergely Orosz 9:59 Oh, yeah. And, and this was observability for Kubernetes, right?
Harjot Gill 10:03 That's right.
Gergely Orosz 10:03 And monitoring.
Harjot Gill 10:04 Microservice. So the idea there was, like, these, prior to microservices, you had monoliths, and you had, like, a Java monolith where people will install an APM agent like New Relic App Dynamics to instrument that. So your function calls were in the code itself. And when function calls became API calls was when you ran microservices, so you started decoupling. Then you had to, like, really observe the network. Your network became the app in a way.
Gergely Orosz 10:25 Yeah.
Harjot Gill 10:26 So that was the idea behind Netsil was can we listen in to these network events to understand application performance. And that was kind of precursor to eBPF and some of those technologies that came out later.
Gergely Orosz 10:37 Mm. After the startup, you, you moved to a larger startup, right? Or, or did you move, or was it acquisition?
Harjot Gill 10:45 We got acquired. Yeah, it was a pretty-
Gergely Orosz 10:47 By, by Nu- Nuance, right?
Harjot Gill 10:48 Nua, yeah, this was a public company at that time, and we, um, um, got a pretty decent offer. Like, we just did one round of funding, but then the offer was pretty good that we, we took an off-ramp. Then I stayed there for, like, a few years, uh, trying to, like, spend time integrating our technology into their stack and the product offerings. Um, then I quit to start another company called Flux Ninja, uh, which was in, like, web scale. Like, how do you do load shedding, graceful degradation, uh, controllability, right? So we built, like, whole PID controllers to, uh, make sure that the applications don't crash with cascading failures during, like, Black Friday spikes and so on. Now, very interestingly, like, at that time, the market had appetite for all these Kubernetes service meshes, but then generative AI happened all of a sudden. We saw that first, yeah, when it touched, uh, GitHub Copilot. I mean, we had Tabnine-
Gergely Orosz 11:36 This was, this was 2021, right?
Harjot Gill 11:37 Yeah. So at Flux Ninja we were using some AI, like Tabnine is one of the products we were using.
Gergely Orosz 11:42 Oh, oh, Tabnine was the OG of, of, o- of, of AI auto-complete.
Harjot Gill 11:47 Yeah. And it would, like, create its own local model or something. I don't know what it used to do, but it used to work, like, but it wasn't, like, as great as, like, when the GitHub Copilot, that was, like, the aha moment, right?
Gergely Orosz 11:56 Yeah.
Harjot Gill 11:57 And this was, I believe, even before ChatGPT had come out. At that point it was very clear. Like, we were a fast-moving team, and at that time the code review was already a bottleneck. So people were doing stack pull requests. Uh, we built our own tool called SPR. We open sourced it. But the, but the thing is that, like, it was still not ideal, like, given the volume of the code just started growing after GitHub Copilot. It was very clear that AI is going to write most of the code, and-
Gergely Orosz 12:23 So just to pause
Early conviction around AI and founding CodeRabbit
Gergely Orosz 12:25 there, when you saw GitHub Copilot in 2021, this was before ChatGPT, you already predicted or kind of like fast-forwarded saying, "Oh, AI will write most of the code"?
Harjot Gill 12:37 That's right.
Gergely Orosz 12:37 This was not a commonly held view. Like, a, a select few might have held it, but what, what made you assume this? Can, can you talk us through our, our, our, our-- your thinking process that time around?
Harjot Gill 12:47 Yeah, I was in the arena, like, as a leader. Like, I was actually coding on the ground. Even at that company, I mean, a lot of the work I was doing, janitorial, like just making sure the developers were productive, adding all the linters, the tooling, everything to go faster, right? As a leader, like, my goal has always been, like, how quickly we can get to the next milestone, right? And when I touched that system, I don't know, like, it just came to me that, okay, this is gonna change everything. The tool was really good in, like, sensing what I need to do next.
Gergely Orosz 13:14 Mm-hmm.
Harjot Gill 13:14 And it-- I was accepting the suggestions. Like, it was like writing really complex algorithms.
Gergely Orosz 13:19 Mm.
Harjot Gill 13:19 Uh, you just provide a few comments on what you want to do and, and the code came out, tab complete, right? Um, it was pretty magical. And at that time it was just clear to me. I don't know how, but I mean, once you are in the arena, I think you see these patterns-
Gergely Orosz 13:32 Yeah
Harjot Gill 13:32 ... long enough, the second order effects. And the same thing was clear to me three months back on how software's gonna be built. Like, I would say I'm still learning. It's not like we're not on a stable ground yet. Things are moving too fast, and some of the things I'm seeing now, uh, we could not have predicted even six months back on, on how the software development is gonna evolve from here.
Gergely Orosz 13:50 You founded CodeRabbit on this kind of notion or, or prediction. W-was that how it happened?
Harjot Gill 13:55 Actually, we didn't realize this would be such a big opportunity. One thing was clear, it will become a bottleneck. But we thought, I mean, the market cap, we were looking at Snyk and a few, um, other companies in the GitHub ecosystems, right? Because we have a installation which is, like, inside the v- version control systems like GitHub. Um, like, there was some precedence around previous generation companies like SonarSource based out of Europe, which are like code quality agents. So we were kind of-
Gergely Orosz 14:18 Yeah
Harjot Gill 14:18 ... like thinking, okay, the market cap would be, like, in that territory. What we didn't realize [chuckles] that this is, like, much, much bigger market cap opportunity because people who were not even traditionally buying those tools would line up to buy AI code reviews. Even though we don't sell to individual users, a large number of our users and lo- are, are individuals paying for the product.
Gergely Orosz 14:37 So you've been a Neovim user. You've, you, you
Switching to AI coding
Gergely Orosz 14:40 wrote a lot of code for, like, decades pretty much, right?
Harjot Gill 14:42 Mm-hmm.
Gergely Orosz 14:42 Your ha-handwritten. How easy did you find it to just move over to AI? Because a lot of people, especially some folks who use Neovim, you know, like, they're kind of into the craft as well, they might have been a bit more hesitant to just go there, 'cause you kind of have to, you know, get rid of that really familiar UI. You no longer even have a UI when you're using agents.
Harjot Gill 15:04 So the-- for me, like, the writing the code is just means to an end. As an entrepreneur, like, you wanna get to the final value and validation quickly.
Gergely Orosz 15:10 Mm-hmm.
Harjot Gill 15:11 Um, I mean, I did enjoy writing code, but I wouldn't say that was the most enjoyable thing in building a company. It's about, like, how quickly you can get an idea out, get validation, grow that business, and so on. Coding was kind of a necessary thing you had to do, and a lot of the work around maintain-maintaining those code bases, DevOps, all comes alongside that, right? So for going from-- I don't miss it. If you ask me like, "Do I miss writing code by hand?" I actually don't. What I love is now I'm able to go from ideas to very quick implementation and just validate. So I'm doing the actually the fun part of software development. So I'm still doing software development. It's not like I'm not doing software engineering. But now what I'm focusing on is a lot more fun than figuring out the syntax and autocomplete. I mean, I mean, my agents are writing Rust code. I have zero idea how Rust works. We recently, like, had an agent goal running for six weeks to, um-
Gergely Orosz 16:01 Six weeks.
Harjot Gill 16:01 Yeah, so for running, so building, like, um, Git, uh, write ahead log on object stores.
Gergely Orosz 16:07 Yep.
Harjot Gill 16:07 Like we, I just-
Gergely Orosz 16:08 The write ahead log
Harjot Gill 16:09 ... when Cursor's origin blog post came out.
Gergely Orosz 16:11 Yeah.
Harjot Gill 16:11 I pointed the agent. It ran for six weeks, and now we have that in production powering a lot of the knowledge bases at CodeRabbit. So I didn't even-
Gergely Orosz 16:19 Wow
Harjot Gill 16:19 ... have to have the real implementation. It just hill climbed over time on its own, all written in Rust, and I have not read, read a single line of it.
Gergely Orosz 16:27 And, and for, for those of us who are not as familiar with this specific term, what does hill climbing mean for agents?
Harjot Gill 16:33 Yeah. So it's some sort of a loop. Like, it all starts at a very basic that agent can verify its work against something.
Gergely Orosz 16:41 Yep.
Harjot Gill 16:41 And, and these loops are like inner loops, outer loops, multiple types, like. So when you're looking at software development, we want to, like, provide agents means to verify its work close to where it's doing the software development. So we do a lot of, believe it or not, a lot of formal methods. So when we first did the Slack bot product like eight months back, uh, we invested a lot in, like, first modeling what this distributed system is going to do in TLA+, then implementing it, and also investing a lot in simulations. So the way we are scaling code development, code review is just one piece which we of course market, but there were like 100 other things you have to do, right?
Gergely Orosz 17:16 Yeah.
Harjot Gill 17:16 Um, so we invested a lot in local, um, um, deterministic, um, simulations, right?
Gergely Orosz 17:21 Mm-hmm.
Harjot Gill 17:22 Um, very recently we started investing in lean theorems as well. So a lot of that is like-
Gergely Orosz 17:26 So lean is another form of formal methods, right?
Harjot Gill 17:28 Yeah. Like, you have the theorems, and it even like used for mathematical proofs, but you can also try to build provably correct software, things like billing service and all. We have got like bunch of lean theorems. I can actually show you after the podcast. It's pretty cool what we are doing there. So what we have been doing is, like, these are all, like, hill climbing. Like, you're first making them hill climb locally, then you're going to the next environment where, let's say, emulators is another thing we have invested a lot in our own development.
Gergely Orosz 17:52 Mm-hmm.
Harjot Gill 17:53 So you, you can now very quickly mock a real emulate, like the real APIs, like Google Cloud APIs or whatever, right? Then you are actually testing against a real, um, Google Cloud or Vercel environment. Those are the cloud providers we have in, in dev. You can have the agent actually spin up services, actually test them, right? I mean, this is all hill climbing. And then in production as well, like, the thing is that I kind of see each release. So we release multiple times a day, right? Every 30 to 45 minutes. We ship code. And each release is almost like a turn in a strategy game. If you have played Civilization, you end your turn, you learn. That's how we see releases. Like, each release is an opportunity for an agent to learn something from that environment. So a lot of the times we will release features under a feature flag, do like progressive rollouts, like maybe 10% for our open source users and so on. And the agent will observe the performance, the logs until it's stable, then do wider and wider rollout. And each release is the opportunity to learn for the agent.
Gergely Orosz 18:48 Interesting, because what, what you're describing, this kind of stage and wider and wider rollout, this is something that Facebook had been doing for a long time, and they had just have a massive user base. We're talking billions of people, right? And they built this automated system. And outside of Facebook, I don't really know-- I didn't know of many other companies that had a kind of automated rollout for a few reasons. One is this only works when you have, you literally have hundreds of millions of users. It's really hard to do it otherwise. It's a large investment to build. Uh, automation is hard to build. You need even pre-AI, you needed some sort of ML, which again just goes up and up. And now you're describing that you at a pretty small company in terms of headcount compared to Face- you know, Facebook, 75,000 people. CodeRabbit is like-
Harjot Gill 19:33 40 engineers maybe.
Gergely Orosz 19:34 Y- y- yeah, and, and you've built, been able to build this for yourself.
Harjot Gill 19:38 That's right. So the way we are seeing it is like the way Netflix, Facebook used to ship with thousands of engineers and the kind of tooling they needed, like all the way from, um, the process on merge queues and everything like they needed to ship at scale, is something that small teams are going to need now with AI. So now teams of 30, 40 people, if they use agents in the right way, have the right abstraction and this whole concept of software factory, they're gonna ship as much as Facebook, which means that all the sophistication they had in terms of testing and production, managing the risk, managing your CI/CD, everything, caching, there's a lot of stuff around it, and they will need it in a small team setting as well. So it's a golden opportunity if you were, if someone was at Facebook at that time, like when they were shipping and figuring out all these concepts, to go out and now actually build products, the market's grown. There's a lot more people who need those products, uh, than ever before.
Gergely Orosz 20:30 And you mentioned software factory. It's a term that is going around a lot. What does it
Software factories explained
Gergely Orosz 20:35 mean for you?
Harjot Gill 20:36 Yeah, it's many things. I think, I mean, my h- we have a slightly, maybe it's the same take as others, but I mean, I'll give you my take on it. Even when you're building with AI, there are I think two ways of using AI. One is like more like on a potter's wheel. You are like shaping the software with each interactive chat with Cloud Code or Codex. Like, like for example, like with UI development, I have the browser open, I'm doing annotation and, and each time I send a message, I'm crafting the software, but with AI. And I can like manage as a human, like my decision capacity is to do only like one or two of those tactical chats in a day.
Gergely Orosz 21:13 Yeah.
Harjot Gill 21:13 I mean, I usually have like 20 chats in Codex right now. Two would be of that manner.
Gergely Orosz 21:17 Mm-hmm.
Harjot Gill 21:18 Around 18 are more like the factory concept, like where I know that I can have the agent hill climb on something, whether it's like building a distributed system, I can be hands-off with a goal, just like the example I gave you with the Git on object stores and also-
Gergely Orosz 21:35 Yeah
Harjot Gill 21:35 ... SQLite LTX. That's the other thing we did with knowledge base is we removed our vector database provider to build our own in-house version on SQLite LTX, right? On an object store. Um, those kind of things you can hill climb. The other stuff is like each time you have a deploy, CI/CD is happening, it's an opportunity for the AI to learn and continuously improve it. So I have agents running continuously to remove flaky tests, right? Each flaky test resets the merge queue. It's really bad. So you have to have that continuously happen. Each time you have a deploy or a CI, each run is an opportunity for the agent to optimize something, do the next thing, right? And then you're running the agents against production as well. So you have agents which own health of some components I care about. For example, I care about billing, I care about the knowledge base performance, like I care about the code graphs we have. So each deploy, I have the agent running in a loop to observe the next experiment now and do either two things, like add more tele- telemetry if it's like flying blind or fix something. So those are the kind of loops I'm running, and those are factories. So the thing is that right now the chat is not kind of the right interface to run those agent fleets at scale. The chat is great when you're doing crafting the software one message at a time. So some of these things we are still figuring out. First of all, that whole maintenance, the janitorial work around shipping software and maintaining the factory should probably be in the cloud on sandboxes and all. And I shouldn't be the one waking up at like 3:00 a.m. I, I mean, it's more like morning shift, night shift now in the factory. Uh, we luckily have a team in Bangalore, so we have a follow the sun model. So I want like if I'm babysitting the agent, so I'm not owning the cone anymore, we are owning agents. Let's say I have an agent looking at the health of a billing service and it stalls on something, it has a question, then if I'm around during the US time zone, I'm able to answer that question. But I want someone to be able to unblock the agent while I'm sleeping. So I would say we haven't figured it out yet. So we are going on the upward trajectory, but most of our development right now is constrained by how many sessions people can run locally, how many releases they can like make their PRs land in. Like we look at the release coverage and the PRs they landed. And we are also constrained on local resources. One, one of the things we found, and that's where I have a cheat code as a CEO, I have a very powerful machine. The developers don't have those. Like they have like 48 GB RAM, 24 GB RAMs, and they max out at like three, four sessions. Beyond that, like the machine just dies with the unit test it has to run locally, right? So we are feeling like the real compute constraints right now in terms of the local development, and we are investing very heavily to go and make some of our cloud-based environments work.
Gergely Orosz 24:04 But I guess it's pretty obvious at this point that a lot of development will move to the cloud because of a lot of the constraints that you mentioned. And I guess the only downside I can think of moving to the cloud, of running coding agents in the cloud is, A, there's a bunch of work that needs to be done that you have an environment that's similar to your local one, security, all, all that stuff. But you need to build that. And there's potentially a cost, uh, question as well because now, you know, like y-you have your hardware, and when you're utilizing it pretty well, that's pretty, that's pretty good, right? You buy it once and, and you do it, and there will be an additional cost part. But the, the, the cost, I, I wonder if that's a smaller question because, you know, like we already know AI has a very real cost, and I know you're paying a lot for AI already.
Harjot Gill 24:45 It's actually better because in the way, like if it's in the cloud and more like a collaborative setting, you clearly see the value created because right now we have some people who are spending tens of thousands of dollars on local tokens.
Gergely Orosz 24:57 Hmm.
Harjot Gill 24:57 We don't know if they're doing like side projects. Like everyone's building their like home automation systems and-
Gergely Orosz 25:01 See what you mean, yeah
Harjot Gill 25:02 ... right? So there's no auditability as well on, on local development.
Gergely Orosz 25:06 Okay.
Harjot Gill 25:06 Right? So sometimes we look at the price per PR, like mine's, I don't know, like 150 bucks a PR.
Gergely Orosz 25:11 Mm-hmm.
Harjot Gill 25:11 Like for some people might be thousands.
Gergely Orosz 25:13 Mm-hmm.
Harjot Gill 25:13 Then you have to question like, are they really doing your work or someone else's work? [laughs] So the thing is that like the cost perspective, I think we. I will be also as a leader, like comfortable spending the money if I can see where it's adding the value. Like if it's in the Slack, so we have a Slack bot over there when, um, CodeRabbit responds to all the Datadog alerts. I clearly see it's adding value.
Gergely Orosz 25:31 CodeRabbit, one of the main things that you do is AI code review. Now, I have to ask you this question.
What’s replacing human code review
Gergely Orosz 25:37 Is code review dead? [laughs] Bec-because, and specifically human code review, there are so many signs that it is on its last legs. It's a ... The p- the teams who are still doing it who I talk to, it's kind of theater. It's like, you know, LGTM, looks good to me, but people no longer add comments. They no longer read it. And, uh, companies like Uber, uh, are, are now making changes to just make it so much easier for people to, you know, like see what a code review does 'cause they also know people no longer ... They, they spend less and less time reading it because there's just more of them, right?
Harjot Gill 26:09 You're right. I mean, the human code review is dead, but the human review is not dead. Some review still has to happen. If you look at the software development sc- space, there used to be like a upstream, um, kind of a guard like when you were like planning the work because your engineering was an expensive resource. So when you were using the tools like Linear, Jira, one of the reasons to do this agile and sprints was to make sure your developers are working on most important tasks.
Gergely Orosz 26:33 We don't like spend a bunch of time and effort building something that we'll throw away, right?
Harjot Gill 26:36 Exactly. So there was a lot of regulation upstream-
Gergely Orosz 26:39 Yeah
Harjot Gill 26:39 ... earlier. Now that floodgate has suddenly opened with AI. Now you can go from idea to code without going through planning. Like you have a support ticket coming in, people will just ask Devin to look at it or CodeRabbit Slack bot or something, right? At the same time, your designers and people who were not software engineers are now doing coding as they should be because they know what to build.
Gergely Orosz 26:58 Yeah.
Harjot Gill 26:58 So a lot of designers are shipping code. You have background agents like Datadog is now going to be opening PRs. Each time it sees an alert, it's gonna say, "I'm gonna do it for you," right?
Gergely Orosz 27:06 Yeah.
Harjot Gill 27:06 So what that means is that your planning layer and the co- the review gate still has to exist somewhere, and that becomes a pull request because everything upstream, there's an influx now happening, right? And when you have a pull request open, doesn't mean that you have to ship it. At that point, you decide whether this is worth spending my time, whether I take it to production or not.
Gergely Orosz 27:25 Yeah.
Harjot Gill 27:25 Because the human taste is the bottleneck now.
Gergely Orosz 27:27 Mm-hmm.
Harjot Gill 27:27 Human judgment. So when a human is coming in, they're first making a decision, is this something I need to ship or is it slop I need to close? I mean, and, and you make a decision after the code is there, right? Second thing you're doing is like, what's the risk? Can it just go through production without me looking at the code, or do I need to understand the broader implication? Is it like a m- database migration it is doing? Why is it doing the migration?
Gergely Orosz 27:48 Oh, yeah.
Harjot Gill 27:48 So you want to sometimes look at the schema. So what, because that's more structural change, right? So we do all that like risk management within CodeRabbit. So some PRs are on fast track to production. We are even letting our interns ship directly to production. So human code review is dead, but I would say humans still review some of the higher order abstraction. And how we are like doing this with SOC 2 compliance is also interesting. So we don't real- so the, the definition of change at CodeRabbit is now a release. So we have a Slack message. We kind of get all the, the indicators around the risk we are taking. Each PR, you have a minimal risk, high risk.
Gergely Orosz 28:23 Mm-hmm.
Harjot Gill 28:23 And over there you kind of make a decision on whether this can go or some things have to be rolled back.
Gergely Orosz 28:27 Mm-hmm.
Harjot Gill 28:28 So each release is what we're reviewing now, not PRs, as a human.
Gergely Orosz 28:31 And then when you say human review, just to be clear, human means software engineer, or can it also mean non-software engineer? I'm just curious if you still think the accountability needs to sit with people who kind of know the internals, or can it be someone else? Can it be product manager or designer?
Harjot Gill 28:46 I think it's still software engineers at CodeRabbit. Like, they're not necessarily looking at the code anymore, but they are actually understanding the risk that we are taking with each release.
Gergely Orosz 28:54 Mm-hmm. So it's about who can understand the risk.
Harjot Gill 28:56 Yeah, because if something goes bad, then they are [laughs] going to be the one dealing with it, like rolling back. Like it takes like 10, 15 minutes to roll back. I mean, that's the worst that can happen if something goes bad, but they are still taking the responsibility. So we ideally want like developers to be around when something goes out so they can babysit what they've just shipped.
Gergely Orosz 29:12 Yeah.
Harjot Gill 29:13 Um, that's why we try to release so frequently. Uh, but at the same time, like with the person who's approving the release, we have SREs and also, they also like are taking some responsibility at that point.
Gergely Orosz 29:22 Yeah. I think it, it makes perfect sense, especially for a business, 'cause we're talking about a business where you actually have revenue, you have users. You know, like if slop goes out and it takes down production, uh, you're gonna just lose customers or goodwill or cause frustration and so on. So I think it's great that, uh, you're saying, "Okay, let's have someone accountable." But the problem that us devs have is there's just a lot more changes, and I'm not gonna have time to read through all of them. What ways are you thinking, either now or in the future, what do you think needs to change so that as engineers, as, as the person who's making the decision, I can just have better information to, you know, like focus my attention where I need to focus? Like for example, I don't really wanna review a manual refactoring that just move things around and it compiles. I mean, I'm ... I could, but you know, that's, there's, there's good use of my time, and there's poor use of my time, and I feel, as you will know from yourself as well, I mean, you've shipped like so, so many pull requests. You will probably focus your time on the critical stuff.
Harjot Gill 30:18 That's right. I think the human judgment is scarce now, right? And you wanna like-
Gergely Orosz 30:23 Yes
Harjot Gill 30:23 ... first of all not look at the code. There's higher order abstraction. You want to understand the risk, understand the blast radius, especially if you are familiar with some area of the code. You might also want to have inputs on the architectural change. Like for example, like if you're maybe writing a migration on a large table, like it's gonna crash in production. You have some of these like tribal knowledge you want to bring in, right?
Gergely Orosz 30:40 Yeah.
Harjot Gill 30:41 Other thing is like also like I kind of see like now driving these agents almost like getting a driver's license, right? So we basically even on the developers put some responsibility on them who are shipping. So we let people ship if they are in the like review exempt list. So we also have like different lists and groups and teams where, uh, based on their experience and their p- past performance, they can either go to production without a human review or we put them in a group where they need a human review.
Gergely Orosz 31:06 It sounds like based on a trust that you built up with them.
Harjot Gill 31:08 Right. The way we have done that is like we do look at how long there was a regression in production.
Gergely Orosz 31:13 Mm-hmm.
Harjot Gill 31:13 So because we look at the code when you fix something, we are able to tie that back into the PR that introduced it.
Gergely Orosz 31:19 Mm-hmm.
Harjot Gill 31:20 And now if someone has been irresponsible, like they did hard prompting there or they did not follow through their change into production, you will have sometimes 100 hours of exposure. Means someone had to point out, like some customer had to point out that something regressed. You didn't QA it or something. And once you have like enough strikes, we take your driving license away. We make you go through the [laughs] code review. Then, then you go through like a, like a proving period with some human reviews.
Gergely Orosz 31:43 Uh.
Harjot Gill 31:44 And then we let you ship on your own. So we're trying to like set up a culture where like I think responsibility is also with the developers to not drive these agents off the cliff.
Gergely Orosz 31:51 I like this driving license analogy on, honestly, because we talk a lot about, well, I talk a lot about or worry about quality because I, I can see that quality seems to have dipped across the industry and clearly it's because there's a lot more things going on and a lot less review. And it's nice to tie back, you know, the feedback loop that look, like I, I think it's human, right? Like if ... The more you ship, the more likely you'll mess up. And as long as they're ... you own it or there's consequences or, or you say like, "Okay, from now on for, for this time period, either, you know, come up with better practices or have a second person, have a second pair of eyes," which is expensive, but this is what we used to do even before AI, right? The intern for you, you give the trust to intern to ship, but we didn't have that. The intern ha- needed to have like two sign-offs, for example.
Harjot Gill 32:33 Right. I mean, it's almost like that. Like I think people have to have responsibility. Like we have seen some developers are really good in asking the AI the right questions around the implementation they just generated. They're more diligent. And some developers will come and give like ambiguous high level goal. They will say, "Just solve this problem for me." And AI will like completely re-architect it. Sometimes we have to pull the plug when we start shipping, seeing those PRs show up, we're saying, "Okay, need to stop that right away." We have seen those kind of things. So some people are like good at even prompting and steering these agents and others are just reckless and, and that's, that's a behavior we then try to penalize.
Gergely Orosz 33:05 Yeah. I, I do like the idea of responsibility as well, but there, there's a question of like why if the AI will get even better, right? Like we're just, we're just speculating here. But at, at that point, why would you pay software engineers? And I kind of keep coming back to the idea that well, you know, why do I pay a plumber to come when I could just watch a YouTube video and I could fix my toilet? The problem is that, you know, like 9 out of 10 times it probably works, but the 10 times I made it might either clog it or break it or something and I'm paying the professional.
Harjot Gill 33:35 Yeah.
Gergely Orosz 33:36 Because if they come in and they mess it up, they're responsible and they will fix it. And I, I'm kind of thinking that as software professionals, you know, like we will be responsible and, and uh, you know, one of the reasons we will hopefully be, you know, be, be paid the big bucks is, is we can take responsibility for a complex system where an outage is just very, uh, costly, right?
Harjot Gill 33:59 No, I think the responsibility to some extent matters, like the things you can foresee. We all, we have to be reasonable with the responsibility.
Gergely Orosz 34:05 Yeah.
Harjot Gill 34:05 Software are still like complex systems. They break in very complex ways, right?
Gergely Orosz 34:08 Mm, fair.
Harjot Gill 34:09 Um, and you saw that today the tweet around like railway employee hacking and then they're blaming agents going off track, but in the end like it's really humans who have to be responsible with how they're driving these agents, right? And they can do your bidding, but they can also go in, in crazy directions, right? I think, I think that's important and the other thing is like with this kind of when you're shipping at agentic speed, your operations also have to be agentic. So one of the things we are like agreeing is that, okay, we, our goal is not to prevent every outage. It's like to make sure that the exposure time of something bad happening in production is as little as possible.
Gergely Orosz 34:42 Mm-hmm.
Harjot Gill 34:43 Which also means that agents need to be looking at production.
Gergely Orosz 34:45 Yes.
Harjot Gill 34:46 If that's out of lockstep, you can't ship fast. You're gonna be more like AWS. You burn your hands with AI and you're gonna say, "Ban AI from my code base."
Gergely Orosz 34:53 What you're saying as I understand is, okay, you know, we'll have agentic, you know, feedback to verify the code or, or give AI code review whatever, whatever that is or run may- maybe Fuzz test, et cetera, but the observability layer should also likely be tied at like it's all part of the, the QA. You-
Harjot Gill 35:10 It has to be because your defect rates maybe are not going down, but now you're shipping more defects.
Gergely Orosz 35:15 Mm-hmm.
Harjot Gill 35:15 Because the volume has gone up, right?
Gergely Orosz 35:17 Yep.
Harjot Gill 35:17 Which means that you have to stay on top of them. Like you have to have agents always looking at not just alerts. In fact, alerts are limited by your imagination on the failure scenarios you imagine. Like actually these agents are really good at looking at logs. They're great looking at traces, right? [laughs] So the way, in fact, those are the areas we are still not like fully scaled yet. Like there's so many areas of our code where we have had longer, um, exposure in production with bad code.
Gergely Orosz 35:42 Mm-hmm.
Harjot Gill 35:42 Because no agent was looking at it.
Gergely Orosz 35:43 Mm-hmm.
Harjot Gill 35:43 Wherever we have agents looking we are able to quickly resolve stuff at agentic speeds as well. So Entire software development from left to right is now going to be driven by agents including production. It c- and including marketing. Even stuff going out you have to now be agentic now.
Gergely Orosz 36:00 Harjot just said how software development is going to be fully driven by agents and when agents write, ship, and fix most of the code you want to know why each change happened which brings us to our season sponsor Entire. A few weeks ago I told you that Entire was building a Git forge. Well, it's finally here. You can now host private repos directly in Entire in the US, Europe, UK, India, Australia, Brazil, and Singapore. And Entire has just reinvented the pull requests for the agentic era. The problem with PRs is how agents write faster than how any of us can review it so Entire replaced it with something called Trails. A Trail connects everything about the work in one place, the intent, the code changes, the agent sessions. A Trail can be spread across multiple repos so the change to your backend and your web app gets reviewed once. With the Trail, you can configure customized evals. They look like heads up display in an airplane and score your work on every push, and you can see how each agent run passes or fails your user-defined gates and decide when you merge based on these. The point is that you control when you allow agents to be allowed to merge the code, like only merging after a human review or after all gates pass. Oh, and one more thing, Entire Ship Continuous Memory. Entire turns your repo into a brain, so your agents can search past coding sessions to find the context behind earlier changes. Basically, answer the why behind code changes. Memories get shared across the team and across the repo. If you want to see what an AI native Git forge looks like, head to entire.io/pragmatic and check out Trails. And once agents are observing production and pushing fixes, someone has to decide what they're allowed to touch, which is why I'd also like to mention our seasoned sponsor, WorkOS. WorkOS provides the auth infrastructure B2B and AI native companies need to sell to enterprise customers with SSO, SCIM, RBAC, and now authorization for AI agents. WorkOS Airlock is intent-based access control. Most access control only asks who you are. Airlock also looks at what the agent is trying to do at the moment it tries to do it. Clear-cut rules like no agent deletes cloud resources are run as plain code. For the fuzzier calls, an AI judge evaluates the policy at runtime. You write these policies in plain English, and Claude can pull the ones you already had out of Notion or Jira and import them for you. Your agents, whether that's Claude or Devin, connect to a single access point, and your MCB servers are behind this access point. Airlock is a new layer on top of your existing permissions, which means that an agent can never get more access than the person who invoked the agent has. When something risky comes up, Airlock can escalate to human for approval, say, over Slack. The idea is to write the rules once and spend human attention only on the calls that need it. You can request early access at workos.com/airlock. And with this, let's get back to Harjot and what happens to code quality when agents write most of the code. Let's talk a little bit about
Technical debt and deslopping tests
Gergely Orosz 38:44 quality because in your previous startups, you know, pre-AI, uh, you've, you've built some infrastructure, uh, tooling, which, you know, reliability is very important at those place, accuracy when, when you have the, the monitoring. I, I assume these are all non-negotiable. But what's happening now with AI, what we're seeing is when you're using AI to generate code, it's great. It, it does it faster than I could do, usually similar or better quality than what I could have done. But it does get verbose. It can just repeat itself. It can just add a lot of tech depth. You know, like once, once you start generating enough, you s- you see that it's, it's just gone a bit more bloated or, or a lot more bloated. What is your observations here on how did you see this used to be for the earlier models? Have newer models changed it, and, and what... Is it still a problem? What are you doing about it?
Harjot Gill 39:30 Yeah, I mean, once you unleash the agents on your code base, there is some chaos. I mean, there's a saying like, "Let the chaos reign, and then you reign in the chaos." That's a philosophy. So rather than-
Gergely Orosz 39:39 Oh, that is-
Harjot Gill 39:39 ... you prevent the entropy, the entropy is gonna increase of the code base.
Gergely Orosz 39:43 Mm-hmm.
Harjot Gill 39:44 Maybe, maybe your security posture also weakens, right? Maybe because you're not reusing the same auth layers or something like reusing stuff. Then you need to have agents which are self-maintaining that code, the janitorial, like the Roombas, right? So that's the approach we are taking. Let, let, let people ship quickly. Uh, let some of the chaos reign. Maybe things are not scalable right now, but then you have the agents also cleaning up the mess, like all the way from flaky tests to bad abstractions, to refactoring the code. For example, recently we ran agents to introduce effect and X State into our code base. So refactors on con- constantly keep happening, right? So you only make sense of that mess once you actually have the mess there. Like then you figure out what's the best way. So I think both have to be like, it's like, uh, reigning in the chaos as they say, and then reign in the chaos, right?
Gergely Orosz 40:29 Yeah. And I, I know you, you've previously talked about things like deslopping tests.
Harjot Gill 40:33 Yeah.
Gergely Orosz 40:34 Can you tell me what, what is that? Is, is that a skill? Is that a-
Harjot Gill 40:38 Oh, it's massive. Like, I'll tell you, our CI has gone through the roof. Like I think we're probably spending as much on, on CI as tokens.
Gergely Orosz 40:45 And you're spending a lot on tokens.
Harjot Gill 40:47 Yes, and also on CI. Like CI, I heard last time, like vendor quoted us like 13 million or something a year. We built our own, by the way, now in one week. [laughs]
Gergely Orosz 40:54 Yeah, but, but that, that would've been your run rate if you would've stayed on the vendor.
Harjot Gill 40:58 That's right. So the thing is like these agents are now, because they're like writing code and CLI commands, they're driving a lot of compute, and all these code changes, you're still make-- going to make them go through CI gates, your merge queues and everything, right?
Gergely Orosz 41:12 Mm.
Harjot Gill 41:12 And you're in fact writing more linters, more simulations because you want to be sure. There are two problems here. One is like a lot of these tests are actually useless, so we already have agents which are deslopping the tests, removing unit tests, doing more integration, end-to-end tests, um, redundant tests. Like we have those running all the time. Now at the same time, uh, it's a constant battle. Like your CI time goes up, but you want to keep them within three minutes ideally. Like that's how much... I think we're like roughly four minutes right now. And at the same time, um, then you're sharding and all, like the costs go up. And this is the reason, like one of the things is like at our scale, like it makes sense, we just build our own. So we built our own CI, right? Running, um, right now what we are hit- hitting is like compute capacity.
Gergely Orosz 41:53 Hmm. Are we talking about CPU capacity?
Harjot Gill 41:54 CPU, yes. Even GCP unfortunately has very little compute available. Anthropic took it all. So we were in the central region, and we couldn't even upgrade our database instance. Uh, we ran out of compute.
Gergely Orosz 42:05 Damn.
Harjot Gill 42:06 And we had to like migrate.
Gergely Orosz 42:07 And, and you're like, just to be clear, you're not the smallest of companies. I mean, you, you will not be a, a Google, but, but you have plenty of, of budget. Uh, you have good funding, like you're, you know, you're ready to give the money to-
Harjot Gill 42:19 Well, yeah, we have signed good contracts with them, but unfortunately they, they, they're all locked in by the large because they're all running sandboxes, RL environments. I don't know what they're scaling.
Gergely Orosz 42:27 Yeah.
Harjot Gill 42:27 But they're also not just scaling GPUs anymore, but they're con- even the RAM, CPU-
Gergely Orosz 42:31 Damn. So, so because I, I wrote about the, the CPU crunch in, in The Pragmatic Engineer a few months ago, but you're, you're a good example where it's, it's happening, right?
Harjot Gill 42:39 It is happening in a big way right now. We have a big CPU crunch as well, and it's just like a few companies shipping at that scale. Like, I mean, for the rest of the world to get there, I think the biggest bottleneck is gonna be the data centers, the energy And those kind of things. The models are commoditized. Like, the models are there, but they drive a lot of that compute as they work.
Gergely Orosz 42:55 But now a little bit talking about this, the CPU crunch. So g- 'cause I think this is r- I'm, I'm super curious to, to learn more on both what you can do to get more CPUs and what you can do to maybe use your CPUs more efficiently, right? Y- you mentioned Rust, and I-
Harjot Gill 43:09 Oh, yeah
Gergely Orosz 43:09 ... I wonder if this is about it, 'cause Rust is as efficient as you can get your code down to, right?
Harjot Gill 43:14 That's what we did. Like, I mean, we were on TypeScript earlier, like, we just moved to Bun. I mean, there was just another ... I was in Tahoe on a beach. I ran a goal, and over the weekend, all the services were migrated to Bun.
Gergely Orosz 43:24 Dude, your, your, your idea of holiday [laughs]
Harjot Gill 43:26 Yeah. I mean, I have the pictures of it. The team was so surprised. I just ran a goal command, and all the services moved to Bun over the weekend. [laughs]
Gergely Orosz 43:32 So, so, so y- just so I understand, so you, you have this thing where you set up a goal, and then you do a long-running session in the cloud or on your local computer?
Harjot Gill 43:41 Unfortunately, it's local. Like, I mean, we do have a cloud agent, but I'm the like ... I should be the one using it [laughs] but I have, like, a powerful machine, so I always go in and-
Gergely Orosz 43:48 And, and then it goes off and it, it, it runs-
Harjot Gill 43:50 Yeah
Gergely Orosz 43:50 ... on your machine and it runs long-running task.
Harjot Gill 43:52 Mm-hmm.
Gergely Orosz 43:52 And it usually completes it.
Harjot Gill 43:54 It does. Like, I mean, it will sometimes, like, stall to ask a few questions, but some of these, like, interestingly, the distributor systems are now far easier to build. They don't need much hand-holding because there are only few ways to get something right when it comes to the architecture. The product, it's, like, open-ended. Like, the UI you can build in 10 different ways, and that's where the taste, the judgment comes in a lot more. But we used to pay a lot for people doing complex scale distributed systems, things like Kubernetes and, you know, advanced caching observability scale, right? To be honest, those things are super cheap now.
Gergely Orosz 44:24 So you're saying that that part, the kind of this back end, even though they're complex, it's been commoditized thanks to the models?
Harjot Gill 44:30 Yeah, because they're like, as this, as the saying goes, like, there's only maybe one way of building Unix. If you were to build it today, you're gonna come up with the same abstractions.
Gergely Orosz 44:38 Yeah.
Harjot Gill 44:38 Right?
Gergely Orosz 44:38 And I guess going back to, for example, Kubernetes, uh, I had Kat Cosgrove on from the Kubernetes team, but she was saying how Kubernetes is, on one end, it's complex to onboard too.
Harjot Gill 44:47 Mm-hmm.
Gergely Orosz 44:47 'Cause it's complicated, but they also have really good documentation. And I guess when you have something like this and there's only, I guess, I assume there's only one or maybe two ways to do things, but when you have this agents are pretty good, when there's good documentation-
Harjot Gill 44:58 Yeah
Gergely Orosz 44:58 ... no matter how complex it is, they can, they can navigate the complexity.
Harjot Gill 45:01 Those kind of things they're, like, great at. Like, things like, okay, I'll optimize my Docker files.
Gergely Orosz 45:05 Yeah.
Harjot Gill 45:06 They can just keep running it against Ops Stack locally and optimize it. Optimize my CI cache or do this Bun migration. I mean, those kind of things are super easy right now. I mean, if you have good unit tests and testing-
Gergely Orosz 45:16 Mm-hmm
Harjot Gill 45:16 ... like, you can validate that, hey, things are coming up correctly and the smoke tests are passing. You can even try it on production. And usually things are working. It's not like we are shipping a lot more defects. Our defect rate, I would say, is going down. I mean, this might be a hot take, but, like, humans were writing slop, not AI. I mean, that's what I'm realizing. Like, AI is actually writing less slop at this point if it can validate against something.
Gergely Orosz 45:36 So you're saying that in the past we shipped, like most teams would, would ship kind of slop or low quality stuff when we didn't validate it. And to be honest, if, if we're just being honest, we didn't always validate it, right? And we shipped it.
Harjot Gill 45:47 We had more outages in the past. I mean, that's my observation. I still have to measure it, but I would say when we were shipping with hand.
Gergely Orosz 45:53 And you're talking about just CodeRabbit, right?
Harjot Gill 45:54 Now I'm seeing actually fewer outages. The more our-
Gergely Orosz 45:57 Mm
Harjot Gill 45:57 ... code base becomes agent ready, the better validation layers become. We are seeing less defects, not more actually, as the velocity is going up.
Evals
Gergely Orosz 46:05 So one more thing about quality is Evals. Because of your business of, of, uh, offering a tool that gives you AI code review, model quality will be really important and cost and all of those things. How do you evaluate new model releases? And I'm obviously talking about the big ones, the OpenAI, the Anthropic, b- but also open models and so on. 'Cause I assume it's kind of a big deal for you to know how good of this model is in terms of the quality, the cost, latency and so on.
Harjot Gill 46:34 That's right. I mean, right now we are also bottlenecked on our Evals capacity. Like we're still like a-
Gergely Orosz 46:39 Oh, how so?
Harjot Gill 46:39 I mean, the models, the cadence that the, they are coming out every few weeks now. [laughs] With-- Because the RL thing has taken off, right? So people are coming up with back to back releases. The training runs are shorter and shorter, right?
Gergely Orosz 46:50 Mm-hmm.
Harjot Gill 46:51 Which means that, I mean, every day we are, like, evaluating something and they're like, of course, three or four classes of OpenAI models, Anthropic models, and we have, like, an ensemble of models in our product. So it's, it's like unlike other products, we don't let the user drop down and choose the model. Like, we do it for them. We know which sub-agents, which workloads the model is going to be a good fit. So a lot of, uh, the time that our applied AI team spends is figuring out the price to performance of each model and the workload, to the point that we have zero bandwidth to even onboard another vendor. Like, we are a big token spender. Uh, we have been working with like even you can see the Gemini's and all, but we don't have bandwidth. That's been a big constraint. Even with the open models, I know they're getting good enough. Each time we think that, okay, we could go use these open models, OpenAI brings down their models in price. Like, Luna is a great model.
Gergely Orosz 47:39 Mm-hmm.
Harjot Gill 47:40 Like, given the price point at that they offer-
Gergely Orosz 47:41 Sounds like you're using it a lot.
Harjot Gill 47:43 Yeah, we use a lot of Luna, [laughs] like, in our product. We use other models. We use expensive ones for deep inference reasoning, but a lot of code exploration, a lot of the context preparation is done by cheaper models these days, right? Uh, but the, the, the trick is, like, the open source is lucrative. The only thing it's helping is, like, putting downward pressure overall on the industry. We ourselves didn't get, like, much time to go and evaluate those.
Gergely Orosz 48:04 Do you build your own models? Do you either fine tune or, or, or do some, like, post-tuning, or have you experimented with this?
Harjot Gill 48:11 Early days we did, and that was a interesting realization, that the frontier was moving faster than we could maintain those fine-tuned versions.
Gergely Orosz 48:19 Mm-hmm.
Harjot Gill 48:19 Like, I mean, the shelf life of those investments were very short.
Gergely Orosz 48:23 Mm-hmm. So it, it was, it was a little bit better for a short time, and then a new model came-
Harjot Gill 48:28 Then, yeah
Gergely Orosz 48:28 ... and boom, your advantage was gone.
Harjot Gill 48:30 Yeah. Then you're in a different zone altogether, right? So, and the other thing is, like, we are in the coding space where I think other people have plenty of data. That's what they are relying on.
Gergely Orosz 48:37 Yeah.
Harjot Gill 48:38 I mean, if we, let's say, had really very different data stream, then maybe it makes sense because the m- m- models are not good at trained on that. But if you look in coding, code review, they're trained to some extent. Where the real trick comes in is in fine-tuning the harness. There are two parts of fine-tuning. One is a model. I think the bigger problem is, like, what's the right context? Because these reasoning models can go off track if they overthink. I mean, you might have noticed Astra is actually sometimes worse at overthings, right? Same thing, like, I mean, depending on the context, uh, we use a lot of, like, semantic analysis of the code, code graphs, uh, to make sure the token costs are, like, lower while we also get very high quality leads to go and track. Um, so, so that's where, like, the effort goes in, is in fine-tuning the harness. And we have very large open source user base, so we are, like, very popular in public repositories.
Gergely Orosz 49:23 Yeah, yeah. You, you, you have this thing where you make it really easy for open source projects to use CodeRabbit, right?
Harjot Gill 49:27 Yeah. And big reason for that was the feedback loop because with open source, each review that happens every few seconds goes back into our pipeline to fine-tune the harness.
Gergely Orosz 49:35 Uh-huh. So, so, so basically just from a business perspective, I mean, you're losing money on every one of these 'cause you're paying for it, but you're getting it because now you're able to make your product better, which hopefully it will be more valuable on the long term, right?
Harjot Gill 49:50 That's right. The reason we have-
Gergely Orosz 49:51 And then, and then this, this could be a win-win.
Harjot Gill 49:52 Yeah. So the reason we have the price to... And this is, like, one of the very few use cases by different industry where a specialized harness is still applicable. Like, everything else is getting eaten by general purpose cloud co-
Gergely Orosz 50:03 Mm.
Harjot Gill 50:03 Even legal, finance, like, it's doing general purpose well. But code review is one area where it's a breadth and a depth problem. You need a good recall precision, and over there, like, a custom harness is very effective. And this is where, like, all this open source data is helping us tune that. I mean, I have couple of agents just running, looking at open source all the time. [chuckles]
Gergely Orosz 50:23 One more thing about quality
Formal methods
Gergely Orosz 50:25 is you, you mentioned briefly formal methods. I'm interested, when did you start to experiment with it? You mentioned TLA+ and, and Lean. How is it going? Where are you using it? Because when I talk with Hillel Wayne, who was on the podcast, a TLA, uh, expert, you, you will know him, he said that he thinks formal methods, especially with TLA+, is probably an overkill for almost all software, even with AI, although he did say this about six months ago, so the, the progress is not there. I'm just really curious on, on where are you seeing success.
Harjot Gill 50:55 So we started using it back in February or March when we built the Slack bot product. So that product had, like, some distributed states. Like, it, it had a queue, like, queue up the messages. You could steer the messages. It had, like, complex states around, like, shutting down the sandboxes. Like, there was a lot of state machine there on, on the warm, warm idle state and so on. So that's where I-- That TLA+ models, they're not great for, like, things which are, like, algorithms. They're great when it comes to, like, messages and actors and interactions.
Gergely Orosz 51:25 Mm-hmm.
Harjot Gill 51:26 And that's what, like, we used because at Phi-3 Codex wasn't as great as now Astra. Now I think it might make pure mistakes, but the way we made Phi-3 Codex successful was first model it, make sure the state machines were correctly done before writing the code. And we did find issues in the initial implementation after we deployed TLA+. It could find all these, like, race conditions which we had overlooked.
Gergely Orosz 51:44 Hmm.
Harjot Gill 51:45 Yeah, simulations also helped, by the way. So there were two techniques we invested back in February or March. We built, like, a entire, like, simulation harness where we could, like, fuzz and try, like, different, uh, interleavings. At the same time, we used TLA+ to model different states and explore those states, um, and then find bugs.
Gergely Orosz 52:03 Nice. And I want to go back to
AI cost at CodeRabbit
Gergely Orosz 52:05 the, the, not your commits, but the CodeRabbit commits. And this is the weekly commits-
Harjot Gill 52:09 Yep
Gergely Orosz 52:09 ... which are kind of similar to, to yours except it's more, more granular. You can see, uh, but there's a huge inflection point in August, very similar to, to how you went. It, it, it went from about less than 1,000 commits per maybe, like, 500 or 300 in July to more than 5,000. We're talking weekly commits at the la-last data point. I wanted to ask, what about the cost of, of AI tools? You, you obviously have AI code review tools.
Harjot Gill 52:37 Yeah.
Gergely Orosz 52:37 You have the AI generation tools. Where is it at right now, roughly?
Harjot Gill 52:40 So our AI cost, I mean, I mean, I mean, mine alone would be 250K a month, 275K, something like that.
Gergely Orosz 52:48 Wow. You're, you're, you're really spending it.
Harjot Gill 52:51 Yes. I mean, that's just me alone. And some of the people on the team are shipping close to me right now, right? And a lot of the also, like, the number two code committer is actually the CodeRabbit's background agent.
Gergely Orosz 53:01 Mm-hmm.
Harjot Gill 53:01 So we also have, like, the agents looking at health and just creating PRs.
Gergely Orosz 53:04 Yeah.
Harjot Gill 53:04 Then we just take them to completion. So we are spending, like, several million dollars a month on tokens, and then we are also spending a lot on, um, the CI/CD, [chuckles] right?
Gergely Orosz 53:15 I wanted to ask about-
Harjot Gill 53:16 And the sandbox environments, um.
Gergely Orosz 53:17 Yeah. And, and it sounds like that was the CI/CD was on track to be a million a month, but you now brought it in-house, so you-
Harjot Gill 53:23 Yeah
Gergely Orosz 53:23 ... probably decreased it up a little bit.
Harjot Gill 53:24 50% roughly is what I'm hearing. I have to see, like, but we just shifted, like, over the weekend, this weekend all, all, all to the local CI now.
Gergely Orosz 53:31 Okay. So for almost everyone else, this would absolutely freak them out, like, having this thing and they would be thinking, "How can I rein it in?" I also heard a lot of, uh, I did research on, on startups on how their costs went, and for most of them, the costs spiked in August with Opus 5.5, and then it went down. Most companies are, are kind of just using it for coding. They're not running as many loops as, as you're doing. How are you thinking about the, the cost of AI and why you're not nervous as, as a CEO and a founder? [chuckles]
Harjot Gill 53:58 So I think you're gonna use both kind of models like this. So if you have a good model routing layer, then of course for some of the, most of the tasks you can even use cheaper models. But some of the things you still have to bring in the expensive model. The other aspect is the fast mode, like you pay double with ultra-fast. Those are really great, right? I mean, as a startup, you wanna get there faster than the next competitor, right? Then the thing is, like, even though AI is getting commoditized, but it's now driving the second order effect of, uh, the, the CI cost and the CPU crunch we are seeing, right? I think that's real. That's probably not gonna go away unless, like, you are getting rid of unit test or finding some other way of validation, like some people are doing local validation and using that, um, kind of attestation to, like, pass the central CI. There could be some tricks, but the fact is, like, these software factories are going to be expensive, right? Earlier when I d- wrote that tweet, I know that I got a pushback on that tweet.
Gergely Orosz 54:49 Yeah. We can, we, we, we, we can talk about it. Let's-
Harjot Gill 54:51 Yeah
Gergely Orosz 54:51 ... let's maybe talk about this tweet. So this tweet, I'll also put it on the screen, but I'll read it to you. You quote tweeted, uh, this,
Startup funding in 2026 and the value of AI spend
Gergely Orosz 54:57 uh, founder engineer who shared how they're spending $32 million per year in cloud spend building this massive multiplayer game. And you said that what we're seeing at CodeRabbit and elsewhere is this parabolic trajectory that even large seed rounds like $8 million plus are no longer enough to build a software business. Software factories need quality tokens at large scale compute to run CI sandboxes and other infrastructure to ship software at scale. At this point, some smaller startups exist only to funnel ideas to larger startups with more tokens. There's a lot of spicy stuff in here, so-
Harjot Gill 55:24 Yeah
Gergely Orosz 55:24 ... but let's talk about, I know X is not necessarily great for capturing the nuance, but, like, what is-- You had an observation there. You said that you're seeing this trend. Is it startups spending a lot on software factories?
Harjot Gill 55:37 I think that's gonna be inevitable. There are multiple, like, software businesses you can run, but if you're looking at hyper-growth, VC-funded startup in these competitive spaces, you would probably need more m- capital right now because now your bottlenecks have shifted. Earlier, you needed to, like, pay mainly for the talent. So you'll get, like, ex-Netflix, ex-Google, ex-Facebook engineers. They themselves will be founders. They will hire from the same pool. Give them nice talk, some better g- really good laptops, and that's it. Like, it was all, like, the brain game, like the IQ was the people, right? They, their talent.
Gergely Orosz 56:06 Yeah. And then, and you-- but you needed to pay for them, right? In, in Silicon Valley, like, your, your-- the base salary would have been 300K plus-
Harjot Gill 56:12 Yeah
Gergely Orosz 56:12 ... and stock and, and all that. We're talking a million plus.
Harjot Gill 56:14 Yeah. The salary was the bigger piece, like-
Gergely Orosz 56:15 Yeah
Harjot Gill 56:16 ... in terms of running a business, like, in early stages. I'm talking about seed, series A.
Gergely Orosz 56:20 Yeah.
Harjot Gill 56:20 Even before you hire sales. Like, a lot of it is, like, the team, like the dev team, right? Now, what we're seeing is, like, the IQ is on the tap, like the tokens. Like, in fact, even the smartest people need to be driving tokens, and a single person like me, I can drive, uh, $2 or $3 million of tokens in a year. Like, blended models-
Gergely Orosz 56:37 Yeah
Harjot Gill 56:37 ... not just always using the best model. And perhaps it should even be more once we have the factory concept taking off. So my point here was, like, the bottlenecks are now shifting in how much you can, like, drive these tokens, number of agents, the CI, and the other infrastructure. Humans can make a lot more decisions in a day to drive millions of dollars of spend. Now you need, like-- It's almost like marketing team, if you remember. Like, I mean, 10 people can drive $100 million of spend. Same thing we're gonna see with engineers. Now, a small team can drive $150 million of spend, or $30 million in that company, right? And that's how you stay competitive because now you're gonna compete with everyone who's writing software in an ef- inefficient way, not in an economically viable way. If you're gonna mass produce, you're gonna see, like, companies, composite startups now spin up with which will be a brand, but with, like, many products they're gonna mass produce, like, just like how you do in a factory.
Gergely Orosz 57:28 I feel you're still a huge exception, and everyone I, I talk to, including at N-Nvidia, inside of Nvidia, there's a monthly budget of $10,000, and you can go above that, but most devs I talk to don't, uh, go above that. So I'll just play a little bit of devil's advocate, uh, with, with you. So you're spending a lot on, on AI, like way, way more than most. You're also a founder, so you, you have a really good idea of where the business is and, and what value it's adding. What value do you think you and people spending this much is adding in terms of, like, does this mean that you're clear you're moving faster, but, but does-- W-what, what is this resulting in? Are you getting more customers? Are customers spending more? Are you just, I don't know, getting more free user? You see what I mean? Like, because you're in a great position. Usually, for an engineer, it, it will be maybe harder to connect, but you can clearly connect, like, what either you're seeing right now or what you're expecting to see as a result of this investment and spend.
Harjot Gill 58:18 Yeah, all of the above. Like, one of the things you're gonna see is, like, this point, SaaS solutions, the days are over. Like, now you're gonna see really, like, companies doing a lot more software, right?
Gergely Orosz 58:27 Mm-hmm.
Harjot Gill 58:27 And this is adding value in all parts of our stack. First of all, like, I mean, as I mentioned, like, earlier, I would, like, rely on vendors for some of the things. Now you can just build in-house, much cheaper. There are many tools. Like CI, we moved in-house. We moved vector databases in-house, bunch of this stuff. Those are very clear areas you can, like, go and build those, um, differentiators quickly, right? And the other thing is, even for the market, like, the people have infinite cap- appetite for software and these solutions, right? So there are companies which are gonna, like-- For example, we are in the SDLC space, so all of our customers want to go and ship faster. Code review is not the only bottleneck they are running. Like, now they are on the market looking for the next thing, which is how do I solve my CI bottleneck? How do I do this? Like, so we're in that business, so we have to, first of all, go and learn quickly where the bottlenecks are gonna be. I mean, we're getting arrows in the back, front, whatever, right? But what that also puts us uniquely in a position is that we can offer these products faster to the customers. But even everyone wants to ship faster. Like, that's the thing with, with the, with the VC-funded companies, right? So that, uh, the, the kind of, like, the range of tech you wanna go quickly build, that's your usually your moat. Like, I mean, you have, like, a few months. Now, the problem with the, with the software right now is, like, even, like-- So a lot of these startups, they used to have a moat of a few months before they got copied, but now they're getting copied within days and weeks. So you can't just go to the market in this competitive market with a idea which is shallow. Like you-
Gergely Orosz 59:43 Mm-hmm
Harjot Gill 59:43 ... you, you get bliffed, like, even, um And, and Tropic's gonna copy you in, in, like, two weeks right now. [chuckles]
Gergely Orosz 59:49 Oh, uh, so I see where you're coming from, but I still wanna push back on the fact that my sense is, okay, you're spending this crazy amount of money. You know, you personally are, are, are spending or you're on track to spend, like, $2 million per year, or let's just talk about per month. Like, your, your-- You and the team maybe are spending 500K in this month. Maybe next month it'll be 600, maybe then 700. But in return, what you're getting is, you know, you, you are in the dev tool space with AI code review. You're now realizing, "Oh, dang, we also have this bottleneck with CI. Maybe we'll do something with CI or help our customers do something with CI, and we do this because we're now so fast. We're, we're seeing where they're gonna be in, like, six months or twelve months. So now we're now gonna start, start this product. We'll spend more. Then we realize maybe it'll be a feature flag, and we'll build a rollout," and so on. And by doing this, you're almost, like, doing R&D for your business, and you're also building your product. And the bet is that you're-- you'll be expanding your market. You'll sell more to customers. But a- what I'm wondering is, are you not a little bit unique that this might be true for anyone who's in the dev tools market, but for some, some company who's not-
Harjot Gill 1:00:50 In any other market as well. Like, so basically, now you're, you're moving in a compressed timeframe. Let's say it took you, like, a while to go to market and do a PMF, six, nine months if you're a YC company, right? Now you can, like, go there and try that idea quickly. And if it's a shallow idea, now the risk is you can get copied. So basically, we'll probably have to, like, do more, but you are now moving at a compressed timeframe. The earlier bottlenecks were in scaling the team.
Gergely Orosz 1:01:13 Mm-hmm.
Harjot Gill 1:01:13 Like, so I needed to have the talent. You will, like, get all this VC money-
Gergely Orosz 1:01:17 Uh-huh
Harjot Gill 1:01:17 ... scale the team. I don't need to scale the team at the same level, right? I mean, maybe the sales still need to scale because they have to do the meetings, but product marketing, engineering, you can now do few people burning a lot more tokens. Right? And this token spend right now, if you... And it's quality token. Like, you can get cheap tokens. It will take you longer to get there. If you have the quality tokens, you, your price per task, uh, will be high, but you get there faster at a high quality, right?
Gergely Orosz 1:01:42 Okay, so what I'm hearing is, is what you're actually saying with, with this tweet, which wasn't as nuanced, is if you are in the VC game, meaning you're raising venture capital 'cause you wanna build a billion-dollar company, which means you actually want to win your, your market. Like inside of dev tools, you might have a niche, may that be like AI code review or AI SRE or, or CI/CD, something like that. If you're starting today, you will probably need to raise a larger amount because you need to win this quickly. The, the game will be over pretty quickly, and you want to get meaningful traction before the big guys, Anthropic, OpenAI, can maybe copy you.
Harjot Gill 1:02:19 Right.
Gergely Orosz 1:02:19 Oh, and, and as, as a added benefit, you can have a small team to start with, and you don't need to worry too much about hiring. By the time you would hire anyway, you would want to build some solution, that point of... and get customers and so on. So, like, everything is just compressed-
Harjot Gill 1:02:31 Everything is compressed
Gergely Orosz 1:02:31 ... and you can pay more.
Harjot Gill 1:02:32 Right. I mean, the attention is compressed. Like, if you don't get the mindshare after building your MVP in, like, maybe one or two months, that's the window you have now-
Gergely Orosz 1:02:40 Mm-hmm
Harjot Gill 1:02:40 ... before someone just gobbles up that idea and runs with it.
Gergely Orosz 1:02:43 Okay. So that is-
Harjot Gill 1:02:44 But yeah
Gergely Orosz 1:02:44 ... that is pretty brutal. [laughs]
Harjot Gill 1:02:45 Yeah, I mean, that's the situation right now we're dealing with, like the, the, the window of opportunity to get the mindshare is become shorter.
Gergely Orosz 1:02:52 Mm-hmm.
Harjot Gill 1:02:52 It used to be that you had smart people, they had built something, right? And maybe it was a lot of product iterations, it was a, I don't know, SaaS product with a lot of, like, craft and, and care or it was a complex backend system which just had a moat, like people couldn't replicate easily. So they would have like a shelf life of, I don't know, six to nine months before something... And, and they had that, their breathing room to get the mindshare. But now it's like few weeks, unfortunately.
Gergely Orosz 1:03:17 Mm-hmm.
Harjot Gill 1:03:17 And pers- and, and the company that's, like, spending a lot on tokens and all can get there faster. So you're gonna see more companies like Rippling, which was a composite startup. You're gonna see a lot more composite companies with these very efficient factories and the processes gobbling up a lot of that software market. You will still have, like, people tailoring at the last mile, but if you're buying these SaaS solutions and all-
Gergely Orosz 1:03:36 Mm-hmm
Harjot Gill 1:03:36 ... that's where we're gonna see a big disruption.
Gergely Orosz 1:03:38 Okay. No, I, I, I think this is fascinating insight, and I mean just a, a caveat. Like, you're seeing this in Silicon Valley where everything is accelerated, always has been compared to rest of the world. Let's say, you know, Germany will be always a lot slower, right? [laughs]
Harjot Gill 1:03:49 We've seen that with the Industrial Revolution, same thing happened. Like, I mean, one example is, like, for example, toys. Like, an example. Like, earlier people used to make metal toys.
Gergely Orosz 1:03:57 Mm-hmm.
Harjot Gill 1:03:57 And very good crafty. When I grew up getting those Matchbox car, like all metal, very good quality.
Gergely Orosz 1:04:02 Oh, oh yeah, the old Matchbox. They were a lot heavier, right?
Harjot Gill 1:04:03 Yeah. I mean, when I was a kid, like, they were very heavy-
Gergely Orosz 1:04:05 Yeah
Harjot Gill 1:04:05 ... really good quality, but now we are seeing a lot of plastic.
Gergely Orosz 1:04:07 They're just all plastic.
Harjot Gill 1:04:08 But mass produce, the market has appetite for that. So yes, the quality is going to go down, right? But the companies that are going to be able to invest in these factories which need now actually as much investment as running a factory. It's almost like building a steel plant now. 50 million out of the gate are gonna mass produce software.
Gergely Orosz 1:04:26 But just to push you a little bit more on that because I, [laughs] I really wanna push, push you on this here. The- these software factories probably cost so much because there's not a turnkey solution, and let's say may- given it's, everything is so fast, in a year there might already be software factories you can buy. But you're saying that by that time, I mean, first of all, e- everyone can have access to it and your window of opportunity might have closed for a lot of businesses.
Harjot Gill 1:04:47 Software factories are expensive. Right now tokens, but then CPU, all the, um, like we talked about the CI. Like you-
Gergely Orosz 1:04:53 Yeah
Harjot Gill 1:04:53 ... can drive a lot of spend just with the velocity alone. A single person cannot drive it, right? So, so the economics are gonna force you into that factory model. It's almost like how like we've seen in the industrial rev- Even, like, like, let's take another example, like watches. Like when the quartz watches came out, you could mass produce.
Gergely Orosz 1:05:08 Yeah.
Harjot Gill 1:05:08 And then you pay for Rolex, which is crafted. Now, the question is, like, would artisanal software, which is very well-crafted, people took a lot of time, will command a premium or is the market just going to be okay with generic-looking, maybe slightly lower quality software, which I think is what the case is, and that software is going to be designed in a factory.
Gergely Orosz 1:05:26 Yep.
Harjot Gill 1:05:27 Right?
Gergely Orosz 1:05:27 No, I, I, I see where you're coming from.
Harjot Gill 1:05:29 Right.
Gergely Orosz 1:05:29 And I, I, I really want to disagree with you in a lot of ways as especially, but I, I see your point and I think I have a hard time disagreeing with the fact that, look, if you do wanna win a market right now, you will want to give yourself the best time, uh, the best opportunity. And, and if, if the th- if the theory is that to win, you, you wanna be e- either, either first or fast and iterate and get it right, you have less time because an interesting thing that I'm also seeing, which, uh, which is, uh, one more thing we could reflect on, is so far we've only talked about, we assume that your competition is either the big labs who, you know, see what everyone else is doing and they launch something, or the other startups who some of them raised more money and might be faster than you in their software factory. But the reality is everyone else is a competitor, like Ramp launched their coding agent.
Harjot Gill 1:06:13 Mm-hmm.
Gergely Orosz 1:06:14 Spotify is launching a platform for feature flags. So you have the existing tech companies who are also realizing, hang on, we built this tool inside because we're now big and we see and we think there might be an appetite. And what you said is every small company, like 50-person company like yourself, will need the tools that Netflix has. Netflix at some point might say like, "Hang on, let's, let's open source this." So now I feel there's, there's just a lot more competition for the same thing. Like just, just coding harness. Again, one would've, would've you thought that a credit card company would have a coding agent? And people take it seriously.
Harjot Gill 1:06:43 Right. And think that whoever is not on this parabolic trajectory you just saw, they're gonna get disrupted.
Gergely Orosz 1:06:49 Mm.
Harjot Gill 1:06:49 A lot of people are not even ready for what we're seeing already.
Gergely Orosz 1:06:52 Mm.
Harjot Gill 1:06:53 Like, their, their, their culture is not gonna let them. So the way they're doing software development, like they have to rethink everything.
Gergely Orosz 1:06:59 Mm.
Harjot Gill 1:06:59 Like, first of all, how many people they need. Like, people who are just good at writing code, they should probably look for other jobs if, if they don't know what to build. Like, people who are not software engineers, like artists, people who have taste, maybe should start coding. And then how do you allocate budgets? Like, you have to now really think about if you're hiring someone at, like, I don't know, half a million a year, expect they're gonna spend more than a million on tokens now. So earlier, like, your, like, when you do, you used to do this plan to show the VCs, you will have the salary as a main component, but now we're starting to show tokens, right?
Gergely Orosz 1:07:29 Yeah.
Harjot Gill 1:07:29 And-
Gergely Orosz 1:07:29 Well, uh, uh, again, I just wanna be clear, like, th- this is for, like, a new startup VC building from zero, because for existing companies that have a business, often, like, physical or in the world, okay, they, they could be disrupted, but you know, they have something stable to be there. And they, they're the companies who are spending the $200 max plan for everyone, right?
Harjot Gill 1:07:43 I mean, if you have a hardware business and all, you have, probably have a better moat at this time. But if you're, like, just software with very high margins, those margins are gonna compress, and you probably also have to, like, rethink. I mean, if you're not on the parabolic, then companies coming from behind. Now, this exponential is scary. I mean, these kind of things start world wars, right? [laughs] It's that scary. So that kind of companies are gonna start coming from behind and start decimating some of the slower dinosaurs, unless they go and transform themselves on this parabolic.
Gergely Orosz 1:08:09 Mm-hmm.
Harjot Gill 1:08:10 And those companies, yeah, they're actually investing in factories more than talent. They're investing in just, I mean, paying for the best model possible. They're not cutting corners, they're just going fast.
Gergely Orosz 1:08:19 Let's talk a bit about
Hiring
Gergely Orosz 1:08:20 hiring, 'cause this is also a very, very interesting topic. First, can you tell me, how did CodeRabbit grow over the years, and how has your hiring changed? 'Cause you did mention that you're hiring a lot slower, right?
Harjot Gill 1:08:30 In some areas, yes. In many areas, we are hiring rapidly. Like, sales is one area where we are really double down-
Gergely Orosz 1:08:36 Mm-hmm
Harjot Gill 1:08:36 ... on the field sales.
Gergely Orosz 1:08:37 Mm-hmm.
Harjot Gill 1:08:38 Because one of the things we found with, um, generative AI, and is that the last mile is still a problem. Like, people still need a lot of hand-holding-
Gergely Orosz 1:08:47 Mm-hmm
Harjot Gill 1:08:47 ... in how to set up the systems in the right way.
Gergely Orosz 1:08:49 Yes.
Harjot Gill 1:08:50 So those kind of things you can't accelerate with AI. You have to have those meetings, you have to have conversations. So those areas we are hiring rapidly.
Gergely Orosz 1:08:56 Yes.
Harjot Gill 1:08:56 But other areas now we are, like, rethinking what we need to hire for and everything.
Gergely Orosz 1:09:00 Well, let's talk about software engineering specifically.
Harjot Gill 1:09:02 Software engineering is the most trickiest area right now. Every month we are learning something. Like, one of the things we have found is, like, the problem-solving skills, system skills. Like, people, like, who came from, like, even I wrote a tweet about, like, I'm looking for people who have played strategy games.
Gergely Orosz 1:09:16 Let me read this tweet, 'cause that was really interesting one. So, so you're, you're, you are hiring and you said, [laughs] I mean, there's a KBIT. "We're hiring. The person who applies must meet one of the following two criterias to join: either has tastefully shipped at least 1,000 PRs in the last 30 days, or has played at least 500 hours of simulation games such as Civilization, Age of Empires, Transport Tycoon, Factorio, et cetera."
Harjot Gill 1:09:40 Almost like now if you ask me what does software building looks, feels like, it's like playing one of those games, right? Now, I mean, I talk about Civilization, like, each, like, you basically make the moves and each turn is like a release for me, right?
Gergely Orosz 1:09:52 I, I remember this. Yeah.
Harjot Gill 1:09:52 And you learn something in the next turn. And while you're steering the civilization, you still have, like, I don't know, workers improving your roads and building fields and so on. That's how the agents are. And each turn you're, like, just making these agents work on something meaningful, right, which we think is, is a great ROI for your civilization, right? Um, almost like now running these agents feels, to me, feels like building, if people have grown playing strategy games, I think they will be very successful in, with, with these agents right now.
Gergely Orosz 1:10:20 Hmm.
Harjot Gill 1:10:21 They can multitask. They have, like, range of things they can... I mean, at least they have a good problem list these agents can then work on.
Gergely Orosz 1:10:26 And how is hiring going for you so far? How easier, more, like, hard are you finding to, to hire these folks who have this kind of experience?
Harjot Gill 1:10:34 So the way we are looking at is, like, either someone who's already a tenured senior engineer at a good place, they have to come and first unlearn, right? Or you hire someone even who's, like, fresh out of college, but they have high agency but can run, learn faster. What we found that the people who are, like, coming with, like, who are not set in their ways and have come, especially from the background of like, like, playing these, like, strategy games and all have been very successful. That's just my take on it, and that's what we are looking for. I think the human agency now matters a lot more than it ever did.
Gergely Orosz 1:11:05 Hmm.
Harjot Gill 1:11:06 Person who was 10X, knew what to build, is now suddenly becoming 100X with these agents, right? Because now you at least have a problem list or the, the most impactful areas these agents can work on, and you could just unleash these agents as much as po- possible in parallel.
Gergely Orosz 1:11:19 So we talked about individuals, but I know you have a concept called pods that you also call it as war rooms.
Harjot Gill 1:11:24 That's right.
Gergely Orosz 1:11:24 Uh, what are war rooms?
Harjot Gill 1:11:26 So what we found is, like, that's another area, like, I think we have to, like, the collaboration has become even more important. So I've, like, in previous startups, done remote teams. We are, like, we're building distributed systems, so it doesn't matter, like, teams, like in India or Eastern Europe, I've done all. Um, but now, like, what we are seeing is that now we are moving so fast, like, some, one, one person is on the steering wheel, but other people are shaping the software as it's getting built. And everyone is able to touch the software, by the way. Even product managers get to touch it, like, with, and the kind of PRs they're making is relevant to, like, change a copy on something or some, um, information architecture flows, right? But everyone's, like, touching the software at the same time, and they, it's great if they can be, like, in a war room setting. So we kind of create these war rooms where we share, like, plans on a Slack canvas. We all agree on it, and we all collaborate in a room in, in a more face-to-face setting and building real time.
Gergely Orosz 1:12:12 So, so you just get, you get everyone in, in a room, into a table.
Harjot Gill 1:12:14 It's all real time.
Gergely Orosz 1:12:14 You kind of close the door and, like, all right, let's solve the problem, right?
Harjot Gill 1:12:17 And we are crafting the software real time. It's like sometimes you will, like, lay the foundation by running a goal, but when you're, like, at the stage where you want to, like, really craft the UI end experience, you wanna get people in the room and really craft it live.
Gergely Orosz 1:12:27 Yeah, it's funny. At, at Uber, we used to have up to 2017 something called the war room, which, which is like this.
Harjot Gill 1:12:32 Yeah.
Gergely Orosz 1:12:32 There's an important project, get everyone in, sometimes fly people in because they're distributed.
Harjot Gill 1:12:36 Yeah.
Gergely Orosz 1:12:36 And then spend maybe a week there, build up. This was pre-AI, so with AI it's a lot faster. But just build a po- bu- proof of concept, whiteboard, code, and then have a prototype, and then everyone goes back to their teams and just kind of builds it from there. But it's now working, the, the core thing.
Harjot Gill 1:12:50 Yeah, now we're doing that more often. Like, that's exactly the idea. And you need, like, people with different tastes. Like, you have product managers in the same room, you have designers-
Gergely Orosz 1:12:57 Mm-hmm
Harjot Gill 1:12:57 ... you have people who understand back end, the front end, and they are, like, crafting the software together.
Gergely Orosz 1:13:01 It, it is hard work and it's sometimes
Work pace, sleep, and burnout
Gergely Orosz 1:13:03 it is late work. 'Cause one of our first interactions on, on X was I, I, I think I was, I was writing something about how, like, teams are working really late up, up to the evening or un- into the night, and you replied, "It's t- it's almost 2:30 AM and we're still shipping." And you're like, "Yeah, yeah." Like, like a, like, you know, like crying smiley face. [laughs]
Harjot Gill 1:13:20 [laughs]
Gergely Orosz 1:13:20 I'm not sure this is, is good or bad. But can you tell me a, a bit on On the pace you talked about, you're not sleeping as much. Uh, before the recording, we talked about how, like, okay, at some point you need to get that under control 'cause clearly there, there's a time for excitement, but also when you're thinking of building something longer term, like you'll, you'll wanna find the balance, right? How are you thinking about it?
Harjot Gill 1:13:39 Yeah, it's crazy right now. Like, I mean, it, it does like impact your health to some extent because ever since these agents gotten better, as I said, like there's like a huge opportunity cost if they are not working all the time. And they do stall. They have questions. You have to like steer them. Um, I read the reasoning traces. I guide them whenever I have to. And for that, like, I mean, I've almost like changed my sleep cycle. I mean, I take a nap, I wake up at like 3:00 a.m. to send like a few instructions because every hour, like one or two agents are gonna get stuck.
Gergely Orosz 1:14:05 Mm-hmm.
Harjot Gill 1:14:05 And you have to go in like... It's almost like a Roomba's getting stuck. You have to like unstuck them. That's what it feels like. But then, then that's kind of become a habit, but it's not sustainable, and we haven't like figured out the right interface to work with these agent, fleet of agents at scale. Maybe it's really like a factory, like companies have to have a morning shift, night shift because instead of owning the code, maybe you're now owning agents. During the day, I make the decisions, but I can be rest assured we are seeing the same session, the same agent that other people can make decisions while I'm sleeping, right? So we are, we are right now because we're developing on our own machines, we are like bottlenecked on, on, on that collaboration right now.
Gergely Orosz 1:14:39 Yeah. The, the, the one thing I'll add is I wrote this book called Why We Sleep. I read it a, a while back, and it actually forces me to prioritize sleep because there's this thing with sleep which there is some scientific evidence, I'm, I'm not an expert in this, but there's a suggestion that not getting enough sleep now, it's fine for tomorrow, and it's fine for next week, and it's fine for next month. But decades later, it can come back and bite you. Uh, and I, like ever since, I don't know how scientifically sound it is, but just as when I was like, "Oh, it, it could happen," I'm like, okay, so like me getting a good night's sleep is not only important for me for tomorrow 'cause I'll be, you know, like you'll know, like you, you, you won't do as well. But for my long-term health, like I, I, you know, I, I'm starting to hopefully try to do stuff that will make me more healthy when I'm 60 or 70, et cetera.
Harjot Gill 1:15:29 Yeah.
Gergely Orosz 1:15:29 So that's my only kind of like addition to like not sleeping. And I, I, I, I would push a little bit like is, is it as important to wake up and, and steer the agent? Can it not wait until the morning?
Harjot Gill 1:15:39 So I think it's more like a polyphasic sleep. I, maybe I'm like exaggerating the health, but I think even if you look at how people used to work pre-industrially, this nine-to-five thing is, is a recent thing. And I mean, I come from like agriculture community, so people will wake up, tend to livestock because you are like working based on other aspects and you have to wake up-
Gergely Orosz 1:15:57 Oh, I see
Harjot Gill 1:15:57 ... and do that thing.
Gergely Orosz 1:15:58 Yeah.
Harjot Gill 1:15:58 Now it's the agents. You have to wake up to tend to that livestock, which is now agents. Almost like FarmVille. I don't know whether that's the right analogy.
Gergely Orosz 1:16:05 [laughs]
Harjot Gill 1:16:05 So people used to have fal- polyphasic-
Gergely Orosz 1:16:07 Or, or like Tamagotchi for a little bit, right?
Harjot Gill 1:16:08 Yes. So people used to have polyphasic sleep, and I'm doing that right now.
Gergely Orosz 1:16:13 Oh, you're right. Like th- this was a thing in the Middle Ages. People would wake up in the middle of the night.
Harjot Gill 1:16:16 Yes.
Gergely Orosz 1:16:17 And they would often like dine and stuff like that and then go back to sleep.
Harjot Gill 1:16:20 That's what I'm doing right now. But I would say the only reason I'm doing that right now is because I have to tend to the agents which are on my local machine. Now, once you've really figured the right abstraction of the factory, once you move to the cloud, then maybe it's gonna be like a factory, like nine to five, morning shift, evening shift, or like around the clock because these are not physical factories. You can have follow the sun model. Like, we're still figuring out, but right now it looks, looks more like polyphasic-
Gergely Orosz 1:16:43 Mm
Harjot Gill 1:16:43 ... with this generation of agents, at least for me. [laughs]
Gergely Orosz 1:16:46 Yeah. But let's talk a little bit about burnout. You wrote about burnout, and I'll quote you here. Uh, "CodeRabbit is my third startup now, and I have worked just as hard for my previous two, uh, Netand Flux Ninja. Burnout never happens due to hard work. It kicks in when your work does not lead to the desired outcome for a long time. On the family front, my two daughters were born during startups. They actually help prevent burnouts." Uh, in just your personal experience, like how have you thought about burnout and also just working so damn hard? 'Cause it, it sounds like, you know, you're, you're on third startup. Like you don't seem to be the, you're the intense type.
Harjot Gill 1:17:18 Yeah. I mean, I mean, I really, when the second startup wasn't doing well, I mean, that's where I did feel dragged down, burnt out because there was no outcome. That's the first time I felt the signs of a burnout, um, because yes, you were like working very hard, but without a very clear result or an outcome, just waiting. The anxiety kills you because nothing's happening. You're doing all this work, but nothing's happening. Nothing's moving. So that's a bad scenario to be in. I think that's a status quo which, which I would not ever want to go back to. But for CodeRabbit, like I'm working hard, but things are on the upside. Like we are like seeing the revenue grow like crazy. Our team is growing. The funding's coming in. It's always something happening. We are actually moving the needle as well. Customers are loving the product, the new capabilities. So even though I'm working hard, but I don't feel that I'm burning out, I can, or I have to stop at this point, and I think I can keep going on.
Gergely Orosz 1:18:04 And I, I wonder like, I think in your case it makes a lot of sense 'cause you were founder in all the companies, so you immediately, you, you were connected to the outcome or, you know, for example, if the revenue goes up, you know this is, this is just good because it, it will help your company grow or raise the next round or get a sale or whatever. You know, like you know what will hap- what, what can come. I'm wondering just being empathetic for employees who are not founders themselves. They might have equity, but it will not n- be as, as big of a stake. Could it be that having autonomy, having visibility for the business, just knowing how their work is connected, that it might actually help? 'Cause in startups, people typically work hard and, and long, especially early stage. But I, I wonder if the, the autonomy here and, and seeing how work is connected to the things, that, that actually just helps, you know, get this kind of kick that you're, you're getting basically.
Harjot Gill 1:18:50 Yeah. It motivates people. Like, first of all, we don't force people to work like I do. I mean, I lead by example. That's what I believe. And even if people can follow to some extent, it's great. But I, I do what I do. Like, and I mean-
Gergely Orosz 1:19:01 Yeah
Harjot Gill 1:19:01 ... rather than managing the people, I'm on the ground in the trenches with the developers.
Gergely Orosz 1:19:05 Yeah.
Harjot Gill 1:19:05 If there's incidents, I'm always like first there to-
Gergely Orosz 1:19:08 Yeah
Harjot Gill 1:19:08 ... help them out, no matter what the time.
Gergely Orosz 1:19:10 Yeah.
Harjot Gill 1:19:10 And I think that does motivate people. They're all like happy that, okay, the leader's right. They feel like much more empathy to their life, right?
Gergely Orosz 1:19:17 Yeah.
Harjot Gill 1:19:17 Secondly, like I found like the people who join startups are looking for this adventure. Like it's a non-linear career trajectory, so they're living, uh, living like- Very predictable salary, RSUs in some big company, right? To get, like, cheaper stock at startups, but then they're looking at the upside. And when that happens, it's, like, even more inspiring, right? When the companies are on the upward trajectory. So I would say, like, when the people who go and join the startups are, are looking for breaking that status quo, which things can be very mundane and boring in, in a larger setting.
Gergely Orosz 1:19:46 Yeah, and you're right. Like, on, on, on a fast road startup, there are a lot of advantages. Like, either, you know, if the fast road startup goes great and you have equity and there's an exit, you could have life-changing amount of money, meaning that, you know, you can buy a house, you, you could have ed- education set up for your kids, and, and you now have the freedom to, like, either take a risky startup or do a boring job, whatever you want. And if it goes like this and it drops, for example, Fast was a typical example in the, in I think in 2022, which went bankrupt. You often, if the startup attracted really good people, you know, those people all go to different companies 'cause now their company shut down.
Harjot Gill 1:20:20 Yep.
Gergely Orosz 1:20:20 Uh, but there's also often this network, this camaraderie that, that was there, and they often call each other years later. They will-- Some of them will end up at the next, you know, like, really successful company. So I've actually changed my mind and, okay, the only downside, I guess, for joining a startup is often you will need to work hard. It will be often be in person. But typically, when you're young, uh, and when, you know, this is not-- you, you don't have other huge priorities on your life, it can be a huge career boost.
Harjot Gill 1:20:46 It is. Like, even financially, I wouldn't say it's crazy like that. That's how the Silicon Valley is. Like, I think it does reward people who took that kind of a risk. I mean, there's no fatal failure somehow.
Gergely Orosz 1:20:55 Yeah.
Harjot Gill 1:20:55 Like, um, and I've seen, like, people, like, who have just been through that experience have gained a, a lot in, like, uh, even, like, if it's, like, a bad lesson, at least they know what is not gonna work.
Gergely Orosz 1:21:06 Yeah.
Harjot Gill 1:21:06 And that's valued in whatever the next job they're going to go look out for. So it is-
Gergely Orosz 1:21:10 Yeah. You're, you're kind of saying it might be more valuable than s-sitting in one place, not doing much, not learning much, right?
Harjot Gill 1:21:15 Yeah. That's where you don't grow. But if you go out and do this thing risky, I mean, there's no fatal failure there. Like, the-
Gergely Orosz 1:21:19 Mm.
Harjot Gill 1:21:19 The people love, in, in the Valley at least, people who have taken risk, whether you're founders or early employees are foolish enough [chuckles] to join. But, but at the same time, people reward that experience, right? And if it's successful, then it's like you, you're on an accelerated trajectory, like. And, and I think that's what I would encourage. Like, even though, yes, I mean, it's becoming a little bit cost prohibitive on the factory front, but still, it's still the best time for people to go out and try this out. Like, before all these, like, factories become more established and start blocking some of these individuals. But right now, it's a very magical time right now. Like, even a single person just driving these agents in the right way can create immense value.
Gergely Orosz 1:21:54 And I, I wanted to ask,
Skills for the AI era
Gergely Orosz 1:21:55 looking back pre-AI to, like, your 25-ish years as a software engineer, as a founder, what are the things that you learned there or, or skills that you built that you think have-- are really useful and helpful right now working with, with AI and in this, like, I guess, AI era?
Harjot Gill 1:22:10 I mean, in terms of, like, entrepreneurship, it's always been like-- I mean, again, it's almost like a video game. [chuckles] Like, I mean, you're playing all these, like, games where, yes, you have to think about, like, the product market fit. I mean, all that is exciting. I think what I love is, like, creating value. Like, that's end of the day. Like, software is just means to an end. Yes, you have to write code, and I enjoyed it, but it was, like, just means to an end for me.
Gergely Orosz 1:22:31 Mm-hmm.
Harjot Gill 1:22:31 Like, the real thing was, like, are people-- Like, it's really, like, really like making someone happy with the value you provide is always, like, the biggest reward, right? Everything is, like, then follows. Like, you get-- You will get the VC money, you will get the team, and people will join the mission. So, so it's almost like being a missionary, uh, in this kind of a world. Like, you're spreading a religion, and that's the fun part of doing a startup. Like, I mean, I l- I mean, I've always been on the frontier. Like, I've seen, like, early telecom. I've seen SaaS, cloud take off, and each time it's always what excites me is like, okay, there's a lot of nonlinear value going to be created at the frontier than, um, at, at other places. And I always try to be at the frontier.
Gergely Orosz 1:23:07 What would you advise a software engineer who one day would like to work at CodeRabbit or at a place like CodeRabbit or just at a fast-moving place? What kind of skills, uh, should they invest in? What kind of practices, what kind of activities should they do outside of playing [laughs] 500 Evals of Factorio?
Harjot Gill 1:23:24 [laughs] Yeah, I think, I think the agency is the number one thing I'm looking for right now when we interview someone. Because right now, like, even if they're in a bigger company because now a lot is happening on the side, if they're not doing side projects today, it's a red flag. Because right now it's so easy to build-
Gergely Orosz 1:23:38 Mm
Harjot Gill 1:23:38 ... that people who have agency are actually building something right now. They can hold themselves back, and that's al-always, like, a great sign. If you have side projects, I, I love it, right? And the second thing we are looking for is, like, people, uh, who are, have a range of experience now. For the depth, you have AI. It can go deep, drill infinite to performance optimize something else. But if you have, like, a range of experience, uh, you can, like, build, let's say, distributed systems. Let's say you came in from one space doing, let's say, observability databases. Now, if you were to give agents, now you can do 10X of that. You're not just building that, you're building 10 other types of distributed systems. So the range also matters now. So those kind of things I'm looking for, right? I mean, it's like systems thinking, uh, whether they have range, um, in skills and high agency to keep these agents busy.
Gergely Orosz 1:24:25 As closing, what's a book or a paper you would recommend that, that you read and you found
Book recommendation
Gergely Orosz 1:24:30 it interesting or fascinating?
Harjot Gill 1:24:31 I think we're living in a black swan moment. I read this book. I mean, I'm not, like, a-- spend a lot of time reading books these days. I've just been, like, bui-building. Um, but I think we are in that, like, the black swan zone. Like, what we are seeing right now, uh, is-
Gergely Orosz 1:24:42 And what was the book?
Harjot Gill 1:24:43 I mean, this was, like, the Black Swan book. I think I'd read, like, I forgot who the author was, but it was really, really-
Gergely Orosz 1:24:48 Yeah. We'll, we'll put it in the show notes below.
Harjot Gill 1:24:49 Yeah, yeah, yeah. But when-- That was, like, pretty insightful of what we are right now going through is a kind of a black swan moment. Like, if you look at five years back, I would not have predicted that AI would get so good that it's, first of all, like, driving my cars, it's driving the software, and it's, like, even some of the capabilities we are still, like, discovering, uh, with what these models are capable of doing. And we have not even hit the peak yet. So yeah. So everything is gonna change, and, and, and, and some of the things we can't even predict today, the way we're gonna see the af-the after effects.
Gergely Orosz 1:25:18 Well, Harjot, thank you.
Harjot Gill 1:25:19 Yeah, thanks.
Gergely Orosz 1:25:20 Well, I cannot recall talking with anyone nearly as AI-pilled as Harjot is. To be clear, I was skeptical to start with when hearing about his $200,000 AI bill for a single month for a single person, and I remain pretty skeptical about several things. But Harjot explaining his thinking put a few things into perspective. Let's start with the part that I cannot agree with or recommend, Harjot's approach to waking up in the middle of the night to prompt his agents. He calls this polymorphic sleep, but to me it sounds like deprioritizing sleep all to unblock agents. This feels very inefficient to me, and I would rather look at longer living tasks in the cloud if you really want to be there to have your agents running at all the time, even when you sleep. But really, I don't see the obsession with agents always running. This is not a competition about utilizing your local laptop to always run at 100%. The part on how Harjot doesn't think a startup with $8 million of funding has enough money to build a software startup, I'm also skeptical. I get it that when Harjot assumes that all startups will send millions of AI like his startup does with 40 people, you burn through funding fast. I also get it that in dev tools it can make business sense to build new products like a custom CI really fast like CodeRabbit did. But outside of startups that sell AI dev tools, I just don't see it making sense to spend massively on AI just to be a little bit faster to market than your competition. The part I do get, though, is how much competition across AI dev tool startup is there today. It's not just CodeRabbit launching new products with their 40 person engineering team. It's also larger companies like Ramp or Shopify launching dev tools products, AI labs, and other large companies all trying to get the piece of the same market. So with this context, I do get why speed matters, especially for small startups, and why spending on AI to build stuff faster can make some sense. Finally, one practice I really liked is the concept of devs losing their driving license if they cause too many regressions. AI-pilled or not, I like this idea to force engineers to care more about reliability and shipping quality code. For more deep dives on software factories and engineering practices with AI, check out the link to Pragmatic Engineer Deep Dives below. If you've enjoyed this podcast, please do subscribe on your favorite podcast platform and on YouTube. A special thank you if you also leave a rating on the show. Thanks, and see you in the next one
Episode duration: 1:27:31
In the uListen app, the transcript lights up word by word as you listen. Get uListen
Transcript of episode 4Cjb7691GJ8