Skip to content

By playing, you agree to uListen's Terms and Privacy Policy and YouTube's Terms of Service.

Aakash GuptaAakash Gupta

The Claude Code Setup That Fixes Production While Engineers Sleep

Shuhao Zhang, co-founder and CPO of TinyFish, opens his real GitHub to show the agent pipeline that lets anyone there ship to production. Then he shows the ops agent he thinks will set teams apart.

SuhaoguestAakash Guptahost
Nov 19, 20261h 2mWatch on YouTube

Summaries, chapters, transcripts, and FAQs on this page are made by uListen, not YouTube.

Great conversations run long. Their best ideas don't have to.

uListen is free on iPhone, iPad, Android, Chrome, and Claude.

See how

Transcript

  1. 0:00 – 1:32

    Intro

    1. SU

      When agents can generate tens of thousands of lines of code in a blink, the human reviewer becomes the bottleneck.

    2. AG

      Tinyfish.ai is a startup that has been on a hockey stick growth path. They raised $47 million from ICONIQ. They went from 100,000 to 200,000 users in 18 days. They went from 200,000 to 300,000 users in just 10 days.

    3. SU

      This system will allow anyone to ship to production, not just engineers.

    4. AG

      Why wouldn't you use, like, CloudCode Web?

    5. SU

      I noticed that CloudCode Web now has more features that I need, but this is something I built, uh, last year, when the features are not available out of the box from cloud. We continuously to improve our service, especially when we build search is a web infra company that will help agents to work with the web. There's a search fetch, and then there's web agents. But we really should consider agents as the new type of users, and there will be trillions of them.

    6. AG

      What are the mistakes you faced along the way building towards this? Before we get into today's show, please take a second to check that you're subscribed on YouTube and following on Apple and Spotify podcasts. If you want access to all of my favorite AI tools, I've gotten them to give you an entire year of their paid plans. Check out bundle.aakashg.com for an entire year of Bolt.new, Airtable, Speechify, Descript, Magic Patterns, Linear, Dovetail, Arise, and Mobbin. And now, into today's show.

  2. 1:32 – 6:52

    How TinyFish lets anyone ship to production

    1. AG

      Tinyfish.ai is a startup that has been on a hockey stick growth path. They raised $47 million from ICONIQ. They went from 100,000 to 200,000 users in 18 days. They went from 200,000 to 300,000 users in just 10 days. A big part of the reason, after I talked to Shuhao Zhao, their co-founder and chief product officer, is the way they build. They allow anyone to ship to production in the company. That's because they hired a bunch of amazing former Meta engineers. Shuhao himself was a former VP of engineering. And they built the system that allows you to ship features. And so they've invested in the coding agents, the coding review, the CI/CD system, the... Everything that you can think of from how you start a PRD all the way to how you measure the results of a feature, that allows you to do that all via agents. And so this is the next-gen way of building products. And in today's episode, Shuhao is gonna walk you through all the steps of that life cycle. He's gonna actually pull up his real GitHub and show you actually how he uses CloudCode so that you too can allow anyone at your company to ship features to production. Shuhao, thanks for being on the podcast. What do people really need to know about this system and how they can build it themselves?

    2. SU

      Let me show you. Yeah. So Aakash, today I'm gonna show you, uh, the whole setup, like what I use on day-to-day basis, uh, from building to shipping into production. And then now I'm actually building the part that let agent to operate and then maintain, as we just, uh, talked about. So, uh, this is my CloudCode setup. I have a dev server, and it's... This is a tmux session so that I can open it up in my mobile phone or on my browser. When I close everything, it still runs in cloud. And it's a, a full EC2 server, so it can install the tools and, uh, the whole stack, uh, as needed. So you can see I have multiple projects going on here, and each of them could be a feature or a new product that I'm building. And in this one, you see it's still, it's live, it's working, and this is a AI gateway product, uh, we call it Tiny Router, that will put, uh, uh, LLM models behind, uh, one gateway with a discount. So what it does is... This is already, uh, the first part is already built. I'm trying to land it into our sandbox environment. And, uh, I can talk about the whole setup.

    3. AG

      So maybe you can break down, for those of us who aren't as technical or don't have as much of a technical background-

    4. SU

      Mm-hmm

    5. AG

      ... some of the things that you just said in the sort of, quote-unquote, "for dummies" version. Dev server, what are we looking at there? Sandbox, what are we... What does that mean there? And then what is this software as well? If you can break down those basics for us, that'd be great.

    6. SU

      Yeah, sure. Let me, let me talk about it. So, uh, a dev server is essentially a, uh, remote, uh, server running in the cloud. It can be a EC2 instance in AWS or any server that you host, uh, or your own MacBook that has, uh, external connectivity. Uh, the goal for that is really that you can run your job, your workloads, uh, without you keep your laptop, uh, running, which is a local environment. And the sandbox is a environment pre-production. Is, uh, the setup is very close to production, but, uh, you don't have, uh, real user data. You have, uh, canned data for you to test your, uh, software before it goes live into production. The software I'm using is, uh, uh, I built it. Actually, this is, uh, built by CloudCode itself. Uh, the, it's called tmux, where you can stream a terminal session through the network, and this is secured, uh, protected by, uh, encryption. So yeah, this is, uh, my primary way of, uh, working with, uh, with my cloud.

    7. AG

      Why wouldn't you use, like, CloudCode Web?

    8. SU

      Yeah. I, I noticed that, uh, CloudCode Web now has, uh, more features that I need, but this is, uh, something I built, uh, last year when the features are not available, uh, out of the box from cloud. But I definitely see a future where, uh, it's, uh, you know, it's, it's just out of the box. I can just use it. But I also like the flexibility of owning my entire instance that I can, uh, install. For example, if I need my VPN access, right? I can install my VPN profile here. Uh, so there are still things that, uh, Cloud Web not Uh, flexible enough that I, I'd love to have the full control.

    9. AG

      All right.

    10. SU

      Yeah. So, uh, this is, uh... You can imagine this is the interface where I talk to the agents. But, uh, uh, as you can see, the... Everybody use Claude. You can see the UI is a lot of their own work log, but the valuable, uh, communication, clarifying product requirements, giving feedback, needs to happen in a different place. And, uh, what I do is, uh, Claude artifact is a great way of communicating. So Claude can generate its plan for me to review and I can, uh, type my response in the artifact. So, uh, I can show one of those artifacts to you. All right, uh-

  3. 6:52 – 8:44

    Ads

    1. AG

      If you've worked at any company bigger than 30 people, you know this one. The CEO sets strategy. By the time it reaches the people actually doing the work, it goes through three or four layers of translation. Half of it gets lost, and nobody finds out until the quarter is over. That's the problem Ariso is built for. It's an AI operating partner for every manager and team. It connects to where work actually happens: the meetings, the messages, the docs, and it turns all that fragmented activity into a clear picture of execution. Managers get real coaching grounded in their team's actual work, not generic advice. Teams stay aligned with strategy as it changes, not as it was last quarter. And leaders see where execution is drifting in weeks, not in the postmortem. One shared memory for the whole org. Everyone finally working from the same picture. If you lead a team, check out ariso.ai/aakash. That's A-R-I-S-O.A-I/A-A-K-A-S-H. Every growth team has a funnel they should be watching more closely. Usually, it's checkout. Amplitude custom agents will watch that funnel for you on the schedule you set. Let me show you. You start by describing the job. "Watch my checkout drop-off." The agent builder will ask a few questions to shape it, and then the agent connectors plug it into your stack. That part matters. The agent reads the PRD, the tickets, the team conversations, so its analysis actually carries the full context of what's happened. From there, it monitors the funnel on a schedule, and when it spots a problem, it acts. Right here, it files the linear ticket itself. The loop between knowing and doing closes on its own. That's the shift. Set up the agent once and it runs the work. Custom agents are free for all Amplitude customers in open beta right now. There's a library of pre-built agents too, including the exact checkout drop-off investigator. Start at amplitude.com/ai-agents. The agent library is linked right

  4. 8:44 – 14:54

    From the agent's plan to an AI-reviewed PR

    1. AG

      there.

    2. SU

      Uh, let me show one of the artifacts. So when, uh... This is... I usually ask my Claude kid to, uh, for any, like, bigger designs, decisions, to generate a plan and, uh, give me, uh, some recommendations, and also give me status update. So this is where I review the plan. Uh, I won't show the entire thing, but you get the idea. Where I can actually respond inline, and it's going to pick up my response and then carry on. So that's kind of pretty much the basic setup where you can talk to your Claude, you can review and, uh, clarify the designs. And, uh, I think this pretty much, uh, everybody has their own, own setup, and it work pretty well. What I'm gonna talk about more is how to push your product into production and then, uh, how to possibly operate it in production environment. And by the way, this is something that I'm, uh, also actively building and, uh, experimenting. So I'd love to hear your thoughts in the comments and, uh, I... You know, if I can update in another episode how things are working, I'd love to do it as well.

    3. AG

      Aren't we all? [laughs]

    4. SU

      So, uh, first of all, to get into our production environment, again, we start with the sandbox, uh, is to go through the code review process. And, uh, as a company, Tiny Fish, we have, uh, uh, quite a few engineers from Meta, so the code review culture is really strong. We hold... are real aligned with code very seriously. However, when agents can generate, uh, thousands lines of code, tens of thousand lines of code, uh, in, in the blink, the human, the human reviewer becomes the bottleneck. So... But how to solve that, we don't want to let go the quality control. We don't want to go the sec- you know, the second opinion, third opinion. So what we do is we onboarded, uh, Co- CodeRabbit. I believe a lot of users love them. Uh, but, uh, how to let CodeRabbit auto-approve your PR and you still stay compliant, you still get the back and the forth as you usually get with your engineers. So what we do is we let CodeRabbit to approve the PR and... but also, just like a human reviewer, it can reject a PR, uh, request for action, and then the agent will pick it up automatically, go back and forth. But if CodeRabbit detect a high risk and requires human judgment, it's going to loop in. The agent will go... will, you know, tag the right engineers to do a human review. But if CodeRabbit is happy about it, it's a lower risk, the PR will be auto-approved and auto-mer- merged. So I'm gonna show you. You can see this is quite extensive, uh, PR about a UI change, uh, which there are a lot of changes, a lot of files touched. However, it's, uh, considered a lower risk in general. Uh, but still, you can see there are quite a bit of, uh, responses from, uh, CodeRabbit, and let me show you a couple examples here. So it request for change, meaning that there are things that it's not happy about, and this is where you're gonna spend... pay attention to. Then you can see this is where it's... CodeRab- the rev- the AI reviewer consider things are not optimized or correct. And then this is actually my agents, uh, picking up the Response, uh, the feedback, and then, uh, make a correction. Finally, the AI reviewer, uh, approve it. So you can see there are multiple ones until both are happy with the results. The PR is merged in the end. So this is one of the examples where when you produce a ton of, uh, pull request and what's the fast and safe way to get them merged and reviewed and merged.

    5. AG

      So basically, you have an independent tool. In this case, you guys are using a third-party CodeRabbit. And CodeRabbit, what it's really doing is it's... has some context around your code base, your coding standards, what your engineering team likes things to be like, and it is acting as them and working with the Claude agent that initially submitted the pull request back and forth, back and forth until it's happy.

    6. SU

      Yeah. So we have the concept of a constitution, which is, uh, installed our, uh, you know, convention, our, our standard, and then the how strict it should be. So, uh, yeah, as you said, CodeRabbits knows the context and, uh, knows how far it goes, how deep it goes in the review. Because you do get into this, uh, infinite spiral where it's gonna be in too nitpicky and agent is trying to please the reviewer, and you're just going into a infinite loop of, uh, picking all the problems, uh, which is, uh, which is not necessary. You, you really wanna catch the ones that matters.

    7. AG

      Yeah, you don't wanna just burn a bunch of tokens on this kind of back and forth.

    8. SU

      Yeah, they can really keep going if you-

    9. AG

      [laughs]

    10. SU

      ... uh, don't have... So that's why we build a knowledge base, uh, or repository of what type of reviews, the review style, the coding style. It's quite extensive when... But this is where I think more engineering teams should invest in, that you build this. It's, it's kinda your organization's, uh, playbook for both your coding agent and the review agent.

    11. AG

      And who's the right person to own this? How do they really execute on this project? How much ongoing effort is required? Like, is it, like, one of your most senior tech leads who's super AI pilled and loves this stuff, they get a project for some amount of time where they're working on the code reviewer agent, and then they have to maintain that, like, every couple weeks for a couple hours? Or how does that process look?

    12. SU

      Yeah, exactly. So we have our, uh, head of engineering, Pasha, to, uh, personally look into, uh, AI productivity. We have a, a channel. Other engineers can chime in, but we have one senior engineer to look into this and experiment with multiple projects and, uh, you know, can be in, uh, production, in s- in staging to repetitively and iteratively validate

  5. 14:54 – 18:26

    Start every project from a template

    1. SU

      this approach.

    2. AG

      So how can people replicate this on their own? What are the elements they need to do to build this up from the ground up, where they feel like they're actually shipping production quality features?

    3. SU

      Okay, so yeah, the best place to start is, uh, you know, is, is a template, so this is, uh, easy to, for your team to replicate. S- Your, your lead engineer can work on this template. This is ours. Uh, so you can see we, we use Terraform for, uh, for every i- infrastructure process and CI/CD. So in this template you have... What's really important is actually this infra folder where you have, uh, say, sandbox environment and production environment. And, uh, you know, you have these, like, template Terraform files which, uh, you know, talk about the how do I do, uh, the requirements. Uh, you can configure your network in production. You can configure the access, uh, the whole, uh, how do you do ECS, how do you do K- uh, Kubernetes. And this will, uh, have a cohesive, uh, you know... Let your coding agent know how your, how your clouds, how your ops are set up, and they will, uh, use the same setup across different projects. And then how the DB should be set up, the docker and containers, and then your app logic lives in this app folder. And, uh, how your test is set up, all the scripts. So with this template, uh, a lot of the, uh, the scaffolding is already in, especially the deployment CI/CD part, which is usually where the agent can be very opinionated and you don't want your, uh, ton of projects have different ways. So this is usually a good starting point to make sure your project is deployable to your environment.

    4. AG

      And you had mentioned Terraform. If people don't know that, what is that and what does it do?

    5. SU

      Yeah, Terraform is a configuration language, uh, done by HashiCorp where essentially your, uh, it's infrastructure as code. If you want to land a new instance, if you want to configure a, a domain pointing to a certain service, instead of going to AWS console, you're actually making changes in the Terraform file, and the benefit of that is you can have your team review. This is like a code, right? To double check, and then once it's merged, it's going to propagate the change all the way to AWS.

    6. AG

      Mm, very cool. Okay.

    7. SU

      Yeah.

    8. AG

      So it's basically all the instructions from, what I, I'm guessing inside this .cloud folder is, like, your Claude MD, your different coding skills, your different architecture of your database like you had already mentioned in Docker. What all goes in this?

    9. SU

      Yeah, so, uh, in cloud, yeah, you can see from the, the folder name, there are best practices. Uh, that's for the entire organization. The rules, uh, is this, with the guardrails that, uh, agent must follow. Yeah, so this is, uh, you know, this is another, like, what we want to share across and let everyone to have, uh, the same coding agent style, and it's proven to be very effective, and we're kinda sharing the organizational knowledge together and put the best practice all in one place.

    10. AG

      Does everyone use, like, the same coding agents, or is this kind of built in a coding and agent agnostic way, so if somebody prefers Codex or Claude, it can work in either?

    11. SU

      Yeah. It's, uh, quite easy to migrate, uh, the Claude- Uh, skills and plugins. Uh, yeah, we, we do have, uh, engineers

  6. 18:26 – 23:52

    From idea to PRD to a 20-hour build

    1. SU

      prefer different coding agent, and we also have a tool that can, uh, transform, uh, the skills across.

    2. AG

      And you use Cloud Code though mainly. So what goes from here? So you got a GitHub project. Like, maybe actually let's start at the feature level. Like, how are you-

    3. SU

      Mm-hmm

    4. AG

      ... initially prompting the feature, creating the prototype? How does all of that work?

    5. SU

      Yeah. So prompting a new feature, uh, depending on how big it is, but I think today... You also have like a agreement with your agent what kind of a language you're gonna use. Like, I prefer using very simple language, and then it talks back to me in a simple language. Uh, Andrew Kapasi talk about, uh, the ICE standard which used in the aerospace, uh, that, uh, there's [chuckles] no abbreviation. You only have a certain vocabulary that you can use. I really like that. But-

    6. AG

      Mm

    7. SU

      ... uh, my works- my workspace is actually this. I have a text editor. It also has a voice mode, you can see from here, that I can, uh, talk to it even while, while I'm driving, and it will talk back. So there's th- this, uh, back and forth brainstorm with the, with the agent and kind of get the context of what we want to build. Uh, I can kind of type in through here. Uh, the voice will also end up here. And then I'll let the agent to design this, uh, you know, from the top level, the product design documents, uh, as a artifact. So that's where I go from there. I also wanted to do enough research on the market, you know, the competition. What's the best product out there? What's, what is missing? So there a significant amount of time is spent on just ideating and making sure that agent knows exactly what I'm trying to build, and along the process, I, I encourage the agent to challenge my idea to propose alternatives and have, uh, you know, small-scale experiment to, you know, to fail fast, to see some early flags. But the end product is a, is a bunch of, uh, documents. You can, you can see how many documents it produces. Uh, let me show one of those. Now, this is kind of a, a scratch pad, which is a repository just for, uh, remembering all the designs. So you can see there is the API design, the, uh, cloud architecture, and, uh, I'm just going through. You can see there are a whole bunch of, uh, documents, and this is like really, it's the PRD, uh, the engineering plan, the implementation plan. Uh, I won't click through it, but, uh, you know, each, each plan is sophisticated enough, and you can see the V2s. So I do go back and forth with my agent, and so I know almost exactly what it's going to do, what is the design decisions, and I always ask it to list the 10, 15 decisions needs my attention. And I also encourage it to make decisions on the low-risk ones. That will kick off, that will kick off a fairly length, lengthy coding session.

    8. AG

      Hmm. So you go from idea that you bat around with the agent, and you ask it to push holes in your thinking, to eventually a PRD, to eventually an engineering specification, and then it sounds like sometimes a prototype, and then you go to code. Is that the life cycle?

    9. SU

      Yes, that, exactly. And, uh, I, I tend to front load a lot of the designs and considerations and have a, uh, integrate entire engineering plan and implementation plans to kind of go off, uh, only 20-plus hours or multi-day of implementation and, uh, go as far as it can. And then I can go start another iteration of, uh, testing. Uh, so I do want to kind of play with the service myself, give some, uh, feedback. But idea is really I want to run multiple agents building multiple product and features in parallel so that while one agent is running for hours and tens of hours, I can focus on brainstorming with a, for another, for another different feature. The pattern of, uh, having an agent ask you a question every half an hour, what I found is you're gonna easily have this AI fatigue, and you're, you'll find yourself kinda drawn in multiple agent sessions and, uh, being the bottleneck of answering questions and approving the next action.

    10. AG

      Yeah. That happens to me all the time, is like I put box out, like two hours to do this thing, and then all of a sudden I just find that I spent the last two hours looping through my 12 different agents and giving them prompts. How do you make sure that you stay productive in those sessions and you eventually accomplish what you were there to do?

    11. SU

      Yeah. So I really, uh, want to find a way that agent can continuously work for, uh, much longer hours, but not drifting, and that's why in the... So that's why I front load with a lot of, uh, documentation process where, you know, from ideating to the engineering design, to the implementer design, and I review those pretty extensively. So it's almost like you, you focus on design with our agent, where you can have a lot of, uh, your own human time, and once you reach agreement, this agent, this particular feature is ready to go so that you're spending continuous time designing and getting feedback, and agent will spend continuous time building.

  7. 23:52 – 24:46

    Why running production is the next frontier

    1. AG

      All right. Where should we go from here? What else should people know in order to be able to build like this?

    2. SU

      Yeah. So now we have seen how, you know, kick off a building session and, uh, having the code reviewed and then push into, uh, get the PRs approved, and then eventually you're gonna, uh, use the Terraform, use whatever, uh, CI/CD that you can. Have your service built by agent running in some environments, let's say it's in a sandbox. And, uh, eventually this will run in production. So this is where I want to... I, I see the next frontier will be, where you let your agents to operate. So what we just talk about is going to be fairly today's commodity. I think everyone has their own favorite way of doing things and set up, but to have the agent operates in production is really something that I believe will differentiate how fast companies, uh, can build their products.

    3. AG

      All right, let's see it.

    4. SU

      So what we-

  8. 24:46 – 28:05

    Ads

    1. AG

      Are you looking to up your AI product management chops? I highly recommend the AI Product Management Certification by Product Faculty. It has a four-seven with one thousand two hundred and forty-nine reviews on Maven for a reason. I myself took the course back in twenty twenty-four, and it was awesome. Since then, they have upgraded it, so now you get to learn from product leaders at OpenAI and Anthropic. On top of that, Paul Hearn, author of The Product Compass, leads the build labs. So you will go from theoretical knowledge about AIPM to a very practical course. It's gonna help you identify AI leverage opportunities. It's gonna help you design trustworthy AI experiences. It's gonna help you systematically optimize outputs for accuracy and relevance, build rigorous evaluation suites, architect AI agentic systems that work, and select the perfect LLM for your use case. It's normally twenty-five hundred dollars, but you get a discount when you use my link. The next cohort starts June twenty-second and goes to August ninth, so do check it out with my link in the description. They have been one of my longest sponsors for a reason. I trust this product, and I think you should consider the cohort. I want to take a second to talk to you about the fourth cohort of Land PM Job. I trained thirty students in cohort one, fifty students in cohort two, and seventy-five students in cohort three, and I am bringing back the program for cohort four. It starts in August, and it lasts three months, where you're gonna have intense sessions. A Monday morning session where I go over your resume, behavioral interviews, LinkedIn. On top of that, Bart Jaworski is gonna be teaching you the PM fundamentals from twenty twenty-six. How to write AI PRDs, how to AI prototype with Claude Code, all of the key skills you need to freshen up your knowledge for this market. And Ankit Virmani is gonna be teaching you AI product management. He is an AI product manager at Uber, and he is gonna teach you how to build AI features that actually work successfully. On top of that, Prasad Reddy is gonna be doing one-on-ones with you for mock interviews, LinkedIn review, candidate market fit review. So it is a full package. It is three courses in one for one low fee. So join at landpmjob.com. I used to think I had a retention problem. Turns out I had a messaging problem. I was sending the same onboarding emails to every new user, whether they activated on day one or never logged in again. I had no idea who was slipping or why. Customer.io changed that. Every message I send is now based on what users actually do in the product. Someone hits a key activation moment, they get nudged to the next one. Someone goes quiet, they get a different path entirely. Their AI agent makes it fast. I describe the campaign I want, and it builds the full journey for me. Triggers, timing, copy, even branching logic. And when I want to know how something is performing, I just ask the agent directly, and it tells me what to do next. They also have an MCP server, which means AI tools like Claude can see directly what's happening in your Customer.io workspace. Your segments, your customer data, your attribution, all of it. So instead of explaining your business context every time you need help, Claude already knows it. Notion used Customer.io to personalize their onboarding and hit nearly fifty percent open rate, improved conversion by six to seven percent with localized campaigns, and pushed open rates up another twenty percent through AB testing. The idea is simple. Customer.io helps you deliver more impact from every message you send. If you're a PM or founder and your onboarding is still one size fits all, try Customer.io

  9. 28:05 – 40:41

    The ops agent that fixes production

    1. AG

      at customer.io.

    2. SU

      Believe is, uh, is going to be data-backed, and agents need to have access to the production metrics, traces, and logs. Well, where we start first is to build a system. Uh, I can show you this. Uh, observability, right? So, you know, every, every company has their own stack. Um, you know, I'm not opinionated, uh, on, uh, any of those, but we decided to build our own observability stack, uh, because of the, the load we're having and the traffic, but also the control, right? So, uh, we use o- combination of open source and also, uh, self, um, built solutions. And, uh, this is a diagram you can see where, uh, applications will have a... It's really easy to plug in. It's a couple of lines. Then your, your application will have the entire observability stack for Tinyfish. So you have the logs, uh, metrics, and the traces, the three critical information you want to gather from, uh, production. And they gonna fan out into, uh, the back end, uh, and this is a very efficient, high throughput system. What's critical is, uh, usually it's the human that consume the observability output, and, uh, when, whene- whenever there's a, a alarm, whenever there's errors, or you just want to look at the metrics and performance logs and find where to optimize. We really believe AI agents are better ones to consume this huge amount of data in production continuously, meaning that you watch this twenty-four/seven, and you can know where your memory has a bottleneck and what the reason before it even, uh, send you an alarm and, uh, you know, service is down. So with... And in the logs, you can watch for business logic problems, errors, and opportunity to improve your, your system. Uh, so this is the system I really want to, uh, build well and, uh, make it work in Tinyfish.

    3. AG

      Can you say more about the difference between logs, metrics, traces, what exactly those are, for people who don't know?

    4. SU

      For logs, uh, logs are... You want to keep a more of a verbal, verbose version of what happened in your service And, uh, developers can, uh, inject, uh, certain... Well, this is logs, right? It's, it's... You can, you can print out what exactly happens for a, say, a user, uh, life cycle. When users start to use this feature, uh, the service say we connect, uh, database connect. We start a database connection, and this is the, uh, connection I got, and the user send a particular request. You can log the request, and then, uh, what your system does across this whole time, uh, per step. You can be as granular as you want. You have the full control. And, uh, the log is a huge amount of data, you can imagine. This is the replay, if you will. If anything goes wrong, you want to see the logs. So metrics is really where you want to see the perform... It's all numbers. Uh, the number of requests you can serve, the CPU load, how memory's doing, uh, all the latencies. And, uh, traces is, is, is the how your system are connect together. How one user's behavior will go through your multiple services. When you have a ton of microservices, how do you trace the entire activity that, uh, across them and, uh, traces usually done automatically, uh, you know, through injection. So all three are important, and you should look at them all together.

    5. AG

      Got it. Okay. And basically, what you're showing here is a system that is... Is the point here that it's figuring out the errors or the mistakes or what is this accomplishing for us?

    6. SU

      Anything that, uh, is, is a, is a surprise or a potential f- uh, you know, improvement potential. So if there's anything that is wrong, you usually have a alarm, uh, system that, uh, you know, certain run, certain execution does not complete successfully, and they usually will... can go back to the logs to see where things fail, and that can, can point you at the potentially the part of the code that you need to fix. And, uh, metrics that will tell you, uh, the performance problems, uh, you know, how well your systems scale and how reliable they are. The traces really tells you that if, uh, all the systems are connected together when you are, uh, identifying bottlenecks in certain microservices, um, you know, if the overall latency is high, but where, right? Is it the login? Is it the database? So looking to this, it's a massive amount of data. We're talking about terabytes of data, petabytes of data that is like you can mine opportunities to improve your software and service.

    7. AG

      Where do we go from here?

    8. SU

      What we go from here is really to have the agent to look into this metrics and logs and traces, uh, continuously. So... And these are usually behind firewalls, behind authentication, so having your agent authenticated to have access, uh, you can have, uh, if you're, uh, it's all behind a firewall, then you can have the, uh, VPN installed on your dev server so they'd have the access. And then build a skill for them to know how your observability system works. What, uh, where things are, the formats and, uh, all the m- uh, the mechanisms for them to access. And then they will start to detect problems. So this is an example of, uh, this is a... I'm looking at a linear project, and, uh, it's, it's brand new. It's actually running today. Uh, you have this AI ops, uh, that's detecting issues from the observability system. For example, this is a real issue where, uh, it finds out when a LM call reach max tokens, it returns a server error. It's actually not a server error. The server is functioning, but this is like the client sending, uh, too many tokens. It should be a, a client error. And, uh, it only detect this because it finds out this behavior from the logs, from the traces. And then, hey, this is a problem. What agent does is, I'm calling it is a ops agent. It's going to first open the linear, kind of mimicking our engineer's, uh, behavior. You know, engineers would open a linear ticket first, and then, uh, it's going to go ahead and then try to fix it. So this is documenting what it finds and the potential fixes, and then it starts to making the fixes. So far so good, right?

    9. AG

      Okay. Makes sense.

    10. SU

      And after it makes the fix, it's going to push a pull request, right? It's gonna push the code into our GitHub and ask for review. And you can see for this particular fix, it's considered a lower risk, and Code- CodeRabbit automatically approve it. And what happens is it's, uh, the agent will know, "Okay, my change is good. Uh, the test, everything is okay. I'm going to merge it." Merge here means it's going to be automatically picked up and deployed to staging environment through our CI/CD. So you see the entire thing is running continuously, watching the production traces and logs, detecting errors, and make a fix, get a AI reviewer to review, and automatically deploy to the environment.

    11. AG

      And how does it know the North Star of what's true that it should give a cut answer?

    12. SU

      Yeah, so usually when you, uh, look into, when you look into the observability, uh, the whole mining part, it's... The... Quite a bit of those are pretty obvious. Like, you have this, uh, exception, you have the error log, you have this super long latency that is abnormal. And, uh, what happens, uh, you know, in the past decades is, is going to either be a alarm sent to your on-call engineer or is going to kinda- Backlog in your, uh, error system. You use Sentry, you use other systems, and engineers will try to kinda crunch through this backlog and, uh, but a, a ton of those are never fixed because they are, uh, very low, uh, percentage and hard to re- reproduce. Uh, the system over time will degrade because of this, and the user has to wait for a, a much longer time for things to be fixed, uh, if they're not but highest priority. But having the ops agent running in production, we're talking about you can have multiple sessions and make a fix and test it properly, deploy to production automatically when the engineers are sleeping. You can also have a gated rollout. You can roll out to 1% of the users in production and making sure there's no regression if you have multiple flags on AB test. So this is the whole new paradigm I'm looking to, and really I want to install this to Tiny Fish and prove it works.

    13. AG

      Okay. And those things you just mentioned are pretty interesting sort of safety mechanisms. How often should you be doing it in, like, the 1% rollout, or should you always be doing it in, like, a phase rollout, always either have a rollout or an AB test? How do you think about shipping these things that are essentially after the engineering spec, it was just done by agents, right? Just Claude-

    14. SU

      Yeah

    15. AG

      ... and CodeRabbit. [chuckles]

    16. SU

      So, uh, if you have the bandwidth and capacity, you should always do this, uh, partial rollout, always, even for a critical fix that, uh, you know, you take a extra hour. It, it also depends on your production volume, right? You wanna take a extra hour to see how this 1% user behaves, um, uh, on this new change, right? And, uh, that always, you know, it's, it's, uh, it's the last defense against, like, something catastrophic. But it used to be very, very costly. Imagine, uh, a company with, uh, a team with dozens of engineers, hundreds of engineers, the changes they make when they roll out to production. It's not realistic to have a partial rollout for every change, and because you need to... Someone, some human needs to look at the partial rollout stats and then make, make a judgment. But now you can actually have agent to do this. So the whole thing is actually designed for agent automation, and arguably, with this approach, it's safer and more efficient than human operation.

    17. AG

      Okay. Is there anything else people need to know to replicate this on their own?

    18. SU

      I think there are existing services doing this, right? There's more of a... You know, we, we kinda came from a vibe coding to more of a, you know, coding agent so that would r- remove the vibe part and to be more serious about it in production. And then there's the concept of, uh, agentic engineering that we start to kind of look at more that agent can do, not just coding, but you know, the whole thing we just talk about. But I believe this going to be what engineering will focus on. As I said in the beginning, that it's no longer matter of what product you build. It's really this is the muscle every engineering organization should build, like the machinery to produce, deploy, and operate your product in production. And you can still use as many third-party tools as, like, just like us using CodeRabbit, but I think the knowledge, the methodology, the framework, and then the opinions you have in the next six months will be the differentiator. And maybe in one or two years, this, this will just become so out of the box. Maybe, uh, the cloud providers will just have this. But for the time being, when everybody still race, you know, to the market to their users, this is really a critical and high ROI investment.

  10. 40:41 – 43:24

    The biggest mistake: thinking too much

    1. AG

      What are the mistakes you faced along the way building towards this, and what are kind of that scar tissue that other people can learn from?

    2. SU

      The biggest, uh, scar tissue is really thinking too much. And, uh, it's kinda a paradox. Like, you... It's kinda self-conflicting. You want it to be your, your agent to run as much as possible, but at the same time, you are thinking of all the things could go wrong and, uh, spending a lot of time telling yourself not to begin. Uh, but I think it's really kinda believe in this future because it's going to be the case, and don't wait for others to hand you this system for your company. You really need to build it yourself. Just don't spend too much time convincing yourself not to make a, make a move, and I think that's... We spend enough time debating and, uh, uh... But now we- I've- we feel that, uh, we could have started this a bit, a bit earlier.

    3. AG

      People are saying in the back of their mind right now, "Well, he's at a startup. They've only raised $50 million." So put your Meta hat back on when you were at Meta or at a much bigger company, you know. What are the things that they're saying, "Hey, we can't adopt this for," and how would you address those one by one?

    4. SU

      Actually, I found the, uh, companies with much bigger resource, uh, they actually gone really far. They've gone further than startups. Not all the big companies, right? But this is really fit for companies who... Let's say Meta, they built their own repository infrastructure, um, monorepo, things that they share externally. So they have, uh, the entire organization with hundreds of engineers just working on the infrastructure piece. And, uh, what is the best infrastructure to build today is definitely this, right? So large companies with more resources actually went pretty far, and they have the production volume to do this partial rollout to get the feedback, and they have so many products and features that c- they can experiment on.

    5. AG

      But many people are probably watching, and they're not at companies that have this at all, you know? They're- Uh, although their engineering org might be accelerated by AI, the product org is not feeling it at all. The product org is not suddenly just submitting PRDs and they're getting built by agents.

    6. SU

      I think this is... Well, with the... Every technological evolution will reset the market, right? I think, uh, this time the way to reset it is not only who builds a product that, uh, people love, uh, it's going to be who really adopt this new way of building faster. And of course, the one that pretty holding, holding on, holding back, uh, is gonna see a very tough time in six months, where everybody else is on a rocket and you're still walking.

  11. 43:24 – 51:13

    Building fast doesn't mean shipping slop

    1. AG

      Why is building fast in particular so important? Because I guess in the traditional sort of Marty Cagan sort of school of thought, it was always, well, building the right thing is more important. If you build fast, you could just ship a bunch of slop and then it could become a product experience people don't like.

    2. SU

      So first of all, building fast, uh, does not mean shipping slop, right? We're talking about production grade. So that's the, that's the tough part. That's where you need a lot of your product and engineering, uh, especially the senior ones, to think of how to ship and operate production grades, the best-in-class services and products using agents, right? That's, that's always the, the aura. And, uh, why building fastest matters today is, is really, if you think of why building right, the right thing is important in the past, is it takes maybe six months to have a PRD locked where you have every director's signature. I, I've gone through that time, right? And then you have the engineer, the ERD, right? That's another three to six months where you have architect design. Because, just because how costly building it was back then, you want to make the... You want to build the right thing, right? But today, building it's so fast. You don't want to spend a year just decide what to build. You want to spend as fast as possible to put it out. The best way to get the feedback is not y- a room of PMs, engineers sitting in a room in your own company. It's from the user, right? So things kinda reversed.

    3. AG

      Can you show us what the right level and quality of input then from a product engineer, product team needs to be so that an agent can go out and operate correctly, like maybe what a quality PRD looks like?

    4. SU

      So this is a research agent, and, uh, I can just quickly scroll through the definition, um, and you can see how, how sophisticated it is. So again, this is-

    5. AG

      It's quite in-depth. What I worry about when AI is writing these long documents is it drifts. So how did you create that right alignment between what you wanted and what that document said?

    6. SU

      Yeah. So there are definitely, uh, multiple... You, you're gonna, you're gonna layer them, right? So I have the PRD, and I also have the, uh, the engineering plan and, uh, um, you can see this is, uh, actually responding to the PRD, and then you have the implementation plan. So what happens is, uh, again, this is something that you want your agent to work for hours and hours on top of, right? And the agent today is getting better and better, uh, to plan for a large scale project, and it's going to actually have its own sub-agent planning. "Okay, I'm going to build one component, and this is what I'm gonna focus on." And, uh, you're gonna have a bunch of sub-agents to kinda create these components together and then piece them in the end. So, uh, more granular, more detailed planning ahead does help. The thing is, when... The reason when drift happens is agents start to make assumptions when things are not discussed, not documented. But when you have enough, lo- large context is no longer a problem, and it really knows to read a certain part of the documents and go back to, uh, work and come back to another part of the document.

    7. AG

      Okay. And then to complete the teaching you were kind of doing for us earlier, y- the initial feature you were doing, you were gonna push to staging. So what happens next? Like, where are basically all the steps where human is in the loop?

    8. SU

      So after you pushed it to staging, uh, again, staging is a environment almost identical with production, and usually, you know, companies invest, uh, building their staging environments, uh, to simulate the production environment, right? Uh, just that you never screw up the user's, uh, data and experience. So pushing from staging to a production is actually a lot faster and easier after you have done enough test. Because the environment is identical, you need to flip a certain flags and again, the, uh, the gradual rollout, partial rollouts, and, and that's it. So I spent more time in staging and, uh, you know, you have a doc footing if you have a UI component in your products or you have a load testing if it's, uh, it's a backend service. And, uh, again, AI is very good at that. Then when, when things look okay, you'll consider rolling out to production. And that's only beginning of the journey, you know, when then agents start to look, to watch the observability and, uh, start to make improvements again.

    9. AG

      And what about the part where a human would typically look at the results of the feature and say, "Do we really wanna keep this in the product even? Do we wanna do more here?" How much have you brought agents into that part of the process?

    10. SU

      Uh, yeah, you definitely want to, uh, as a human, you want to experience the product no matter what it is. So I can show this is, uh, this is our product. So we, uh, one of the products, this is the, uh, model router I talk about. And, uh, initially this is, uh, the color scheme is not this, right? It's, it's green. And, uh, Tinyfish, uh, color scheme is orange-ish. And so this is the miss in the initial, uh, design with the agent. And, uh, this is the part where you really want to kinda go in play and to see if the playground works when you choose different models. And then, uh, you can, you can see the... how this latency feels like as a human. Um, you know, definitely the latency is all captured in the, the traces and, uh, the metrics, but you still wanna kinda get a human touch. You wanna try different things. We also have a test automation agent that can open a browser and click around, because Tinyfish is a web agent, so we kinda use our own product to do test automation for UI. Um, yeah, but, you know, you... a human has to experience and, uh, dogfood. And then all of the findings can be, you know, collected and are sent back to the coding agent again.

    11. AG

      So what we've done so far in this episode is we've really helped those non-technical PM backgrounds get into all these technical concepts. So can you also explain, for those who don't know, the test automation and how that all works?

    12. SU

      Yeah. So test automation is really, uh, you know, for all the code that, uh, either engineers or AI agents created, you need a way to test. And, uh, you don't want to, uh, test everything manually. It's just not, uh, scalable. So usually you have, uh, anywhere between unit test to end-to-end integration test, and then load test. Unit test will cover, uh, more s- smaller features, whether you can do one plus one correctly, and then you just have a ton of these un- unit test covering the low-level logic. The end-to-end test is more of, uh, h- how the user experience will be as a user, and that will not capture all the problems, but, uh, you wanna m- you kinda have a good idea of, uh, you know, as a user, will they easily run into problems or it's gonna be okay? And load test, of course, is, uh, when you have a large volume in production, can your system hold? So AI agent is actually pretty good at creating all of these tests. You just need different set of, uh, infrastructure. And unit test is the most straightforward. It's, uh, just another set of a coding. Uh, load test is to kinda bring in, uh, simulation of traffic. And, uh, end-to-end test, you need some sort of, uh, agentic solution to be able to open

  12. 51:13 – 57:12

    How technical a PM needs to be

    1. SU

      a browser and then navigate.

    2. AG

      Wow, okay. So how technical do people need to be getting in order to use a system like this? Like, there's probably some component where you have this really deep engineering background, so you can go this technical, but how technical does one need to get to? I imagine they can partner with an engineer who is handling a lot of this, and where do they need to get to?

    3. SU

      Yeah. I, I think I really... If there's one takeaway from this episode, I really want all the product builders, uh, PMs or, um, you know, you, you're, you're interested in becoming one in this area, is don't be intimidated by the amount of, uh, technologies and engineering requirements that, you know, pushing things into production sounds very scary when you have a large user base and, uh... You know, it's really, it's, it's a reset. So, uh, but you, you do need a good engineering partner, like, like you said. If you don't know how to drive, you gotta have a friend who can drive. So it's really kinda knowing the, what's the high level risks, uh, what good looks like, what are the common practices. Uh, the unk- unknowns are usually very difficult, but if you have a engineering partner, they will kinda easily to tell, "Okay, uh, this is, uh, someone who does not have a technical background. What are the areas they may miss?" And then, "Let me fill that in." And when you have the right setup, hopefully, uh, your engineers are very supportive in the system. The ideal case is really the PM will s- spend a lot of time talking to the agent back and forth, define the requirements, and then, uh, give feedback in the lab test. The rest will be taken care of and supposed to be taken care of by your company's machinery.

    4. AG

      So we started at the beginning with this idea of when you typically ship the feature, you'd leave an engineer hostile. [laughs] You'd have... Like, be holding them hostage because they would have to maintain that and know that and have the accountability around it. Is that now shifting to the PM? Is it the system? How do you make sure that that accountability still exists somewhere?

    5. SU

      Yeah. Well, uh, it's a spectrum, right? So traditionally, you, um, when you build something, you're responsible for, for it for forever. And, uh, uh, no matter if you have a dedicated support team, when it comes to tricky ones, it always comes to who, uh, created the, the, the service. And, uh, then you have a limit, right? Then you start realizing someone spent half of their time maintaining, supporting what they built, and eventually may... They make... They, they may just be able to run two services, two features at most, which is what I mean by ho- holding someone hostage. It's, it's not, it's not supposed to be that way. It's, uh... You always want to create, improve, innovate. So agent's really going to play that role, uh, holding the, holding the light and, uh, uh, and keep improving. So I don't think it's also PM's job to do this either. PM's, uh, going to watch the market re- reaction and, uh, watching the top, top lines and see how we can en- enrich, improve the product and service, but not spending a lot of time on the low-level stuff.

    6. AG

      So just to put it in concrete terms, what is your thought on Where the rule of PM begins and ends and developer begins and ends in this future because where really... If the engineers are building the system to allow PMs to build a bunch of features, one could forecast a potential future then where engineers more focus on, like, this infrastructure, the system to build the system. PMs are actually focused on inputting the system. So the, um... To build a particular feature, essentially, the amount of engineering you should need would be less. Is that true, or have I played that out in the incorrect way? How would you react to that?

    7. SU

      Yeah, I think you're, you're, you're correct. And... But, uh, we are also in this, uh, amazing world when we don't really know what the models, uh, can do, uh, in six months. So whatever we say today could be different. But what I'm envisioning is really, it's still the best time, and it's going to be a better time for both engineers and, uh, PMs. What I mean by that is if you think of, uh, the amount of work that's, uh, uh, really not adding much value done by these two professions in the past, 90% of those will be gone. And, uh, today, you're seeing PMs, uh, or anyone that can build something, and it looks amazing to show. But there will be a inflection point where PMs will continue to go back to focus on the market, focus on the users, and building the, you know, creating the right things. And building will just become even invisible, right? Today, you still need to prompt and do a lot, and then, uh, work with cloud codes. But, uh, there will be a point, I believe, in the near future, you don't really need to see the cloud code session anymore. And PM is really going to use all the tools to know where the market goes. And the engineers will make sure the machinery behind the scene will be very competitive, you know, in the, in wherever the landscape you are in. So they are still kind of going back to where they're really good at. But today, we're kind of sort of seeing a mix, but I don't really want to mix building prototype, uh, putting s- it out for a little bit as a, a real building. And, uh, if PM feel great about, "Okay, I can use cloud code to put something out, I don't need an engineer," I think that needs to kind of change. And, uh, you know, you still need your engineers to think much bigger, and how

  13. 57:12 – 1:02:35

    Agents as users and the cost of not building

    1. SU

      can you partner together to have this machinery.

    2. AG

      One of the most interesting product philosophies you shared, and I would love to hear m- the next layer of the onion on, is this idea that, okay, the cost of building used to be as- astronomical. Now the cost of not building [laughs] is astronomical.

    3. SU

      You know, there, there are so many reasons why we spend a lot of time finding out what the right things to build. We talked about that just now. But today, because of, uh, the cost of building, it's so low, and it's gonna be lower. And you're not the only one in the forest. You're not the only one that is, uh, racing. So if you look around and people are, uh, launching much faster and at a higher quality, and they keep the service up, keep improving in production, and you don't build that muscle. So th- I think that's the real cost. It's not really that you're missing one product opportunity. It's really you're missing the, uh, the opportunity to build this muscle. And, uh, not having this capability in six months, I think, or a year, that time when you look back, you realize how much you missed and how hard it is to catch up.

    4. AG

      So talk to us about how you embracing this early, how that has helped your growth. Like, you guys have seen this basically hockey stick growth, where you're going from 100K to 200K fast, but then you're going to 200 to 300 even faster, and you, you continue to see that sort of exponential right now.

    5. SU

      Yeah.

    6. AG

      How has this way of building helped drive that?

    7. SU

      Yeah. So we see the hockey stick, uh, in, you know, we can correlate to a certain, a few events. But first of all, uh, we really should consider agents as the new type of users, and there will be trillions of them. Every human will have a whole bunch of agents, and these agents are going to be the smartest users. They will look for new products. They will have their own way of testing, not listening to the product owner's words. They will have their own way of benchmarking and then selecting. They even can check out the price, calculates everything, and then make decision. So we have that in mind. So it means you have a really picky user, that you have to be absolute the best choice among the competition. So what we did in the beginning that really help us is we adopt this improving in production, or you can call it hill climbing in production, that we continuously to improve, uh, our service, especially when we build a search. We build... We... It's a web infra company that will help agents to work with the web. There's a search fetch, and then there's web agent. And if you see the use cases coming from the other agents calling us, it's very diverse and versatile, so... And we, we ran into all kinds of edge cases, which is impossible to fix by, in the traditional way. So having this process running in production allow us to stay ahead of competition and allow the agents to, um, build a reputation of, oh, Tinyfish is the best in class for this particular job, and that's why we're seeing a lot of traffic from them.

    8. AG

      Hmm. So you built for AI agents. That sounds like one of the key unlocks here.

    9. SU

      Yeah, exactly. It's, uh... I think the services will become more headless, and, uh, a few super agents will stay to continue to interact with the humans, the users, and becoming the, you know, the gateway and of all other services. So they are actually the ones that making choices and decisions.

    10. AG

      If people wanna learn more, where can they find you online?

    11. SU

      Uh, they can go to tinyfish.ai.

    12. AG

      All right.

    13. SU

      And, uh, yeah.

    14. AG

      Suhao, this has been a wealth of information. Not too many people are willing to open up their actual system, break it down in so much detail to help the rest of the community. Thank you so much.

    15. SU

      Thank you. This is, uh, really fun. And, uh, I, I look forward to comments, and, uh, I want to exchange notes with the community.

    16. AG

      All right, guys, so drop it down below. What do you think you're doing that he might want to experiment with? We'll make sure he gets all of that feedback. You can reply to the email if you are just listening to this on Substack. Like something over 10,000 of you regularly episo- every episode, by the way. So appreciate you guys. We will see you in the next one. Have a good one. I hope you learned as much from today's episode as I did. If you can do one thing that's totally free that would help the show, it would be to check that you're following on Apple and Spotify Podcasts. Check that you've left ratings and reviews on those platforms. Check that you're subscribed on YouTube. Leave a like and a comment on this video, and then share it with your friends. We're trying to make better and better podcasts. After two years, we think we've gotten something pretty good going. So let us know what we can do to make it even better, who else we should interview, and we will put on the best shows we possibly can. Finally, don't forget my offer for the bundle. You get an entire year of my paid newsletter, plus my favorite AI tools, bolt.new, Airtable, Speechify, Descript, Magic Patterns, Linear, Dovetail, Arise, and Mobbin'. That's $27,000 worth of value for just $150. So check that out at bundle.aakashgee.com if it interests you, and I can't wait to share our next episode soon.

Episode duration: 1:02:44

In the uListen app, the transcript lights up word by word as you listen. Get uListen

Transcript of episode -xQPRlk7VRE

Get more out of YouTube videos.

Read along on your phone, search it beside the video, or ask your favorite AI app what was said.