The Claude Setup That Let a PM Beat 30 Engineering Teams

1. 0:00 – 2:47

   Intro

   1. JNJyothi Nookula0:00

      Understanding which surface to reach for which use case becomes one of the core PM skills that will help you become 10X more effective.

   2. AGAakash Gupta0:08

      Jyothi Nookula, she's been an AI PM since before it was cool. She's been an AI PM at Netflix, Meta, and Amazon. You posted this on LinkedIn and it caught my eye. You said that you'd won your internal hackathon against 30 engineering teams, and you used this concept of adversarial agents.

   3. JNJyothi Nookula0:23

      Anthropic had just released a blog post around hard messes and long-running agents. So I looked into the blog post, and they had this concept of adversarial agents. That was what got me the hackathon prize.

   4. AGAakash Gupta0:34

      Where does the product manager line end and developer line begin in 2026?

   5. JNJyothi Nookula0:39

      Get comfortable with building. Get comfortable with, say, Claude code, with all the Claude ecosystem that we learned today, and get comfortable building and putting your ideas out there.

   6. AGAakash Gupta0:49

      How do you use Claude Design? How do you build a knowledge-based MCP server for all of your PM context to make Claude 10X more productive? That is what we're gonna answer in today's episode.

   7. JNJyothi Nookula0:59

      Now there's this new role coming up called AI builder. Anthropic's adopted it. OpenAI's adopted it. Making and building is easy now. Taste is what is important for us to develop.

   8. AGAakash Gupta1:10

      Can you do the big reveal now and help us get that set up going in Claude code?

   9. JNJyothi Nookula1:14

      So here's the thing.

   10. AGAakash Gupta1:18

       Before we go any further, do me a favor and check that you are subscribed on YouTube and following on Apple and Spotify podcasts. And if you wanna get access to amazing AI tools, check out my bundle, where if you become an annual subscriber to my newsletter, you get a full year free of the paid plans of Mobbin, Arise, Relayapp, Dovetail, Linear, Magic Patterns, DeepSky, Reforge Build, Descript, and Speechify. So be sure to check that out at bundle.aakashg.com, and now into today's episode. So I've been thinking about something. We've had advanced tutorials on Claude code with analytics, on PM OS setup, but how do you actually take the entire Claude ecosystem and make the most out of it from scratch? I keep getting DMs from people who say, "This episode is too complex," or, "I'm not at this level yet. I'm still stuck on ChatGPT." If you're one of those PMs, this episode is gonna build you from zero to 80. We can't get you from zero to 100 in a single podcast, but we're gonna get you the 80% you need to know in 20% of the time. I have brought back Jyothi Nookula. You guys loved her last episode, and specifically the feedback I got was that her structured communication was amazing for beginners. So she's gonna break down for all the beginners how to make the most out of Claude today. Jyothi, welcome back to the podcast.

   11. JNJyothi Nookula2:44

       Super excited to be back. Thank you for having me.

2. 2:47 – 4:30

   How Jyothi won a hackathon against 30 engineering teams

   1. AGAakash Gupta2:47

      Jyothi, you posted this on LinkedIn and it caught my eye. You said that you won your internal hackathon against 30 engineering teams, and you used this concept of adversarial agents. Can you break down exactly how you won the hackathon?

   2. JNJyothi Nookula3:01

      Yes. So a few days before the hackathon, um, I was trying to see what I could build, and Anthropic had just released a blog post around, uh, harnesses and long-running agents. So I looked into the blog post, and they had this concept of adversarial agents, where you build an agent and then you set up configurations in another agent, uh, telling it what matters most to your company in, in a way, not like evals, but more around, um, um, capabilities that you want your, um, uh, agent to test. And so I set up-- I, I started with that idea and then I said, "Let me take this idea." I went into Claude code, and I was jamming with it for almost a day, uh, with different configurations, and there we go. We s- I, I had an adversarial agent evaluator running. Um, it was exactly how I pictured it to be. I even pointed it at our, uh, company code and integrated that into an actual production running code. Um, and that was what got me and my team the, uh, hackathon, uh, prize.

   3. AGAakash Gupta4:17

      So that's the promise for you guys. We're gonna help you get to that level. Where do we start, Jyothi? How can we break this down in a structured way so that people can get to this level at the end

3. 4:30 – 7:17

   The five-layer Claude stack overview

   1. AGAakash Gupta4:30

      of the episode?

   2. JNJyothi Nookula4:31

      Great. We'll tackle it today. So we'll start with understanding the Claude stack first, and then getting into some of, uh, the basics, um, like how do you use Cowork, and then getting into Claude code itself. So let's get started on the Claude stack. So at the bottom of the stack is your models. So Claude has Haiku, Sonnet, and Opus. They're all different intelligence profiles, and which one you need to use when, um, they have different cost profiles, different intelligence profiles, so that's a decision framework we'll get to in a second. So this is your layer one. On top of your models are what build your surfaces, which use to access these models. So your surfaces could be like claude.ai, which is on your browser. It could be a desktop app. It could be a mobile app. Um, it could be your Chrome plugin. These are all the interfaces that you use to interact with Claude. Now, these are not the same product with just different UIs. They have completely different capabilities, and understanding which surface to reach for which use case, um, becomes one of the core PM skills that will help you become 10X more effective. So this is our second layer. Now, on top of this layer is your knowledge base. Now, this is where your institutional knowledge lives, your projects, your skills, your memory, your custom instructions. Now, this is the layer that I think most PMs under-invest in. It's this layer that makes Claude go from being a generic chatbot to actually knowing your context. On that stack is your layer four, which is your integration fabric. For example, your MCPs. Now, MCP connects Claude to every external system that your organization uses, like your Slack, your Google Drive, your Jira, your Salesforce, your internal databases, or your own local files. Skills are what extends what your Claude knows how to do and what to do with that data. So this is your layer four. And now on top of that is your agents and orchestration. This is where your Claude Code, Cowork, Design, Channels all sit.

   3. AGAakash Gupta7:05

      Got it.

   4. JNJyothi Nookula7:06

      That's how I think about the Claude stack. Now-

   5. AGAakash Gupta7:09

      What do people need to know about layers one and two in order to make the most out of the top layers?

   6. JNJyothi Nookula7:14

      Yeah, so let's get into, um, the models.

4. 7:17 – 9:55

   Which model to use, Haiku vs Sonnet vs Opus

   1. JNJyothi Nookula7:17

      Now, Haiku is your speed machine. It's the fastest, cost efficient, and it's really great for tasks where you need volume over depth. So let's say you are trying to generate a large number of variants of something, or triaging, like, a pile of documents, or you're doing some quick classification or maybe even some tagging. Haiku handles this really well. Now, the output won't have the reasoning depth of, like, your Sonnet or your Opus, but for tasks where depth isn't needed much, Haiku is sufficient for your use case there. Sonnet is where 90% of my work lives. It has the best quality-to-cost ratio. So when I'm drafting PRDs or I'm synthesizing user research or I'm doing competitive analysis, um, or I'm doing stakeholder briefs, or I'm thinking about roadmap, I use Sonnet. S- Sonnet handles all of this extremely well. So Opus is for your high-stakes, high-complexity reasoning tasks. So let's say if you're doing some complex trade-off analysis, or you're synthesizing genuinely contradictory research, or you're doing some long-horizon, um, planning where you need the model, like, work through second and third-order implications, Opus is really good. It has really strong, um, reasoning capabilities. But I've also noticed from my day-to-day, uh, working with Opus, that it also tends to get into this, um, hallucinated stuck mode a little bit quickly than Sonnet, where I would use Opus, and it would get into, like, one reasoning decision point, and it, and it would keep, um, revolving in that local, uh, maxima. And I would have to, like, literally turn off the chat and move to a new chat and then start all over again to get it out of that thinking mode, for example. And that's when I sometimes move back to Sonnet, because even though it may not have as high a reasoning, it's generally a very efficient model to work with, and it's also more cost efficient than Opus.

   2. AGAakash Gupta9:36

      Got it. So bring the right model to the right task. It sounds like for 90% of the tasks for PMs, you'd recommend Sonnet.

   3. JNJyothi Nookula9:43

      Yeah. I think that's a good place to start with, and then if Sonnet doesn't work for the depth that you want, you can always, like, have, uh, open up a chat with Opus and start there.

   4. AGAakash Gupta9:53

      What do we need to know about the next layer?

5. 9:55 – 10:50

   Which surface to use, Chat vs Desktop vs Mobile vs Chrome

   1. JNJyothi Nookula9:55

      So next is your surfaces. Now, claude.ai, your, which is your web or browser, I think this needs no introduction. Um, everyone's pretty familiar with this. This is where you can use to chat with it. Um, the downside is that it doesn't have access to your local system. So if you have some files that you want to access, claude.ai may not be able to, like, directly go and change. Of course, you can have, like, an MCP server, but still, it's, it's, I don't prefer it for, um, for anything that I, that needs local access. That's when I use Desktop. So my Claude Cowork runs here. It's able to access my files. It's able to, um, access all the other systems that I have, integrate, and, and run some scheduled runs, which I'll show you, um, in a second.

6. 10:50 – 14:53

   Ads

   1. JNJyothi Nookula10:50

      [upbeat music]

   2. AGAakash Gupta10:50

      I built a podcast guest prep agent in Hyper Agent. The job is simple. Before every interview, give me the guest's recent appearances, strongest arguments, company context, sharp question angles, and stuff I should avoid asking because everyone else has already asked. For this run, I pointed it at Howie Liu, CEO and founder of Airtable. The useful part is it can actually go do the research. It's browsed, pulled sources, worked across files and integrations, and then turned the whole thing into a brief I can use before I hit record. Here's the output. Recent appearances, public arguments, company context, question angles, and what not to ask. This is the kind of prep doc I actually want, not a generic summary. It shows what they believe, where their thinking has changed, which questions are obvious, and where the thinking tension might be. Then I saved it as an agent. The output is useful, but the saved agent is the real goal. I don't have to rebuild the whole thing. I point the same agent at a new name, and it already knows the format I like, the sections I care about, and the kind of question framings I come back to. Podcast prep is one example. The bigger idea is recurring work becoming reusable agents. Hyper Agent is built by the team behind Airtable, but it's a separate product. They're offering $1,000 in credits to their first 1,000 subscribers who use my link. Claim yours at hyperagent.com/productgrowth.

   3. JNJyothi Nookula12:14

      I also use Mobile for when I have a, a run kicking off, and I can just go for a, a walk, and I can come back and write while I'm still doing my walk. I can look into my phone and see if any of the tasks need my attention. So this has been really helpful, um, that way. I also use Claude for Chrome plugins, especially it's very helpful if you want to u- do computer use. So, for example, when I'm launching an ad and I want to do some competitive research, I'll kick it off for, um, through my Claude plugin, and it'll use browser use, and it will open up a browser, it will do the analysis, it will click through things and say, "Here is what you need to know on how your ad should be against competitors," for example.

   4. AGAakash Gupta13:10

      Good for getting into, like, data that AI agents can't otherwise, like LinkedIn or other things like that.

   5. JNJyothi Nookula13:16

      And also good for user testing, where you can put up your product up and have, um, give an instruction to Claude saying, "Go find, check, check out this item." And you can see how it goes and finds things to see, um, how well your, um, product can be understood by agents and where does it fall. And, uh, it also gives you a really good user summary as well, if you say, "Behave like a real user and try it." And so it'll tell you, "Here are all the things that were confusing." Um, and so you can do... use it for user testing your products too.

   6. AGAakash Gupta13:54

      Mm-hmm. And are you using Claude Code in the desktop app, or are you using it in Terminal? Where does that fit in?

   7. JNJyothi Nookula14:00

      Oh, yeah. That's a good one. I use Claude Code in IDE because I use Claude Code to build. And so I use Cursor or, or VS Code, and today I'll show you with VS Code because it's really beginner friendly. So, um, I use Claude Code extension in VS Code.

   8. AGAakash Gupta14:18

      Is there anything else people need to know about layer two, or should we move on to layer three?

   9. JNJyothi Nookula14:21

      Let's move on to layer three. And before we move on to layer three, I'll come back to show you the knowledge base on how to create. But first, let me show you how you, you as a PM can 100X your productivity by running a few skills and scheduled runs in Cowork.

   10. AGAakash Gupta14:42

       Awesome.

   11. JNJyothi Nookula14:43

       Because that will bring us all together on, like, building your own chief of staff, and then I'll show you in Claude Code how you can, um, you can do something much more fun.

   12. AGAakash Gupta14:52

       So

7. 14:53 – 26:08

   Cowork automations, morning brief, standup, end of day

   1. AGAakash Gupta14:53

      should people be using Chat at all, or should they always be using Cowork?

   2. JNJyothi Nookula14:57

      So Chat is conversational to get you, like, I have a question, what does this versus that, or, um, tell me about, a little bit about this information. So it's, it's more like a place where you go to search, uh, instead of going to Google Search. Now I just find myself going to Claude in Chat and asking it some questions. I use Cowork for automations, um, and I'll show you a few today that I use. Like, I have a morning brief, um, I have my, uh, G- uh, Jira, uh, connected, so I, I get my standup brief. So every day it kicks off and tells me, "Here are all the Jira tickets that need your attention, and here is, um, how your project is progressing. Here are four blocked. Here are three things that have changed." So it gives me my, uh, brief even before I go to the standup and end-of-day summary. So there are a lot of things you could do in Cowork, um, i- in terms of automations to just make your, um, work life much more easier, um, so you're focusing on things that need the most attention.

   3. AGAakash Gupta16:03

      Awesome. So can you show us how these work?

   4. JNJyothi Nookula16:05

      Sure. So Cowork is there on your desktop app, so you need to have your desktop app, and you also need to be at least a Pro member, which is, like, about $20 per month. So with Cowork, I can schedule my automations. So you can see I have a few that have scheduled, like end of day, daily briefing, daily standup briefing, chief of staff, and I'll walk you through each one, um, right now. So every day at 9:00 AM, this runs for me where I can say, uh, and I'll show you a few, um, as well right now. So you can see my instructions. I'm saying, "You're my chief of staff. Generate my morning brief for today. Here are your data sources." And I connected it to Google Calendar, Gmail, Google Drive, and Jira. How did I do that? Let me show that to you in a sec. So go to Customize, go to Connectors, and click on the plus. Right now you can see I have connected to Atlassian Rovo, uh, Gmail, Calendar, and Drive. But there's plenty other connectors that you can connect to, like Canva, Figma, Notion. Wherever your data lives, you can connect to it. All that you have to do is just hit a plus, and that brings it in. And it'll... You'll have to authenticate it, um, and beyond that, that's all you need to do. So I said, "Here are my data sources. I need you to go into Google Calendar, Gmail, Google Drive, and Jira. Pull today's calendar events for each meeting, capture the title, the time, the attendees, the description, and any attached docs. For each meeting with external attendees or something that looks important, search the Google Drive for any attached doc or recent docs with the meeting title or attendee names. Read enough to know the agenda and search Go- uh, Gmail for recent threads. Pull Jira items needing my attention. Scan Gmail." You can also add Slack to it and have specific channels that you want it to review and send it to you as a morning brief. And I said, "Here is my output format. I want a morning brief, top three things that I need to focus on today, calendar today-" Um, here are the things from inbox that need my attention. Here are the things from Jira that need my attention. And here are some rules, and this is important, is I said keep it under 400 words because I don't want to be reading a coffee table edition the first thing in the morning. So keep it under 400 words so it's very easy for me to skim through and understand what I need to focus on, what needs my attention immediately. Claude can sometimes pump you up, so I said just give me facts. No, like, great news, so don't hype me up.

   5. AGAakash Gupta18:44

      [chuckles]

   6. JNJyothi Nookula18:45

      Um, never invent deadlines or action items, so this is like a guardrail I've put in there. And I've asked it to filter aggressively so that I don't have to read anything, um, or everything all the time. And if it's a light day, just write a three-line brief and stop. So this is my-

   7. AGAakash Gupta19:03

      And I've seen you've used mic- markdown formatting in order to help it with the headings as well.

   8. JNJyothi Nookula19:09

      Yes. That makes it easy for Claude to read.

   9. AGAakash Gupta19:12

      Cool.

   10. JNJyothi Nookula19:12

       And so you can see there are a few things that have run previously. So one thing to remember is these automation tasks run only when your laptop is turned on. If you close your laptop, it doesn't run until your laptop turns back on again. So when you choose the timings, re- just remember that and so have it at a time when you think your laptop will be on. But otherwise, when you turn it on the first time, it'll ask you, uh, and will run that, um, automation at that time. So, like, for example, let me r- show you something that ran. So I ran something that, that, um, that's from May 8th. So it captured a few inbox things that, um, needs my attention, and I can run something now and see how that works. There's a run that's started now, so it'll go and collect things. And you can see the whole process of how it's thinking. And if you notice, I'm using Haiku for this. I didn't go and use Opus just to, like, save, um, some tokens. It asks you for permission. It'll go and pull up things. It'll search email threads. And while that's happening, let me show you the next briefing. So that was my chief of staff morning brief. I also have an end of day, which wraps up my day, which runs at 5:00 PM every day. Now, my end of day instructions are very similar. The data sources are similar, but the steps are different. So I said, "Read the morning's brief," and that's what I had planned to do. Pull what actually happened today, like which meetings happened, which were canceled, and pull tomorrow's calendar as a preview. And so the output format is, like, tell me what's shipped, what slipped, and what's new from today, and show me tomorrow at a glance. Again, I have some rules. So this is my instruction for end of day.

   11. AGAakash Gupta21:15

       And I guess you could even enhance these if you were interested, right? You could probably connect up your analytics. You could add in more context from other systems like your CRM. The limit is just your imagination here.

   12. JNJyothi Nookula21:27

       Absolutely. You, you can connect it to as many data sources as you want, um, be it even sometimes your, um, uh, Facebook ad systems or your CRMs or even your YouTube, um, and you could get an end of day summary, um, that captures... And you c- you could also say, "Create a nice dashboard," which I'll show you, um, that I did for Jira, where I said the, the results, uh, print it up in a nice dashboard that I can view, and it does that for you.

   13. AGAakash Gupta21:56

       And so this is basically taking over a lot of what people would've hired a Relay or a Lindy last year, or a Gumloop or a even Make.com, and now you can just build it in Claude.

   14. JNJyothi Nookula22:08

       Yes. And one thing it's different from all of those other ones is you would have to, like, pay, um, box by box, think about how the interaction works, connect each of those, and if one thing fails, your entire loop fails. That was, like, how you used to do it before in, like, say, N8N or Lindy or Gumloop or other things that you would want. But here you see I'm just giving natural language instruction. I can even convert that into a skill. Um, so it's pretty, uh, robust, where it's very easy for me. I don't have to think about the architecture. I don't have to think how it's connected, which box flows into which, where is the conditional formatting. I don't have to think of any of those.

   15. AGAakash Gupta22:52

       So end of day, chief of staff, what are the other two scheduled tasks doing for you?

   16. JNJyothi Nookula22:57

       So this one is my standup briefing. This is the one that's connected to Atlassian. Um, that is my Jira, Jira board. And so I said, "Use this Atlassian connector to fetch all issues in the active sprint." And, uh, here's the brief I wanted, done since yesterday, which are the issues moved to done in the last 24 hours. In progress issues, blocked or at risk, new since yesterday, and what's the sprint held. And I said, "Keep the total under 250 words." And I'll show you an example of this. So yes, it's asking me for some approval. I approve it, and it's actually rendered it to me really nicely for me to view. And because I'd asked it to create a dashboard, it's running that.

   17. AGAakash Gupta23:44

       I think what people don't realize is how much better these systems got around December of last year. What really happened that enabled all this to work so much better now?

   18. JNJyothi Nookula23:52

       Improvements in the LLM reasoning capabilities, where previously if it-- um, previously as well it was much better than what it was two years ago, so we're constantly improving. But compared to last year, the, uh, new word now is hardness, so the memory, the reasoning capabilities, uh, the tools that it can access, all of the underlying systems have imp- improved. And so the latest, um- ... improvement is this harness engineering that is adding so much value into how your systems, um, behave now.

   19. AGAakash Gupta24:31

       And now we have your standup brief. W- how, how would you rate this? Is this a good standup brief or is this just okay?

   20. JNJyothi Nookula24:36

       I think this, this is a mocked up one, so therefore it's showing me a few things, which is still a lot better than what I would have had to like go and listen in a call. Um, but there's definitely ways I could improve this much more. Like for example, um, it's telling me like there's no progress in 24 hours. Um, there's-- I could look at which are, which are those ones that have not moved at all, and see who is the assignee on those, and set up an automation for, uh, Claude to go reach out, like ping them on Slack and ask them for an update, for example. So you could set up nested more, more automations, um, as well. So it's, it's really helpful to keep away your busy work, so you're focusing on actually going and, uh, solving your customer problems.

   21. AGAakash Gupta25:29

       So if these are the four scheduled tasks, are there any other scheduled tasks that you recommend PMs invest the time in building?

   22. JNJyothi Nookula25:36

       So what ha- I have here is some examples, but there are lots more you could do. So here's an example. So let's say if there is a ticket, a Jira ticket, or even a customer support, uh, ticket that's come from your, um, from your customers, it could automatically be... You could create a Jira ticket from it. You could point your Claude code to get activated so that it can actually go and implement that and cut a PR, and so there's a PR waiting for review.

   23. AGAakash Gupta26:07

       Very

8. 26:08 – 34:28

   Building skills that beat prompts

   1. AGAakash Gupta26:08

      cool. So if that's scheduled tasks, I think the next thing you had mentioned in this section were skills. What do we need to know about skills? What skills should we have? How do we create them?

   2. JNJyothi Nookula26:17

      Yes. So if you go to customize again, and you can see skills, this is where you can add different skills. I'll show you some examples of some skills. So here's my skill on synthesizing customer interviews. So as PMs, we have, we sit through a lot of customer interviews, or at least we get lot of customer interviews for research, for feedback, for focus group testing, for beta testing. We do lots of that. And I wanted an easy way for me to have or understand what's key, what's important, and then generate insights from it. So this is my skill that does that, which is, uh, synthesizing customer interviews. Um, uh, it has like when do you use this skill, and, um, what's the checklist? So it has step-by-step like inventory the inputs, extract observations with citations. Now that's important. I'm not asking to just extract observations. I want it to cite so that it hallucinates less. Use the speaker's own words, um, do not interpret yet, um, and separate behavioral observations from stated preferences. I also have additional, um, .md files listed in here linked so that it could leverage those if needed. Now that's the beauty of skill, is skill is not just a markdown file. You also can add functions into it. You can have it link to other skills, for example. So what used to happen before a skill was that the whole tool would be loaded into the context. And now imagine if you have like 40 tools, all of those are loaded into the context. It eats into your context memory. So by default, your, um, your LLM or your model would have very, uh, limited memory, uh, for even before you even began asking it anything. What skill does is similar to like progressive disclosure, where it a- it, it just loads up 50 words of just like the name and the description into the context. Now you can imagine the load is so much lower. When the model decides during orchestration, based on the question you have asked, it goes through the list of tools to see is there a tool that I need to use or is there a skill that I need to use? If it decides that there, this skill is valuable based on the description, then it will load the next set of instructions into memory. So that's why skills are powerful, because it doesn't eat up or clog your context, uh, window for your models, and it progressively disclosures. And the third is you can link it to more files or more, uh, skills or functions even. Like you can have a function where it needs to go run and do something. So in my case-

   3. AGAakash Gupta29:16

      Yeah, I think it was around February of this year when they made skills not just a single markdown file, but you could have multiple files, and if you aren't using multiple files and your main one isn't less than 500 lines, you're really missing out, I feel.

   4. JNJyothi Nookula29:30

      Yeah. And so for example, I have this, um, evidence_rules.md, which I'll show it to you in a second. Um, so that's in step two. So if step two is invoked, then it will go and see evidence_rules to ident- to understand selection criteria or how to handle ambiguous statements. Then step three is like cluster into candidate patterns, and look, I'm here again linking it to another one called, uh, jobs to be done framework. Then I said, "Then apply the pattern threshold," um, and then surface the contradictions, and then draft hypothesis, and then validate every claim against the source quotes. And that's when I said, "Assemble the final output," but I want it in this template, and this template is output_template, so I'd give it my template. So if it gets to step eight is when it will load- the output template.md. And I-

   5. AGAakash Gupta30:27

      And right now we're paying a lot of attention to what is actually in the skill file. How important is that for PMs versus just letting Claude kind of handle what's in the skill file?

   6. JNJyothi Nookula30:38

      So a lot of times we do use Claude to write the skill file too, but it's also shown, research has shown that, um, AI-generated skill file is less effective than human-written skill files. So that doesn't mean you don't use AI there. Um, what I-- The way I interpret this is put in your human domain knowledge in there to make it work for what you need versus just taking it and automating it from Claude and putting it in there. So, uh, I have used, uh, Claude a lot to help my skill files, write my skill files, but then I go in and I add my own tweaks, like, "What's the template that you want? How do you want it structured?" And I work with Claude to keep making that changes, and from there, um, add, uh, and tweak furthermore, um, to get to the skill file that I want.

   7. AGAakash Gupta31:36

      And how often should we be updating our skill files?

   8. JNJyothi Nookula31:39

      As often as things change for you. So the way to think about skill, skills is this is, um, kind of like a, a guidebook or a playbook for your Claude to know how to do a task for you. So let's say for example, PRDs. Now, if your company doesn't change the template of how a PRD is, maybe that's fine, but, but your domain may change or your understanding of your domain may continue to change, and you do want to like come back and review your skill files, um, maybe say once every quarter, depending on how often things change. So the parameters for you to decide is how often does things change in your domain? How frequently do you use that task for? And the third primary thing is how is the output currently? Because if you're not satisfied and you're like, "Mm, it was good, but now it doesn't seem to be as good," maybe go back to your skill file and see, do you need to update it? So it's like that drift as well that you, that gives you a cue that you need to go and update it.

   9. AGAakash Gupta32:47

      And what are the most important skill files for PMs to create?

   10. JNJyothi Nookula32:50

       Backlog triaging, give it context, and I'll show you in a second how to do that, uh, from a context point of view. But give it context, so backlog, um, writing PRDs, customer interviews, um, even your, um, support tickets. How do you take a support ticket and how do you put it into a Jira? That could be an automation, but it could also, like, it could be a skill that is scheduled to run every time there is a, a trigger. Now, in that case, your trigger won't be something that runs time-based because there's no, like, one particular time you're going to get the, uh, a support ticket, but it could be a trigger when, uh, this, uh, whenever there is a support ticket added in your, uh, ServiceNow or Zendesk or wherever your support, uh, forums are.

   11. AGAakash Gupta33:46

       So is it fair to say you're gonna have more skills than scheduled automations? Some of your scheduled automations might reference a skill.

   12. JNJyothi Nookula33:53

       That's true. Uh, and the way to think about it is most of your scheduled automations are time-based, so things that, um, are more personal productivity based that happen at some sequence. Like I know I meet my manager once every week, so I know the meeting is always on Wednesday, so I run my, um, automation on Friday evening just... and it maps out saying, "Here are the things that you need to talk to your manager from all these other meetings that you have sat through."

   13. AGAakash Gupta34:22

       Makes sense. The last layer you talked, or I think you were gonna show us how to do context in

9. 34:28 – 59:24

   Building an AI chief of staff from scratch

   1. AGAakash Gupta34:28

      this skill.

   2. JNJyothi Nookula34:28

      Yeah. So here's the thing. So until now, what you have done is you've connected it to sources. It can go read all of those sources and, uh, go and do the task for you. But it doesn't learn the people around you. It doesn't learn your connection to people. It doesn't learn... It doesn't have that knowledge graph or the knowledge base for what you're working on. And so I wanted to build a chief of staff that understands and is grounded in the knowledge base that I have. Um, and so I went to Claude Code and I said, "Let me spin this up." So I'm going to show you what I'm going to do there. So I'm on VS Code. Now, for those, um, who are looking at this IDE for the first time, Explorer is the place where you can open up your folders, and for you to find, um, uh, Claude, just go into Extensions and search for Claude Code for VS Code and, um, you'll find it. There'll be an install, just like how you see something else that I haven't installed. There'll be an Install button that you'll have to click on, and that's it. It'll install, and then it'll ask you for your login and everything when you, um, install it, so that way it, you're logged in and ready to go always. And it'll show up here, um, as an icon that you can click and, and it'll ask you whether it's a new session or existing session. I'll click on New Session, and you, you'll see how it makes it so much better now that I can just talk to it right here. Of course, I can open the terminal too, and it'll-- I can see if it, if I need to like run some commands, but right now I can just talk to it right here in natural language. So here's my chief of staff, um, template that I have written where, let's say I've just joined some company, I'm the senior director there. Um, I've, I want to build this personal agent that helps me navigate strategy, execution, people, politics. So the agent should learn from my meeting trans-transcripts, and I use Granola for my meeting transcripts. So it should learn from my meeting transcripts. It should ingest documents like strategy docs, org charts, PRDs, emails, and build a knowledge base over time about people, dynamics, topics, company context. And so I said, "This is my architecture overview of inputs. Here's my context and my agent." And I said, "Here's my documentation pipeline."

   3. AGAakash Gupta37:08

      And Claude wrote this, right?

   4. JNJyothi Nookula37:10

      Yes, Claude wrote this. Yeah.

   5. AGAakash Gupta37:12

      Cool.

   6. JNJyothi Nookula37:13

      Um, I told it in natural language, like, "I want XYZ. Here are all the things," and it kind of created this whole, um, .md file that I could use now with Claude again, uh, to build it. So let's say I joined a company as whichever role, and I say I want to build a personal AI agent that helps me navigate my work, like my strategy, execution, people, and politics. So the agent should, like, learn from my meeting transcripts. I use Granola. You could use Zoom, you could use Team, you could use whatever you use for transcripts. You just have to, like, mention that here. Ingest documents and build a knowledge base over time about people, dynamics, topics, company context. And here's the architecture overview. Now, I gave my use case to Claude, and it wrote this up for me and put this architecture overview that I could use it, then give it back to Claude again to code it up. And so for part one, there's, uh, here's my document ingestion pipeline. So I have, like, strategy docs, what to extract. Like, I want to extract goals, priorities, metrics, timelines. And as PMs, we are so cross-function. It's not just our docs. We read 50 docs in a week. So this is, like, really helpful for me to, like, just feed that in, and it'll read it up. It will store it into a knowledge base, and I'll show you that in a second. Um, and it's really cool where the other day I was, um, um, I was in a meeting, this person was showing me a few things, and after the meeting got over and, uh, we record transcripts, um, uh, in Google Meet. And so when the transcript came through, my, uh, chief of staff reviewed it and then it said, "You know what? You should make this person your ally because this person is good at X, which you're trying to, like, get into." And so I'm like, "Oh, okay. That's great." And then there was something else that, um, I needed to convey to, um, somebody, and my chief of staff said, "Hey, this is extremely sensitive. Have you thought about XYZ people that you have to inform first before you convey to this person?" And that's so thoughtful because now it, it's, it understands my org, it understands who is doing what, it understands their personalities. So it's, like, really powerful. It's like really I have this chief of staff that's telling me always what I need to do. So here are all the supported documents I wanted to ingest, and here's the document extraction prompt. So I'm saying, "You're helping me build a knowledge base about my workplace. I'll share a document, extract relevant information." Um, and so I said, "For strategy or planning docs, extract this way. For orgs and org charts, um, extract these. For PRDs, extract these more. For emails or communication, extract these, um, capabilities." So I have this for each of the ones that I need. And I said, "Format the output in this way, uh, for it to store in my knowledge base." Um, and here's the knowledge base structure. So it has its context KB. It has people, topics, meetings, documents, company, um, and my context, like what are my priorities, my OKRs, my preferences, notes, questions, um, insights like political landscape and patterns, um, that it, uh, re- identifies or extracts. It can save it here, and these are, like, patterns observed over time. And you can add more as well, like to-do, for example. It, it could be a running to-do, um, that your chief of staff could be maintaining for you. And here are the templates, um, for the different types. So I said, "Extract this," um, for it to, like, save it into the knowledge base, any document that I give, uh, extract the metadata, the summary, the key points, people involved, what's the relevance to me and my vertical, what are the action items, and some raw notes. Um, and for people profile, again, extract these metadata, how they operate, their communication style, meeting behavior, what works or doesn't work, what they care about, what are their motivations. So, um, and what's their relationship to me, and then over time, keep reviewing the relationship quality, whether they're a strong ally, friendly, neutral, cautious, friction.

   7. AGAakash Gupta41:53

      By the way, guys, if you want the exact information that Jyothi is sharing, you can get all of those in the GitHub link in description.

   8. JNJyothi Nookula42:03

      Organizational dynamics, the observation log, company strategy templates. So when you have your companies sharing you the strategy, saying, "Here's what we're going to do in 2026," here are the key things I wanted to extract. Org structure template and meeting transcript extraction. So when I give it a meeting transcript, what I need it to extract. Um, the agent system prompt, and this is my, uh, prompt for the agent, uh, on you have access to my context KB. Your job is to help me ramp up fast, give me strategic advice grounded in context, help me prepare for meetings, coach me on people or politics, help me think through decisions, connect dots across documents and meetings, and keep me focused on my priorities. Again, style is like my style, what I like. Don't sugarcoat politics. Um, when I share a document, extract key information, update relevant KB sections. And so I've given it all of this information, right? So this is all about, like, what it needs to do. So I have this. Now I'm just going to point my Claude code to it and say, "Now can you build this knowledge KB, and can you put this behind an MCP server so that I can use my Claude Desktop to access my knowledge base?" So I have an implementation. You can see my implementation. I'm saying Claude Desktop plus MCP, so build MCP servers for KB read and write. Chat with Claude Desktop, can also connect to Google Drive, Slack directly. And so I said, "Create the context KB folder structure, write my goals, initial priorities, ingest any onboarding docs, and after your next meeting, run the transcripts for extraction," and the KB compounds over time. So I'm just going to go to Claude code, and I'm going to say, "Can you implement?"

   9. AGAakash Gupta44:03

      So you're using the @ command to pull up that specific file and reference it?

   10. JNJyothi Nookula44:07

       I have, yes, so that way it's effective. It just knows which one I'm referring to. But even if you don't do it, if you tell it chief of staff agent design, it can go and search through your repository, um, and find, uh, the right one for you, so it can implement. Now here I'm... If you see what I'm doing, um, there are-- There's one thing I want to, like, show, um, you, is Shift+Tab. I can go into plan mode, which I can use it for planning. Again, if I do Shift+Tab, go into auto mode. I can go Shift+Tab, ask before edits mode. I can go into Shift+Tab again, edit automatically, where it gets into, like, actually coding and doing. So if you're planning, like for example, how I planned with it to create that .md file, I was all in the plan mode, where I was like, "Let's just plan. Don't start coding anything. Let's just talk." Um, and now once I'm ready, um, I can Shift+Tab again and go into edit automatically, and it will set it off to go do a few things.

   11. AGAakash Gupta45:18

       And why do we want this as an MCP server?

   12. JNJyothi Nookula45:22

       That's a good call. So, um, if you want your knowledge base in, say, Obsidian, you can connect it that way and, and put it behind an MCP server and capture it, and here at that point, you just have to say, um, uh, to put this in, um, Obsidian at that point. But I'm using local. I'm showing it on my local file system because it has a few interesting things. When you're working at a company, you don't want such really personal, private data living in some cloud, and you want it, for example, to live on your laptop. So the day when you walk out of the company, the laptop goes to them anyway, so you walk out with no data on your hand. Um, and so I prefer, because this is just so much of knowledge base and very private and personal, I keep it on my, um, laptop. But you can keep it on Obsidian or Notion or whichever one you want to use for your knowledge base. You just have to change the system prompt at that point.

   13. AGAakash Gupta46:28

       And what does putting the MCP server on top of the knowledge base help with? Why can't it just be like a set of markdown files and folders?

   14. JNJyothi Nookula46:36

       Yes. So what a- what it allows it to do is your... You can talk to your knowledge base from your desktop app.

   15. AGAakash Gupta46:46

       Mm-hmm.

   16. JNJyothi Nookula46:46

       Because otherwise, where is the knowledge graph sitting? It's sitting in some place, and if it's sitting on your computer, then it can read and it can write to it. So all those things that we said, extract this, extract that, it will actually go and write it on into your knowledge base automatically.

   17. AGAakash Gupta47:06

       Hmm, okay. So it makes it a little bit more portable than a Claude code web session.

   18. JNJyothi Nookula47:11

       Yes. And so you can go back and even look at all the .md files to see, like, what it extracted from which, um, meeting. But over a period of time, like right now, I have my, my knowledge base is, like, really huge, um, and so I, um, I don't even go look into the .md files. I just ask desktop, um, uh, uh, Claude, saying, "Hey, um, I'm going to meet my manager one-on-one tomorrow. What should I know?" And it will go and dig up all the context in the knowledge base and say, "Here are all the things you need to know," because it has my to-do there. It knows the style of my manager. It was really interesting. It said, "This person is a no-fuss person, and so you should just get to it versus preambling a lot around it."

   19. AGAakash Gupta47:57

       Hmm.

   20. JNJyothi Nookula47:58

       Because, uh, it's capturing across various conversations patterns too.

   21. AGAakash Gupta48:03

       Quality of the data going in is the most important thing. What is the data a PM needs to make sure is m- hitting their knowledge base?

   22. JNJyothi Nookula48:09

       Your meeting transcripts for sure, because the number of meetings that we attend, there's lot of data. There's lot more richer context there around people, their body language, when do they push back, how do they react? Um, so it's, there's lot of, like, understanding of context that happens there, so definitely your meeting transcripts, your, um, key documents that you receive, like, say, strategy docs that I write, I, I say push this into KB so that it, uh, remembers so the next time I say- I'm working on this project, it knows it has context directly. Um, and any other, um, documents that you review, you can push that, um, to your knowledge base too.

   23. AGAakash Gupta48:54

       Got it.

   24. JNJyothi Nookula48:54

       And then your Slack, your Slack threads, that's the other place which is super rich, uh, beyond meeting transcripts.

   25. AGAakash Gupta49:02

       And so do you need to, like, update your KB somehow, or do you set a scheduled task to update your KB, or how do you make sure that it's kind of not missing folders?

   26. JNJyothi Nookula49:09

       No, so every time, every time you have a meeting transcript, it writes to the KB.

   27. AGAakash Gupta49:15

       Hmm. And how do you set that up?

   28. JNJyothi Nookula49:17

       Yeah, I'll just show that once this is done. So, uh, it's again your MCP. Um, so if it is... Let's say, um, you have Granola, so every time, or you have Google Meet, and every time there is a new Meet recording that hits or a transcript that hits your inbox, um, you could set up a Cowork, um, automation to say, "Use this, um, and update KB," for example.

   29. AGAakash Gupta49:42

       Got it. So set up some sort of automation to make your KB updating. Make your KB an MCP server so that you can access it from regular Claude chats, not just Claude Code web sessions.

   30. JNJyothi Nookula49:52

       Yes.
10. 59:24 – 1:02:16

    MCPs and integrations every PM needs

    1. JNJyothi Nookula59:24

       Yes. So MCP is the way that allows you to connect to different capabilities, like your Gmail, Slack. I think we connected to a bunch, uh, in our cowork. Um, and so I didn't-- I showed you two things. I showed you remote MCP. I also showed you local MCP, like your knowledge KB is your local MCP that, uh, you're accessing.

    2. AGAakash Gupta59:49

       What integrations or MCPs do PMs need to make sure that they have?

    3. JNJyothi Nookula59:56

       So look at the tools that you use more often. So, like, Gmail, uh, for example, assuming your company uses Gmail for emails, you want to connect that. Calendar, you want to connect that. You want to connect your Slack. You want to connect your, uh, meeting transcripts wherever they are stored, like if that's Granola or if that's, um, um, Google Meet or Zoom recordings, you want to connect those. You want to connect your CRMs, your dashboards, um, your, um, uh, Jira boards, your, um... Maybe you're using Amplitude for analytics. Connect that there. Uh, maybe you're using some other tool like Radar for observability to monitor your, the performance of your application. Connect it. Um, you-- It's the, the possibilities are really endless. Um, so for example, one of the tool that I had connected at work, um, is, uh, NVIDIA BioNeMo model, uh, to, um, help, uh, show, uh, and do, uh, drug prediction based on a few, uh, component libraries. So it's, like, really, like, it's, you're only limited by what you can imagine. But that doesn't mean you go, you just go on a MCP shopping spree. So I would say start off with, like, connecting what works for you and what use cases you're trying to solve. And so if you're a beginner, try to Follow through this video and do some of the initial automations that I showed you in Cowork to just get started, get your hands, uh, wet, and then go build this chief of staff for yourself. And you could ask your chief of staff, "What else should I connect to?" And it will tell you, "Here are the list of servers that you need to connect to, because I'm seeing this being mentioned in meetings, and you don't have the access to that."

    4. AGAakash Gupta1:01:44

       Love it. So you can actually progressively build on your connections with your chief of staff, start with that meeting transcript.

    5. JNJyothi Nookula1:01:51

       Yeah.

    6. AGAakash Gupta1:01:51

       Let's move into layer five, shall we?

    7. JNJyothi Nookula1:01:53

       Yeah.

    8. AGAakash Gupta1:01:54

       Lovely. Perfect.

    9. JNJyothi Nookula1:01:55

       We can do this. Okay.

    10. AGAakash Gupta1:01:56

        All right. So that covers layer four. We've now done layer one, two, three, four. The next is five. What do people need to know about agents and agent harnesses?

    11. JNJyothi Nookula1:02:06

        So we have built Claude... We have used Claude code for building our capabilities. We have used Cowork, which are all sitting in your five, layer five.

11. 1:02:16 – 1:08:39

    Claude Design for decks and prototypes

    1. JNJyothi Nookula1:02:16

       Now I want to show you design, Claude Design. So the thing with Claude Design is you have to do claude.ai/design. Um, so let me show you that, claude.ai/design. It is not integrated directly in your claude.ai yet. You have to go through claude dot... Oh.

    2. AGAakash Gupta1:02:35

       There we go.

    3. JNJyothi Nookula1:02:35

       Okay. And so it, you can see it's in research preview. Now, you can, as PMs, we do a lot of, um, design work. We prototype, we create slide decks, um, we create mock applications, and Claude Design really works with a lot of those things. So, for example, you can... I'll show you a few things here. So prototype, I can give it a name. You can see there is wireframe and high fidelity, so you can choose which type you want. On slide deck, uh, you can give the project name, and you can even attach, um, your, um, speaker notes, um, and it'll create a deck. Again, you can use an animation-based, um, template to create, uh, something. And you have an other. Now, on your right, you'll see you have recent, any designs that you have worked. There are examples, um, that you can use to get inspiration and you can use as templates. And there's something called design systems. Now, design system is something interesting. Now, if you, if you have a brand, um, color, like for example, companies that have a design guide, um, you'd want your slides or your, uh, wireframes or your mock-ups to look similar to what your console is or what your company's colors are. And then you can use this design guide here. You can just click on create. You can, um, link, you can either give it a link on GitHub, or you can upload a Figma file, or you can add all your assets here and create a design system. I'll show you an example of a LinkedIn post I did. I just gave it my post, and I said, "Can you create visuals for it?" Um, and, and so it created this carousel that I wanted in the colors of my product, NextGen Product Manager. Um, so, um, it created this eight-card carousel based on the, uh, text I gave it. So I gave it my post, my LinkedIn post. I said, "Here is what my LinkedIn post is about. Can you create this?" And it created this for me. Now, here are some cool things I want to show you. Now it's built this. Now let's say I want to mock it up. I want to tell Claude to change something. So maybe say I wanted to tell, uh, make layers and use orange highlight color, and Claude can go and change just this one piece.

    4. AGAakash Gupta1:05:24

       So it's got that visual editor built in now.

    5. JNJyothi Nookula1:05:27

       Yeah. And you can also draw things. That's pretty interesting, um, for, uh, uh, for editing. So I can edit, I can, I can give it instructions right here and say, "Edit this."

    6. AGAakash Gupta1:05:41

       Mm.

    7. JNJyothi Nookula1:05:41

       I can leave comments.

    8. AGAakash Gupta1:05:44

       You change the color.

    9. JNJyothi Nookula1:05:44

       Yeah, I can change, I can, I can do comments like-

    10. AGAakash Gupta1:05:48

        Oh, like we used to do in Figma, but now-

    11. JNJyothi Nookula1:05:50

        Yeah

    12. AGAakash Gupta1:05:50

        ... the AI will execute the edit.

    13. JNJyothi Nookula1:05:52

        Yeah. So I can give it comments right here and send it to Claude. Um, and I can, um, even drag, uh, and I can, like, just draw and say, "Make it counterclockwise."

    14. AGAakash Gupta1:06:08

        Wow. So this is a carousel, but should PMs basically be creating all their presentations in Claude Design now?

    15. JNJyothi Nookula1:06:14

        I used to be a big Gamma fan, and now I just use Claude Design for everything. It, well, it consumes more tokens. The token budget is different, um, but it's been very rewarding where I don't have to sit and create slide decks anymore, and does it in my brand guide. Um, and so it just doesn't even look any different. So, um, it was, uh, funny, I had a, a meeting with my CEO, um, and one hour before, I, I have my content, I pushed it to Claude, I made it create a slide deck. It's, like, looks so, um, professional. Doesn't look like it was just done, like, a few, uh, minutes before. It looks like I spent several hours to sit and create it.

    16. AGAakash Gupta1:07:00

        Wow. So it created a CEO-level presentation for you in an hour that looked like it took hours. Very cool. Should PMs be making prototypes in Claude Design?

    17. JNJyothi Nookula1:07:12

        So here's the thing. So you, there are different types when you need different, uh, levels of, uh, prototypes. So for something quick and dirty where you're like, "Is this how I want this to be?" Um- Use, um, use Claude Design if you need, but I am more a Claude Code user because I'm like, I'll just spin up and go create that app really quickly, and I had a... It's something like an app, so people could, like, click on it and see how it works, and you get really good feedback that way. But you also could use this to create your slide decks to make presentations to your company about what's the feedback from that user interview. You could use this to create quick, uh, s- um, design patterns, um, that you want to, like, share because your app may be, like, easy for user testing, but maybe you, you want this to, like, um, uh, create some marketing content to share with your, uh, marketing team, or, uh, you want to create training, uh, uh, playbooks for your sales and accounts team, um, to tell how to, how to go use this product. And these are, like, really quick designs you can generate, uh, without having... Which looks polished and professional for them to, like, just go put it into the, um, into wherever your, uh, uh, knowledge base is.

    18. AGAakash Gupta1:08:37

        Awesome. You mentioned

12. 1:08:39 – 1:13:01

    The hackathon reveal, building the adversarial agent

    1. AGAakash Gupta1:08:39

       cl- you use Claude Code for prototyping, and that's where I wanted to take it next. So we started this episode with adversarial agents and your Claude Code setup to win the hackathon. Can you do the big reveal now and help us get that setup going in Claude Code?

    2. JNJyothi Nookula1:08:53

       So here's the thing. I have to create that whole thing here.

    3. AGAakash Gupta1:08:56

       All right. Let's do it.

    4. JNJyothi Nookula1:08:58

       Or do you h- we have the time?

    5. AGAakash Gupta1:09:00

       Let's do it.

    6. JNJyothi Nookula1:09:01

       So I'll create a new session. Let me actually open up a new project. Uh, so you can see I'm opening up a new window, so my previous one doesn't interfere with this. And I'll go create a folder for this so I can open it up. So now let me open, and you see it's a clean folder. I'm just going to start a new session here, and I'm going to say, "Let's build an adversarial evaluator."

    7. AGAakash Gupta1:09:31

       And what is GAN?

    8. JNJyothi Nookula1:09:33

       So generative, um, adversarial networks-

    9. AGAakash Gupta1:09:37

       Mm-hmm

    10. JNJyothi Nookula1:09:37

        ... were very popular before LLMs, uh, came into the picture, and that was primarily how first, our generative AI industry even started, mostly applied to images, where there are two networks. There is a generator and there's a differentiator. The generator generates, and the differentiator, um, tries to predict, is this image real or fake?

    11. AGAakash Gupta1:10:02

        Yeah.

    12. JNJyothi Nookula1:10:02

        And the optimization loop is that the generator should get so good at generating images that the differentiator gets confused whether it's real or bad.

    13. AGAakash Gupta1:10:13

        Hmm. Interesting. I've never seen this built before.

    14. JNJyothi Nookula1:10:17

        The same architecture. Let's kick it off, and we'll, we'll massage it along the way, noodle it, and figure out what, how we want it to work.

    15. AGAakash Gupta1:10:29

        So what are, while this is building, what are the keys to winning a hackathon, [chuckles] outside of creating this GAN-inspired adversarial agent?

    16. JNJyothi Nookula1:10:37

        So here's the thing. It's not about, um... Writing code has become so easy now, right? So building is easy. It is thinking about the new capabilities and how you want to go solve the problems that your customers have. So it's more imperative now to put on your product hat and see where are the problems today, how, where, where are the most friction points, so that pain and problem first mindset or first, uh, design principles that we have as product managers should continue to stay here. So you can see now it's asked me for a few, uh, questions around agent interface. How will I call your agents under test? Um, so let's say for simplicity, it's just Claude system prompt. Uh, which model should power the adversary and the evaluator? Um, so let's just keep Sonnet. How should the results be presented? You can go really, like, even a web UI. You could build a Streamlit application. I'm just going to go CLI and JSON.

    17. AGAakash Gupta1:11:46

        And what are the pros and cons of those various options, web versus CLI and JSON?

    18. JNJyothi Nookula1:11:50

        So CLI and JSON, uh, JSON shows you right in the terminal. It may not be pretty and not, and may sometimes overwhelm people as well. Uh, Streamlit gives you a really nice web UI and a dashboard, um, makes it really presentable. Now, that's where you have to think through who are your users. If your users, let's say it's a developer who is going to, like, use this application, which is what I had built for, they're very comfortable staying in their terminal, reviewing things in their terminal, and so I don't have to complicate my life further by going and creating the Streamlit. But if I was building it for, like, say, my mom, she's not comfortable looking at things on a terminal. So I would want to present it in a way that's easier to look, understand, and access information. So you have to think about the kind of users and where do they see this and what's their use case to think about these options.

    19. AGAakash Gupta1:12:53

        You've been a senior product manager at Amazon, a lead product manager at Meta, director of product at Netflix. Now you're a senior director of product

13. 1:13:01 – 1:28:51

    The new AI builder role and changing PM ratios

    1. AGAakash Gupta1:13:01

       at a startup. How do you think about the future of the product role? Here, the PM is basically doing coding work. This traditionally would've been in the developer sandbox or set of tools. Where does the product manager line end and developer line begin in 2026?

    2. JNJyothi Nookula1:13:22

       Different companies are trying it in different ways. Now there's this new role coming up called AI builder, or you can see it as me- uh, member of technical staff. Anthropic's adopted it, OpenAI's adopted it. This, there's less of like engineer, product manager, designer. These roles are all combining into being a member of technical staff. And the ratios are also changing. Uh, previously, if you see one product manager, um, works with eight engineers, now it's like two product managers, one engineer. So the roles are also like collapsing quickly, where your engineering is, um, helping you guide in terms of how do we scale the systems, how do we harden the systems, whereas, uh, you as a product manager, you're like well enabled to go and tackle those PR issues yourself, to tackle the user, uh, feedback yourself along with Claude code.

    3. AGAakash Gupta1:14:32

       So if you're a PM and you've watched this video and you wanna become a builder PM, nab one of these AI builder PM roles at a startup like yours, join your team, let's say hypothetically, what's the roadmap to get there?

    4. JNJyothi Nookula1:14:47

       Get comfortable with building. Get super comfortable with, say, Claude code, um, with all the Claude ecosystem that we learned today. Underst- and get comfortable building and putting your ideas out there. I think now is the time where building speaks a lot more, and this is what I tell even my, uh, students when I teach, uh, AIPM and agentic AI cohorts, uh, at NextGen Product Manager, where I tell them the way to transition now is by building and talking about the challenges that you have learned, um, how you went about navigating those challenges, and why did you choose this approach versus this other approach, and what happened as a result? And you can see a lot of companies now start, uh, putting even, um, a cursor or Claude code prototype, uh, as part of the interview process itself.

    5. AGAakash Gupta1:15:43

       You just recently went through a very senior level AIPM job search. What was your experience on the job search? What are the, like if you were to try to describe as a pie chart the interviews you faced and the various categories, what were they?

    6. JNJyothi Nookula1:15:58

       So broadly, they're still around product sense. Like you saw here, it's even more imperative now in this world of AI to have product sense to understand how do we want to tackle a problem? How do we want to scope a problem? Which user problem to go attempt? How do you want to approach it? So the product principles stay true even now, um, and really strong product managers, uh, in AI are actually really strong in their fundamentals as a PM. So you have product sense, product, uh, analytics, um, behavioral interview, but you also have an AI round as well now, where, um, you're asked to, uh, code your idea. So like in product sense, whatever idea I would've come up with, uh, they're like, "Could you pull up Cursor or your favorite, uh, IDE and let's start coding." And through the coding, they are able to, uh, see how I think through, like why did I choose this option versus this other option? How am I navigating? Am I just taking the first, uh, thing that, um, the AI tells me as like, "This is great," and wrapping it up? Or am I looking through things to say, "Okay, this is good, but what about this edge case? This works well, but what about this other instance?" Um, how am I corralling and shepherding my AI to work with me to get it to where I want? These are all the things that they are looking into. In addition, I also had a technical round, um, where they test you on your AI knowledge, like do you understand basic terminologies? Because you'll be working with machine learning researchers and scientists, and you just don't want... You, you won't be able to communicate to them. So, um, y- you, you are tested on fundamentals of AI, um, as well, not from a coding or engineering system design perspective, but more around do you understand what that means as a PM and how does that impact your product, for example?

    7. AGAakash Gupta1:18:01

       Mm, got it. So how are adversarial agents looking?

    8. JNJyothi Nookula1:18:05

       So let's see. It's built, and here's a GAN-inspired architecture. Uh, let's go and see. You can see it's built a bunch of things. Um, and so you can see it's went and built my red teamer designer. It's built an agent.py, um, my evaluator, so here's where I can like give it my rubric. Okay, so it's done a few things. So let's see. Um, as soon as I build... So you can see I, I want it to kick off as soon as I build an agent. I want it to automatically go and do a red teaming and adversarial example on it. So as soon as I build an agent, I want to kick off my adversarial agent until, and the feedback from my adversarial agent is passed back to my generator agent until it passes the criteria of my adversarial agent.

    9. AGAakash Gupta1:19:22

       Mm. So you're gonna set it off on essentially its own red teaming improvement loop?

    10. JNJyothi Nookula1:19:26

        Yes. Yes.

    11. AGAakash Gupta1:19:28

        So is this the secret sauce?

    12. JNJyothi Nookula1:19:30

        Yes. Um, there's... The, the secret sauce is how the system is built, and the second aspect is what am I asking it to test for? What, what are my configuration parameters? And that's where domain knowledge becomes very important, where you've got to say, "Oh, this important," or "This-- What about these edge cases? What about these use cases?" You can work with it and say, "Here are three. Are there anything more?" But it's like making and building is easy now. Taste is what is important for us to develop. What should adversarial feedback iterate on? So I'm saying, okay, there are a couple of options. For now, I'm saying just iterate on the system prompt because that's the easiest, uh, right now. And what's counts as passing, and I'll say mean score of later than eight on all criteria. And how many iterations before giving up? I'll say five iterations. Let it do this, and then we can test it out with a simple agent and see how it works.

    13. AGAakash Gupta1:20:27

        Awesome. And one of the cool features, I guess, is we could queue messages here. So should we queue up our message for the test?

    14. JNJyothi Nookula1:20:34

        Sure. We can queue it up, but I, I want to see what it comes back with because sometimes if that's-

    15. AGAakash Gupta1:20:39

        Oh, it might have some questions for us. [chuckles]

    16. JNJyothi Nookula1:20:41

        Yeah.

    17. AGAakash Gupta1:20:41

        That's the one downside of queuing. Okay.

    18. JNJyothi Nookula1:20:44

        And the other times is it would go and implement it in a way and you're like, "Ah, no, no, no, no, no, I don't want it that way. I want it this way," and-

    19. AGAakash Gupta1:20:50

        So we were talking about those AI rounds. I think I heard like almost two different AI rounds that you encountered in the job search. One was more like, "I want you to vibe code or prototype in this round," and another was AI fundamentals. For both of these, how do you succeed and prepare in those interviews?

    20. JNJyothi Nookula1:21:07

        So it comes down to you understanding the basics. Um, vibe coding is building, right? There's no shortcut to it. Um, just build, um... And I always say this, don't build them as projects. Treat them as products. Like, find problems in your area. F-Find problems that are finicky enough for you to want to go build a solution. Go build solution and see who else wants something like this. Have them come, um, and use your product. You have real users, you have feedback coming in saying, "Oh, I don't like this, I don't like that." So that's like real user experience of iterating on your own, um, product that you have built. And that really gives you a lot of confidence when you talk about your projects, uh, to interviewers, because you're not just, like, building something in an hour and calling it a project. You've actually had to think through how the user experience should be. You have real users giving you feedback. Um, you're, you are parsing that through to figure out how you want to prioritize, uh, which one you want to tackle first, which one you don't want to, all the things that you do as a product manager in real world. So I always say this, don't build projects, try to make your projects as products. That tackles the vibe coding part. Now, preparing for your, um, know- AI knowledge, you can... Depends on how structured you want it and how you thrive. So if you're really structured and you can do it everything by yourself, there's tons of like, um, very good information on your newsletter, on, uh, YouTube videos. Um, so go read them up, um, and gain that knowledge. Or if you want something structured, like, like, saying five weeks, I want to, like, understand every, um, fundamental aspect of AI, then come take a course, uh, where you have, uh, uh, five weeks or cohort-based courses. Um, I offer one too through NextGen Product Manager. So you can come sit and, uh, it's structured, you know, um, with someone teaching you week by week, you know what's coming, and by the end of five weeks, you understand, um, the concepts without you having to, uh, get overwhelmed. So it really depends on your style, how much time you have, and how much you can dedicate.

    21. AGAakash Gupta1:23:32

        All right. Looks like the next round of GAN output is here from Claude Code.

    22. JNJyothi Nookula1:23:36

        Yes. So now it's good. Um, you can see it's run, it's added a few examples here. Okay, great. So now I could literally say, um, either I could start or I c- Let's say I don't know what I should do. I could ask, "What is my next step here? How do I test it?" Okay, so I have to set my API key, and I could run a mini, tiny smoke test first, uh, with the example, and then I can ex- I can inspect the output. All right, so-

    23. AGAakash Gupta1:24:08

        All right. The moment of truth.

    24. JNJyothi Nookula1:24:09

        Yes. So let me just set my API key for a second and stop sharing, and then I'll share. Okay, so I added my API key, and now I can, um, run a tiny smoke test. So it's given me what I could run, so I, um, I'm just going to copy this. You can actually just copy. And if you notice, I have a terminal that I use. So you can just go to terminal and click on the terminal, and it'll open up a terminal for you. So now I can run this command. So you can see it is iterating. You have... It's using Haiku, uh, Claude Sonnet. Um, it's going in the first, um, round. It's tempting the bot into breaking. So there's like a simple bot that it built so we could test. So you can see the adversary is generating three attacks, and here's the score trajectory. Um, there's a mean of, mean score is 9.13. Here's the final hardened system prompt. So it's gone and edited the sy-system prompt for, um, uh, for making it better based on, um, where it did not do well. Now, in this case, it did fairly good overall, so it, it, this is your final system prompt. But you could also, like, see examples where, um, I can say s- show me an example of where it will underperform so that I can see The iterator working and improving the system prompt. So in this case, it passed in the first iteration, but we can see if it can generate an example where we can try it to do it across multiple iterations. It's created an agent, which is a weak support bot. Let's see how it'll do it there.

    25. AGAakash Gupta1:26:08

        So it's improving itself.

    26. JNJyothi Nookula1:26:10

        How do I run it? Give me the exact code as well that I can use to run. So it can also run, run it automatically for you by default. So I can say, "Run it for me," so I don't even have to go to the terminal. It can directly execute bash commands.

    27. AGAakash Gupta1:26:27

        So is this your preferred way to use it, the ex- Claude Code extension in VS Code? Is that the best way to use Claude Code?

    28. JNJyothi Nookula1:26:33

        It's the least overwhelming way for, uh, folks, so I really like to show this. Cursor is also another good one, but if you've never used Cursor, there's lots going on that it could make you feel overwhelmed. So I prefer to show VS Code, uh, because it's like a really gentle introduction and doesn't overwhelm you much once you know how and where it is, which I have already walked our users through, so hopefully they're not overwhelmed.

    29. AGAakash Gupta1:27:00

        So we've been doing the Claude ecosystem, and you mentioned, like, learning the Claude ecosystem is one of the most important things to becoming a builder PM. Compare and contrast the Claude ecosystem, the OpenAI ecosystem, I keep hearing, like, Codex might be better than Opus now at coding, and the Gemini Google ecosystem.

    30. JNJyothi Nookula1:27:17

        So here's the thing. The flavor of the month keeps changing, um, because all these models are getting really better. Um, what I have found is rather than chasing behind the next big one, I'm, what I'm trying to improve is improving my productivity. That's what if, coming back to, like, first principles, what's my goal is to improve my productivity. And I have all the systems and connections right here for me to go leverage all the hooks, um, and harnesses. Oh, harness is a word I've used couple of times. I want to, like, break it down. So previously, you would have orchestration, um, where your agent orchestrates across tools, across different capabilities. Now, being a- being able to, um, provide the, the right capabilities, like the memory or these, uh, evaluators or the various systems that your, um, agent or your LLM, uh, orchestrator brain can interact with to enhance the experience and the output you receive is what is harness engineering. Um, that's, that's become very popular, um, now, especially as models have become better, the context windows have improved, uh, are fairly large, and so harness engineering becomes, um, more important.

14. 1:28:51 – 1:32:04

    The self-improving product loop

    1. JNJyothi Nookula1:28:52

       What's something interesting that you have seen in how, um, your folks on your podcasts use, um, Claude? How do you, how do you use Claude?

    2. AGAakash Gupta1:29:07

       Oh, wow. Big question. I mean, I use it all day, every day. So one of the most interesting things that I've seen people do, set up their entire system as a self-improving product loop. So they will have support tickets and bugs come in. They have the PM agent that is triaging and understanding those. Then they have the PM agent understanding, "Okay, this is the feature we wanna build." It even goes and does user research and creates the prototype itself. It comes up with the prototype that works. Then they have their coding agents set up by their engineering team that code the feature. Then they have their analytics team agent that creates the right telemetry. All of that goes to an engineer who reviews it. Once they PR review it, actually ships. And they have their own analytics agent that automatically is analyzing it. And so they have, like, the entire product development life cycle built into Claude in an automated improving loop, especially on, like, support related, easy front-end changes. That to me has probably been the most powerful thing I've seen recently.

    3. JNJyothi Nookula1:30:13

       Yeah, it's, it just empowers you so much, uh, than before.

    4. AGAakash Gupta1:30:18

       Yeah, it's crazy. It's not just, like, writing documents or anal- doing analysis. At this point, it's like closing the loop, but actually building.

    5. JNJyothi Nookula1:30:26

       This is where it takes time, where it goes and tries to think through and comes back. Um, so this is the piece with, uh, Claude Code that takes time. Whatever it comes with, we can end with it and be like, "Okay, here's an example of how it iterated."

    6. AGAakash Gupta1:30:39

       Perfect.

    7. JNJyothi Nookula1:30:40

       Okay. So it's come up. It's done a few iterations. So, um, you see in first iteration, it was, it scored an 8.52, but the agent caved on some format conflict attacks. So it didn't pass. It went back to the, uh, generator agent to improve the prompt. And in second iteration, it scored a nine, and in the third iteration, it scored 9.08, at which point it passed our threshold. Um, and so you can see for each iteration, it went back and improved the system prompt until it passed the threshold, and that's when the, um, agent got, uh, a pass sign. And so this is where you're not just building an agent, you're actually building another evaluator to go break this agent in different ways, uh, that's important for you to know about or for your users that you care about. And, um, and th- this loop can continue until the agent that's built is not strong enough. And that's the beauty of, um, this technique, uh, age-old technique being applied for how agents will be evaluated.

    8. AGAakash Gupta1:31:57

       What a masterclass, Jyoti. Thank you so, so much for walking us from layer one through layer five.

15. 1:32:04 – 1:33:20

    Outro

    1. AGAakash Gupta1:32:04

       We have ended on self-improving agents for you guys. As we promised, we were gonna take you from zero to 80. Now, the remaining 80 to 100, you could spend 10 hours. We just spent two hours, a little less than two hours here today on it to go learn the later, la- next 80 to 100, and that's on you. So no more watching. We have the GitHub repo down in the description below. Go check that out. Go fork the repo. Start to use some of these skills and go win your hackathon. I hope you enjoyed that episode. If you could take a moment to double-check that you have followed on Apple and Spotify podcasts, subscribed on YouTube, left a rating or review on Apple or Spotify, and commented on YouTube, all these things will help the algorithm distribute the show to more and more people. As we distribute the show to more people, we can grow the show, improve the quality of the content and the production to get you better insights to stay ahead in your career. Finally, do check out my bundle at bundle.aakashg.com to get access to nine AI products for an entire year for free. This includes Dovetail, Mobbin, Linear, Reforge Build, Descript, and many other amazing tools that will help you as an AI product manager or builder succeed. I'll see you in the next episode.

Episode duration: 1:33:20