If you only have 2 hrs, this is how to become an AI PM

1. AKAman Khan0:00

   Companies are laying off entire teams, entire orgs, and PMs are sort of grouped up into that. I almost never see it where an AI PM team is gonna be laid off to some degree.

2. AGAakash Gupta0:10

   Can anyone become an AI PM?

3. AKAman Khan0:11

   I think we're all kind of feeling it, right? Like, as product managers, the expectations on us, we kind of know our role is changing.

4. AGAakash Gupta0:19

   What is the right way to teach this material? What is the right sectioning of this material? And we've come up with five steps for you guys. So we're gonna go through AI prototyping, which is kind of the heart and soul of it all. We'll go into observability on top of our prototype, evals on our prototype, the difference between RAG, fine-tuning, and prompt engineering, and then we'll end with working with AI engineers, working with researchers.

5. AKAman Khan0:41

   So let's hop into AI prototyping.

6. AGAakash Gupta0:45

   So for AI PMs, you'd really recommend they learn Cursor over the other tools?

7. AKAman Khan0:49

   I would recommend getting familiar with it, definitely. Yeah.

8. AGAakash Gupta0:52

   When it comes to people creating AI PM content, Aman Khan is amongst the most insightful and informed, and that's because he's been an AI PM since 2019. He worked at Cruise on self-driving cars. He's worked with Spotify on their AI systems, and now he works at Arize, one of the leading observability and evals companies. So if we go back then and we compare those three terms, that fine-tuning, prompt engineering, RAG, how do those all compare?

9. AKAman Khan1:17

   I think it's helpful to have just, like, a really quick diagram here of, like, what is each thing. It kind of depends on what your goal is. So if your goal is to adjust the tone or the instructions, I think prompt engineering is really helpful for that. With RAG, you can provide context over a lot of data. Fine-tuning is, think of this as adjusting the model layer a little bit, so it's actually taking the LLM and making it more specialized.

10. AGAakash Gupta1:41

    Working with AI engineers and researchers, working on these longer development timelines, how can AI PMs master that?

11. AKAman Khan1:47

    Yeah. So I think this is where-

12. AGAakash Gupta1:52

    Really quickly, I think a crazy stat is that more than 50% of you listening are not subscribed. If you can subscribe on YouTube, follow on Apple or Spotify podcasts, my commitment to you is that we'll continue to make this content better and better. And now on to today's episode. Welcome to the podcast, Aman.

13. AKAman Khan2:13

    Thanks so much for having me, Aakash. It's great to be here. I'm, I've been waiting for this one for a long time. I'm so excited to speak to you, so...

14. AGAakash Gupta2:20

    Yeah, I think that there's no better person to really give us a crash course in all of the key AI PM skills as they stand here in June 2025. But before we even get there, I need to know, can anyone become an AI PM?

15. AKAman Khan2:36

    Yeah, I mean, I think the, the whole narrative here of, like, you know, I think we're all kind of feeling it, right? Like, as product managers, the expectations on us, we, we kind of know our role is changing. Our stakeholders are expecting more from us. Our customers are expecting more from us. And I think we're already feeling that role of AI in our day-to-day life more and more. I mean, that's the reason why that, that narrative is really sticking. It's that, you know, can any PM become an AI PM? And I really think to just define what an AI PM is, it's really some flavor of either adopting AI in your day-to-day workflow, think of this as, like, an AI-powered PM, or building AI into your product, which is, you can think of that as, like, an AI product PM. And I really don't think that, you know, being an AI PM is not an either/or. I really view it more as an X, meaning, like, you can think of yourself as a fintech X AI PM or a healthcare X AI PM. And the reason I say that is because AI is really powering your workflows as a product manager rather than taking the job you have away. You really want to be able to take that core insight and knowledge and specific industry, uh, sort of knowledge that you have and apply that towards the field using AI at, you know, sort of to power those workflows. So that's really how I view it. I think, I think every PM will become some flavor of AI PM, either using those tools or building around them if you aren't already. And I wouldn't view it as mutually exclusive with the type of product managing you, you might already be doing. So that's kinda how I view it. Think of it as, like, more of an accelerator on top of the workflows you already have.

16. AGAakash Gupta4:22

    Agreed. And I think that people often come up with the edge cases like, "Hey, I'm an internal tools PM," or, "I work in this really regulated industry." But in the last few weeks and months, I've been talking to exactly those types of PMs implementing AI. I talked to an experimentation PM who is dealing with the problem that everybody else has a slight variation on their PRD template by getting an LLM to convert that into a clear output of what the hypothesis is, what the North Star metric is, what the golden rule metrics is. So, genius use case to standardize input into his experimentation system. I've been talking to people over in the financial industries. They're working on new credit models based on AI. So it seems like whatever exception you draw up, there's gonna be a counterpoint to that exception, and just about every PM needs to learn how to build AI features.

17. AKAman Khan5:07

    I think that's, that's totally true. Like, uh, and I, I think there's definitely a feeling where there's maybe some am- some amount of, like, hesitation or, you know, un- unsure of, like, wanting to brand or label yourself as like, "Oh, I'm an AI PM." Like, kind of worried you might be jumping on some sort of, like, hype train. But I really urge folks to think about what the market for product management looks like and what the roles and skill sets will require in the future. And that's really why I think that, you know, the, the sooner that you kind of think of yourself as an AI PM building in fintech, building in healthcare, the faster you'll kind of adopt those tools, the faster you'll become a leader in, in your own space in that, uh, using, using AI as well.

18. AGAakash Gupta5:52

    So enough talking. Let's get into the five skills. You and I have been going back and forth on what is the right way to teach this material, what is the right sectioning of this material, and we've come up with five steps for you guys. So we're gonna go through AI prototyping, which is kind of the heart and soul of it all. We'll go into observability on top of our prototype, evals on our prototype, the difference between RAG-fine-tuning and prompt engineering, and then we'll end with working with AI engineers, working with researchers. All right, so now we're gonna get into these skills, starting with AI prototyping. So maybe even consider opening up your browser alongside Aman as we walk you through these key skills.

19. AKAman Khan6:31

    Okay. So let's hop into AI prototyping. Um, so what we've got here, if you haven't seen this tool before, this is Cursor. Cursor is basically a fork of VS Code, which is a really common tool used by developers for actually... You know, has been u- used for many years to kind of write and iterate on code in an IDE, which is an interactive developer environment. We're gonna hop into using Cursor as our prototyping tool, just because the amount of improvements that have been made to it in sort of the recent weeks and months have made it really my go-to tool for prototyping, even relative to some of the others right now. Um, y- you know, there's, there-- And just to maybe linger on that point for a moment, there's a lot of tools out there like Lovable, Bolt, Replit, Vercel, v0, and I think they all have their place when it comes to prototyping. For instance, you know, Vercel is really strong at front end. Lovable and Bolt is really easy to deploy and get started with. Uh, Replit is really powerful for Python-based applications and having an agent built in. But the reason I really like Cursor is just because of the amount of control and flexibility it gives me to be able to iterate on specific components. Um, I completely admit, like it, there's a little bit of a learning curve to get started with using Cursor, but I promise you, if you, you know, spend a little bit of time on being able to just be able to kind of feel comfortable with the interface, you're gonna get a lot more out of the tool just because of the, the sort of the features and components it has built into it from a usability perspective. Um, maybe just to-

20. AGAakash Gupta8:09

    So for AIPMs-

21. AKAman Khan8:10

    Yeah

22. AGAakash Gupta8:10

    ... you'd really recommend they learn Cursor over the other tools?

23. AKAman Khan8:14

    I would recommend getting familiar with it, definitely. Yeah. I, I think that the other tools are gonna keep improving, and they're really helpful for building a really quick and dirty mock, uh, you know, you know, to build like just a quick UI. But if you really wanna get a little bit deeper than that and understand, "How do I implement, let's say, an agent next?" Or, uh, "Can I have more control over the system?" You're going to need a tool like Cursor, um, definitely.

24. AGAakash Gupta8:41

    Okay, cool.

25. AKAman Khan8:42

    Yeah.

26. AGAakash Gupta8:43

    Yeah, maybe we can prototype like a agentic system since that's what's hot.

27. AKAman Khan8:47

    Yeah, absolutely. I think that's a great idea. So let me go ahead and actually start, uh, here from scratch. So when you first load up Cursor, you're gonna get, um, you know, the screen where you can either set up a repo or set up a directory. It doesn't really matter what you get started with here. I have a starting point of a, of a workspace, but the two t- this t- the two commands that you wanna kinda hit right off the bat on your laptop are Command+T, which pulls up your terminal. And don't worry, you can actually just type in natural language instructions here to get started with, uh, terminal commands as well. So that's actually running the code on your computer. And then Command+K, which is, uh, really how you spin up the, the, the agent, um, which you're, which you're gonna be using for, uh... [keyboard clacking] Oh, looks like that command's... Oh, sorry. Not Command+K. Uh, what you're gonna wanna do is actually hit Command+L to pull up the agent, and the agent is this new kind of s- somewhat new feature in Cursor that allows you to, uh, go ahead and, and actually it will write the code for you and actually run the code for you too. What I've been using recently is Claude 4 Sonnet. Claude 4 is just, I think, a massive improvement on top of previous models here when it comes to understanding commands and writing code. So really, I just go ahead and start typing in what I want this agent to do, and it's able to kind of get started from there. Um, let's take an example. So when we were talking about agent-based systems, I kind of pulled this up. This is in our, uh, repo in Arize. It's a fully open source repo. It's a workflow for actually using CrewAI, which is a very kind of popular framework these days for setting up agent-based systems. Um, you can either use CrewAI, there's a ton of others out there. Uh, it doesn't really matter, but all I'm, all I wanted to do is pull up some example context that I can use and plug in so that I kinda know what the output looks like. So this is a, a notebook that just creates a CrewAI agent, really just starts spinning up a workflow for research and deciding, you know, being able to do some market research. It doesn't matter, but it's mo- more so just for grounding the, the agent in the first place. And what I'm gonna do is actually rebuild and re-architect this system entirely on the fly just using this code example. So the instructions I'm gonna give are, "Build me a Trip Planner Agent using... Instead of CrewAI, use LangGraph," which is just another framework, really just to show, like, it doesn't matter which agent framework you use, you can be really flexible here. "The Trip Planner should have a front end I can use as an application." So what I'm doing here is, is basically defining I want this agent to have a UI that I can actually click and interact with further. So we've got kinda two components in here, uh, which is, "Build me a Trip Planner Agent. Here's the framework to use." And then if you need to, you know, y- what you used to be able to do, uh, in the sort of before was actually use @web, and, um, and @web allows the agent to go and sort of search the internet as well and take action, actually search, look at documents, and take that information and apply it in the code. So let's go ahead and-

28. AGAakash Gupta12:16

    Interesting

29. AKAman Khan12:16

    ... and hit Enter here and see the agent sort of go off on its own and, and see what it generates.

30. AGAakash Gupta12:22

    And it's not a very complex prompt, really

Episode duration: 2:04:35