If This 81 Minute Video Doesn't Make You an AI PM, I'll Delete My Channel

1. 0:00 – 1:29

   Why AI PM is the new baseline (and why most roles won’t be labeled ‘AI PM’)

   1. AGAakash Gupta0:00

      There are two types of PMs in the world right now, one who is using AI to get better bonuses and hike their salaries, and the other who is stuck in the old way of doing things. This is the only video you need to watch to become the former. What are the steps you need to take to become an AI PM in twenty-twenty five?

   2. ASAnkit Shukla0:19

      What you can see on the deck and what we are going to present in the session today, like a power-packed one hour thirty minute session. That is all you need. I'm going to break down all the steps that has helped multiple people get into AI product management in a very structured way. I don't understand why people think it's so difficult to become an AI PM today.

   3. AGAakash Gupta0:35

      Ankit Shukla has helped hundreds of people land AI PM jobs in twenty-twenty five. In fact, I don't think there's anyone else in the world who has helped more people get placed this year. And some people ask me, do AI PM jobs even exist? Are they even a big percentage of jobs?

   4. ASAnkit Shukla0:51

      There are not a lot of companies which are promoting their jobs as AI product managers. I think every PM job is an AI PM job because even if you are not developing AI product on your own, still you have to use a lot of AI tools in order to make yourself more productive. If you want to learn AI product management, what you should do is look at-

   5. AGAakash Gupta1:08

      If the stakeholder management is the part of the job that is really creating the stickiness in the job, you need to make sure you do a good job. You don't want them to say, "Hey, I could just replace Aakash with an AI agent who is creating the PRD for me."

   6. ASAnkit Shukla1:21

      If you are building any product, you have to look at it from four angles.

   7. AGAakash Gupta1:28

      Ankit, welcome to the podcast.

2. 1:29 – 5:21

   Compensation reality check: US vs. India AI PM pay premiums

   1. ASAnkit Shukla1:29

      Thanks for having me here, Aakash. It's nice to have me here again. So Aakash, can you tell me how much AI PMs are making in the US?

   2. AGAakash Gupta1:35

      So I actually pulled some numbers on this, and in general, what we're seeing is that it depends on the percentile of PM you are. So in the US, there's a huge range. If you're a product manager in the middle of America, you might just be making seventy-five thousand dollars base, but the AI equivalent version of you is making ninety-three thousand dollars base. And if you're in an expensive high cost of living area like San Francisco or Seattle, average product manager is making a hundred and ninety thousand dollars base. The average AI product manager is making two hundred and fifty-four thousand dollars base. So what we're seeing overall is that AI PMs are getting paid way more than regular PMs. And in fact, there are some AI PMs at OpenAI, Anthropic, Meta, that are making millions of dollars. So it is one of the highest paying jobs in tech right now. What about India? What does the s- compensation scene look like for AI PMs there?

   3. ASAnkit Shukla2:33

      Yeah, I think that is the same proportion and similar kind of trends that we are seeing in India. So if you just go ahead and look at job descriptions, they might not explicitly say that it's an AI product manager job, but looking at the description and what they are doing, you'll understand that building evaluations, building AI system, or finding how teams can go ahead and build AI capabilities in their product is an integral part of their job. And what we have observed with our data by talking to people and helping the students out is that an average salary of a product manager with, let's say, a couple of years of experience in India is around twenty-five to thirty lakh rupees per annum. But for AI PM, we have seen that skyrocket from about twenty-five to thirty-five to even forty-five to sixty-five lakh rupees. And as you go ahead and get senior, you understand AI product strategy, you understand how to go ahead and understand the whole AI product development life cycle, that can increase much further. So I think the data is correlated.

   4. AGAakash Gupta3:19

      That's what I've heard as well. I was working with somebody who was handling a Google AI PM offer, and their total compensation was looking like, including stock options and bonus, like hundred and twenty-five LPA. So it seems like the highest paying jobs are really in the AI PM niche.

   5. ASAnkit Shukla3:36

      Yes, yes, yes. Correct. And, and also, I think not only, uh, in the PM, we are also seeing that even for, let's say, other things such as growth roles, which are related to product management, like understanding users, AI engineers, AI operators, like these are the things-- Like, because companies have a lot of leverage that they can get with an AI person, I think there is a lot of leverage that you can get in your career as well.

   6. AGAakash Gupta3:56

      And some people ask me, do AI PM jobs even exist? Are they even a big percentage of jobs?

   7. ASAnkit Shukla4:01

      Yeah. I'll tell you. So, um, a, a good way to look at it is that if you go ahead and just search for AI product manager jobs, I'm sure you'll not be able to find a lot of jobs. Why? Because there are not a lot of companies which are promoting their jobs as AI product managers. But if you go ahead, look at the job description, the responsibilities, you'll keep on seeing that, yes, now there are words mentioned such as RAG evaluations, iterations, making sure that you have the foundational understanding of LLMs. You are using tools in order to make yourself more productive. So I think every PM job is an AI PM job because even if you are not developing a core product, AI product on your own with your company, still you have to use a lot of AI tools in order to make yourself more productive. So engineers, if they are adding, let's say, ten X of their value with the AI tools available, I think product managers also have a chance to at least do X, say, X their values.

   8. AGAakash Gupta4:52

      One hundred percent. It seems like all PMs are eventually gonna become AI PMs, and there's PMs building AI features even for platforms, even for internal tools. So everybody's having to build AI features. Everybody's having to use AI in order to have more high leverage time in their work, and everybody is having to use new tools like AI prototyping. So let's just get into it. Let's start with the AI PDLC. Can you walk us through it?

3. 5:21 – 8:59

   Defining AI PM paths: AI-enabled PM vs. applied AI PM vs. core AI PM

   1. ASAnkit Shukla5:21

      Hmm. Yes. So this is what we are going to cover today. We'll go ahead and first talk about the AI PDLC. This is super important. Understand everything beyond this can go ahead and change, but this is the fundamental of product management. After that, we'll talk about things which are, let's say, more malleable, but they are still important to learn today. So let's go ahead and get started with the most important questions, okay? Whether there is someone called as a traditional PM in this world or not, okay? So unless you are working as a product owner or a program manager or a project manager, if you are working in those role, that's okay. Otherwise, every PM is going to either become an AI PM or you are going to get obsolete. So I generally divide AI PMs into two sections.The first part is the AI-enabled PMs. This is 100% of the PM even right now. If you are not using AI tools such as ChatGPT, Lovable, NotebookLM, in order to, like, say, productize, like make yourself more productive, then you are, you already losing out on that front. The second part, which we are going to most probably talk about today, is the applied AI PMs or AI product PMs. So here also I can divide them into two parts. First part is core AI PM, and second part is applied AI PM. Core AI PMs are people who are working on the deep technology level or the infra level or the model level. Okay, so we divide it into three parts. One is people who are providing the infra, then people who are providing the cloud, then people who are providing the models. So you can understand Google Cloud, OpenAI are some of the companies there. So if you are looking for, let's say, those kind of companies, not at an application level, but at an infra level, then this is the job for you. So people who have been building Google's Vertex AI or Pinecone database or working on the model of GPT, they are core AI PMs. Here, right now, it is necessary to have an AI, ML, or a data science background in order to excel here. But-

   2. AGAakash Gupta7:08

      Yeah

   3. ASAnkit Shukla7:08

      ... here is where most of the people make a mistake. What they think is that most of the roles of AI PM are Ps, and that is far from reality. If you look at the internet, if you look at, let's say, UPI revolution in India or anything, you'll understand that, yes, there are a small number of players who are going to create the infrastructure, but most of the value can be harvested in the applications, and that is where you need to learn. Okay, so even if you are someone who's coming from some non-tech background, I think with some kind of skilling up, you should be able to become this. So what are applied AI PMs? These are the PMs who are now leveraging this infra in order to go ahead and build these things. To give you an analogy, the National Payment Corporations of India, big body, has put in a lot of money in order to create something called as UPI, United, like, Payments Interface. They have became-- Because of this, they have became, let's say, the, the-- India has become the fastest growing fintech, uh, ecosystem in India, okay? They have done it. That is a government-based company. Now there are a lot of co-unicorns and companies such as Char, such as, uh, Paytm, such as PhonePe, who are built on the top of the same. Now, this is the AI opportunity for you in the same way. So people who are actually building products such as Notion AI, built on top of GPT. Building Grammarly, built on top of other models. Building ChatGPT, which is actually built on top of GPT. And Lovable, which are built on top of other models, and Cursor as well. This is where most of the value is, and these are the things that can definitely be learned, and you should be able to grab this piece of the AI revolution.

   4. AGAakash Gupta8:34

      Yes, and I think that people would assume, "Oh, just because I'm not seeing product manager jobs of OpenAI in my area or Anthropic in my area, those are just in San Francisco, that there are no AI PM jobs." But it's this applied AI PM category where there are jobs absolutely everywhere, all over the world, because everybody is having to build AI into their products.

4. 8:59 – 12:14

   What doesn’t change: PM fundamentals that keep the role ‘sticky’

   1. ASAnkit Shukla8:59

      Yes. Yes. That is correct. That is correct. So the first step that you need to do is, okay, and this is the absolutely fundamental of product management. So before I could talk on this, I want to talk about one more idea, which is that whenever a new big change comes in the society, all of us are afraid about what is going to change, how do I make myself relevant, and everything is going to fall, uh, out of place. My mental model is that before you ask what is going to change, you should ask what is not going to change, and can I make it my strength? Can I make it my fundamental? For example, if you look at this meme, building an AI product in the end is building a product. So the fundamentals of products are same. The first principle is you have to make sure that you are building something that is helpful for the users. You are building something that is giving the businesses outcome. You should build something so that the world is a better place than it is right now. So if you focus on these fundamentals, then everything else is just a learning. You have to keep on upskilling. Even, let's say, 10 years from now, AI is going to do something else, some new technology will come. But if you focus on the fundamentals and then you keep on layering on the, uh, new top skills on the top of the same, I think you should be able to become more relevant. So that is if you take only one slide from this.

   2. AGAakash Gupta10:11

      [chuckles]

   3. ASAnkit Shukla10:11

      For people who want to be relevant in the long period of time, this is this particular slide.

   4. AGAakash Gupta10:15

      The fundamentals still matter.

   5. ASAnkit Shukla10:17

      Right. And, uh, then, uh, the fundamentals, I'll, I'll repeat it again. That is user empathy, ability to understand the user. You should have problem-solving capabilities, and you should be a great stakeholder manager. Right now, your stakeholders may be engineers, designers. In the long term, it could be leaders only or maybe more of your customers and maybe some AI agents, but that's a different game. Right now, you have to go ahead and focus on these three fundamentals. They are almost never going to change in product management.

   6. AGAakash Gupta10:42

      And I think this speaks to why AI product management is here to stay.

   7. ASAnkit Shukla10:48

      Yeah.

   8. AGAakash Gupta10:48

      Because everybody is very afraid, hey, maybe the AI engineer is gonna start doing the product work because he can just have Devin, his AI agent, actually do the coding, and he then has more time. Or maybe the AI designer can just iterate on the PRD themselves. But those people, they really wanna spend all their time just talking to users, just managing stakeholders, just understanding the business value, working with finance and marketing. So I think it's this inherently people aspect that makes the product management role sticky in the future.

   9. ASAnkit Shukla11:24

      Yes, correct. And you have got it right that people expect not only from the, uh, angle of your own internal people, because now we are seeing teams getting shorter because time to value has reduced. Uh, teams getting smaller because the time of, uh, getting value has reduced. I think it's more about external stakeholder management. Most important are your customers and your partners. And a recent example is that Perplexity, they have a very short engineering team or a very small engineering team, but now they are focusing on making sure that they are able to partner with multiple people. They have done a partner with Airtel. They have done a partner with multiple universities to make sure that they're able to get the distribution. So most of your time as a product manager was being spent in managing all the engineers, making sure that you are able to prioritize a lot because you have the engineering cost, uh, the engineering bandwidth is very costly. Now you can spend it at places where you actually want to go ahead and spend it.

   10. AGAakash Gupta12:13

       100%.

5. 12:14 – 16:17

   AI-powered product development lifecycle: from inputs → roadmap → execution → GTM

   1. ASAnkit Shukla12:14

      Now I'll go ahead and take a moment to explain-What a product development lifecycle looks like. Okay, so this is the fundamental part. No matter what kind of product that you are creating, this most probably will remain the same. The first part is you as a product manager, whenever you are working on any kind of situations, you generally start with some kind of business problem or maybe a strategy, or maybe sometimes business can give you some kind of explicit goals that this is what we need to do. If you are working, if you are a founder yourself, maybe you are starting from the customer problem, but it's a reality, harsh reality of the business systems that you are given some kind of OKRs or some kind of metrics that you need to follow. So that is how the quarterly planning happens. Now, you also will have a lot of, let's say, because you want to understand what is happening in the market, you will do a lot of market research in terms of trends, what are happening. You will also understand what is that your competition is doing. So, for example, Notion AI might have understood-- Notion might have understood that now this new technology, GPT, has came, and people are facing a problem that, uh, uh, they cannot search documents in their natural language, so we can go ahead and add that capability. So they might have got an idea from the trend of AI. So now from market, we have few kind of ideas from our business problems, and then we have few kind of ideas. Similarly, you are going to work with your partners. For example, if you consider Stripe, then Amex, Mastercard, other banks are their partners, and these guys also keep on suggesting, let's say, new ideas to make the product much better. And then you also have your stakeholders. We have seen that, uh, your customer-facing stakeholders, such as your customer support, your marketing team, your sales team, they can bring the fantastic ideas, especially in the B2B environment, because sales team is actually the account management team as well. And the last part that we have seen is where the insights came from is the data. So it can either be secondary data, primary data. You go ahead and understand a lot of things. You understand funnels, cohort, use various kind of tools in order to make sure that you're able to get some insights from data. Data generally works. Understand that if you are a zero to one product, you can do a lot of market research, secondary research, but generally you have to play by your gut. But in case there, there is a mature product, the scene is different. Whenever you are taking the bets, you have to make sure that you are backed by data in order to make sure that leadership and everything is aligned.

   2. AGAakash Gupta14:30

      AI evals are one of the most important skills for PMs, and I know you know they matter. The question is, are you doing them right? Most teams are winging it with basic metrics and hoping for the best. Meanwhile, the teams that actually ship reliable AI, they've cracked the code on systematic evaluation. Today's episode is brought to you by the AI Evals for Engineers and PMs course by Hamel Hussain and Shreya Shankar. This live Maven course will teach you the battle-tested frameworks from Hamel and Shreya, who are the engineers behind GitHub Copilot's evaluation system and twenty-five plus production AI implementations. Four weeks, live instruction. Next cohort starts July twenty-first. Start shipping AI that actually works. Enroll at maven.com with my code ag-product-growth for over eight hundred dollars off. That's ag-product-growth. Today's episode is brought to you by Maven. The problem with most courses online, like Udemy, is there's no live component, and the instructors aren't experts in their fields, they are professors. At Maven, you get direct live access to experts and operators from the world's best tech companies. You can't get that access anywhere else, in any university, and you usually can't find them on YouTube either. I featured so many of Maven's experts in the newsletter and podcast for that reason. To help you out, I've put together a collection of courses I recommend at maven.com/x/aakash. This includes courses like AI prototyping for PMs, product sense for PMs, and getting an AI PM certification. Visit it now at maven.com/x/aakash.

6. 16:17 – 22:50

   Tooling leverage (and the ‘don’t outsource your brain’ warning)

   1. ASAnkit Shukla16:17

      So after all of these, understand these are the, uh, these are the canals that are filling up your idea or your problem or your solution, whatever you want to call it. This is like a one bucket where everything gets filled that, yes, these are the ideas, these are the inspiration that we go ahead and do. Apart from this, you are also a product manager. You'll also do your discovery work to go ahead and fill this up, right? And, uh, then the real work starts because you understand you cannot implement everything. So you will have your own frameworks of validation and definition, which is whether whatever people have given you the idea, mostly what happens is if a sales team or a marketing team is telling you something, their ideas are mostly suggestions of the features. So now it's your responsibility to make sure that if someone gives you a solution, you have to reverse engineer into a problem, and then you have to understand and validate whether this problem actually exists or not, and there are multiple ways of the same. And then you have to find out what could be, if it is like a good enough problem, where does it-- it stands in my roadmap. A common mistake that people do is they are suffering from this thing called as shiny object syndrome. Like any new problem comes, I want to go ahead and solve it. So they lose the context of the whole roadmap. So I think by doing the right kind of stacking-- So as a product manager, you should always have the roadmap in your mind so that you are able to understand, no, this problem is better, this problem is better, et cetera. So you have to have that strategic mindset. Once you, let's say, go ahead and validate it, then it is your responsibility and your team's responsibility to find a better solution because you have the subject matter expertise, you have the domain matter expertise, you understand how your technology and your team works. And then eventually you will go ahead and implement this idea if this is, let's say, checking all the boxes. So you will do the validation and the definition. And then here again, you can split all of these things into problems and solutions, right? And then you can go and again, uh, you can again run a round of validation here. Now what happens is, after you have identified the problems and the solutions, you are list-- you are ready with a list of featuresStill you have to do some kind of prioritization again, because even if it is important, it cannot go in all one go with your engineering team or your sprint team. So then you go ahead and prioritize, right? And after you prioritize, you are going to end up with a product roadmap. Okay. Now, this is for the traditional, uh, PM job. If I have to go ahead and tell how AI is going to change this, there are two ways. First is, all of these things could be rapidly-- all of these things could be rapidly improved with AI. It can make you more productive. For example, initially, I have to go to Mixpanel in order to create the funnels manually, and if I miss any events, the data is wrong. Now they have an AI feature. Similarly, if I want to understand that, let's say, some of my partners are giving me suggestions or they have some kind of tickets, initially, I have to learn a language in Jira called a Jira Query Language. And many PMs were not, like, they are not very happy with that language because how it is structured. So what they did was they would just go ahead and maybe do the cookie-cutter one, because it generally takes time, right? And, uh, but now they have launched something called a Jira, uh, Atlassian Intelligence, with which you can just in natural language, you should be able to do this. Okay. A lot of product managers also face challenges in communicating to the stakeholders. So what happens is someone has suggested you an idea and you don't know, let's say, how to-- how do I say no to this person, right? So many tools like ClickUp, Monday, and this Atlassian Intelligence, now they have a method, like it's a very small feature, but I think it's, it is very impactful. It helps you re-articulate yourself. So people who are not very good with communication, they can write something, and then they can give a prompt that please make it nicer, more formal, more friendly or something. So that now-- And you will look alike it, you will understand it is not the same person who is taking this thing, who is talking about this thing. So these are like small productivity things. Along with this, all of us understand that in order to market research, you have NotebookLM that can make you more productive. You have ChatGPT, which you are using at all the places. And then for identifying business problems and all, there are tools which are going to automate the OKR creation for you. Okay. I-I'll just add an anecdote to the same in, in, in just a moment as in how far you should go ahead in using these tools because it can also backfire, but we'll cover that later in the section. So for now, it is in the AI cycle, in the AI world, this cycle is going to become more shorter. Also, in validation, a major problem was that you have to be very good with validation so that you are able to give only the good ideas or the best ideas to your developers because development time bandwidth was very costly. Now, with the help of AI tools such as Lovable, such as Cursor, such as, uh, V0 and so many other tools, Windsurf and all, it's very easy for you to build quick prototypes. That will make sure that you are able to test the product with real people very, very quickly. At least with a beta tester, you should be able to go ahead and do it. And then your-- Like, you can use Lovable and all, and then your engineering team can use Cursor and Windsurf in order to make themselves more productive. So one caveat that I'll tell you, that you should actually, as a product manager, motivate all of your engineers to use these kind of tools by obviously giving them a security that they'll not get fired once they go ahead and, uh, do this kind of work. So I think that has to come from top-down because I've seen a lot of-- I've observed a lot of people because we take master classes for, let's say, thousands of people at once, and some people have genuinely asked a question that my company does not allow me to use AI tools. What should I do now? So that is something that should come from the top down. So that is the first part, how you can become creative, how we can become productive with AI. The second part is validation and, uh, definition, and a most important part is problem and solution. Now, this is very important part. Understand you have to find problems, but you also need to understand that it is not a reality that every problem has to be solved by AI. There are few things that AI can solve better, and there are few things that maybe your normal CRUD application can go ahead and do much better. So make sure that you, as manager, don't think of AI as a solution to everything. That is, uh, uh, I would say, uh, uh, not a good approach to have because AI has a cost, and if you try to retrofit a solution to every problem, then you are going to end, uh, yourself in some trouble. So that is the first thing, that please understand the problem with very first principles, understand it very objectically-- objectively, methodically, don't think about the solution at once, and then try to make sure that does the problem stand on its own or not. After that, you can go ahead and think about solutions. Now, in terms of solutions, if there is an AI use case, then you should be able

7. 22:50 – 27:01

   Choosing the right AI approach: predictive AI vs. generative AI

   1. ASAnkit Shukla22:50

      to contain it within few categories. Okay. So now I divide the AI into two parts. The first part is the predictive AI, which is also called as these days, called as traditional AI. Yeah, this was since the time that Amazon started creating recommendations and Netflix started creating recommendations and all, and Google Search also has this kind of AI. The second AI, which is, let's say, more cool and what people want to learn these days, is generative AI. So the kind of solutions that are being offered by predictive AI are either you can use it to rank stuff. For example, on Google, uh, uh, on Google Search, we have a lot of results. How do we make sure that every time we don't have to codify these results and then it is able to automatically rank? So it trains on a lot of data, and then it is able to generate the rank, which is the-- predict the rank, sorry. After that, you have some kind of recommendations. Recommendations means Netflix can suggest you what you should watch later. Amazon can tell you what is the next thing that you should buy along with this. Similarly, we have another kind of use case, which is finding the anomalies, which is breaking the trends. For example, uh, if there is a fraudulent credit card transaction, your bank would be able to understand that this person does not transact mostly from this particular location or this particular website. So they'll be able to tag it, and that also happens automatically with AI. And then one more thing is categorization. For example, if you want to categorize whether the c- incoming email is a spam or not a spam, that is what Google and other emails does. So these are generally the traditional use case. Understand, even if we are in the generative AI phase, that does not means that these use casesAre obsolete. If there is a problem, and these are very common problems that they solve, ranking and categorization and all, so they are going to remain relevant. My recommendation is that please understand the cost of generative AI, understand what its best use cases are, so that you can fit the right use case rather than just putting generative AI into everything. And then for, yeah.

   2. AGAakash Gupta24:43

      I think people heavily underestimate how effective just adding in predictive AI and upgrading your predictive AI can be. For instance, one of the biggest things that Google shipped in the last few years was creating-- There was this paper called Attention Is All You Need, where they released the transformers, and basically what this allowed is when normally you used to type in Google, let's say you wanted to search AI product management, you would just search AI product management. Now, because of these transformer models, you can search something like, well, what is the AI product management product development life cycle? And it will actually be able to transform that in order to figure out what are the right results. So there are all these evolutions happening within predictive search to understand natural language better. Or even just to go from what is the HBO Max, Amazon Prime Video level recommendations to Netflix level recommendations. You look at Netflix, they're worth so many hundreds of billions of dollars. There's a lot of upgrades you can do just within the predictive space that can be really fruitful.

   3. ASAnkit Shukla25:58

      Yes. Wonderful. A-and that is correct, and also that paper was made to make sure that predictive AI makes-- becomes much better. And that is the foundation of the generative AI as well. Because they were able to add context to everything. For example, when I say that the bat is hungry. In the predictive model, what used to happen was it will just go ahead and look at one word at a time. So it will not be able to understand what the context is. Okay. But now if I say the bat is hitting runs. So now you understand these are different statements. Here, the bat is an animal, and here bat is actually a cricket bat or a squash bat. Now, with, uh, the transformer model, it is able to add context based on what is added here. So for example, if it says hungry, that is only a something that an organism can go ahead and do. But hitting runs is something that a cricket bat can go ahead and do. So that is, uh, like a model that was created for improving the predictability has actually became the foundation of the generative model. Thanks for adding that. I think that's a very important context.

   4. AGAakash Gupta26:58

      Yes. So where do we go from here in generative AI?

8. 27:01 – 36:17

   Generative AI use cases + how to build your personal ‘AI use case database’

   1. ASAnkit Shukla27:01

      Yes. So now in generative AI, so generative AI means contextual content generation. Contextual content generation, which means informed content generation. Rather than just putting anything that is outputted, it is going to give you a very contextual content generated. Now, this opens up a lot of opportunities because content can be your text, and then if you go ahead and combine, like see what is the possibilities with text, text can be used to create summaries. Ca- text can be used to create poems. At the high level, at the extreme level, it can be also used to create code. Now, when it can create code, now magic starts happening. Now you can do a lot of stuff. Similarly, this can also generate videos. It can also generate, let's say, images, and then audio also. And you understand video is just a combination of an audio or images out there, okay? So with this base understanding that, yes, these are the use cases of A-- let's say generative AI. And one more exercise that I'll give you because we'll not be able to go into depth here. One exercise that I give to everyone is, what you should do is go to these top LLM companies, which is Anthropic and, uh, OpenAI and Gemini. On each of their website, you should be able to see a page called as Customer Stories. Let me show you. So if you go to this, you'll be able to understand how people are actually using generative AI in their use cases. And then you'll understand you should up, up-- after going through this, if you're really serious about product management, you should create a list of use cases.

   2. AGAakash Gupta28:30

      Today's episode is brought to you by the AI PM certification on Maven. Run by Miqdad Jaffer, who is a product leader at OpenAI, this is not your typical course. It's eight weeks of live cohort-based learning with a leader at one of the top companies in tech. OpenAI just doesn't stop shipping, and this is your chance to learn how. Run along with product faculty and Mo Ali, the course has a four point nine rating with a hundred and thirty-three reviews. Former students come from companies like OpenAI, Shopify, Stripe, Google, and Meta. The best part? Your company can probably cover the cost. So if you want to get five hundred dollars off, use my code AAKASH25, and head to maven.com/product-faculty. That's M-A-V-E-N.com/product-faculty.

   3. ASAnkit Shukla29:17

      So now you have done a great research as a product manager. That's your traditional job. You can use AI tools for the same. The next step is try to go ahead and create your own database of use cases of AI, and then you're trying to match that whether this problem can be better solved with AI or something else, or something else out there. Another website I would recommend you is, let's say, if these are too focused on the company, what you can do is you can just go to Product Hunt. On Product Hunt, you should be able to see, let's say, top products. So I, I'm sure there would be a launch archive. Most loved launches. Okay, so these are the best products which might not be so popular, but community really loves them. Okay, so now you can go ahead and see which are the top products. And then mostly because AI is very popular these days, most of them are using AI in some or the other way. So the second step is after you have done all the research and everything, one thing that AI changes is if you have a list of use cases that AI can do, that is your own database of training for your own mind. And then you can go ahead and combine and mix and match in order to understand what can AI do here. And that makes a lot of difference. So now you are ready with the solution that was, "This is my problems are."This is what the use cases are. Now I have matched. I should now do this in order to make my customers much happier with my product. So once we are, let's say, done or okay with the product roadmap, we have prioritized everything, we are ready with the product roadmap. The next step to make sure that you as a product manager understand product roadmap is a high-level document. It is a strategy document made for three months, six months, nine months. You cannot just give as is to the developers and designers in order to do the work. You have to convert this something into a product backlog or maybe a bucket list of items which are, let's say, clearly defined. In between, you also have to create something called as PRD or a product requirement document. That is important. So once you do this, after this, what you can just do is run the sprints with your team. With AI, you will find that the sprints are going to give you more outputs, more, let's say, more number of outputs now, shippable products. And then eventually what you do is you just go ahead, work with the sales or marketing or, uh, I would say a more popular word these days, which is more cool, is the growth team. So that is a growth team, and then you create a go-to-market strategy. And eventually what you do is you'll just connect this th- this thing, you give it to the users, and users will give you the feedback that will generate data or qualitative feedback, and then this funnel is complete. And now you can understand that this is a traditional PM cycle, which is powered and enabled by AI. And here at all the on-- in, in all the parts of this, you can use some kind of AI tools in order to make yourself more productive, give better results. And also the main AI part happens here, like where you go ahead and make sure that you have a set of use cases for AI available. Okay, in the next part, we are also going to talk about what are nuances that you need to take care in order to, let's say, use these LLM models and generate the-- create the generative AI products. But one thing I want to mention, which is super important. Now, let's say take one example, which is PRD. Now, AI can generate a PRD, right? But that does not mean that AI should generate a PRD. The difference is, for example, you should consider it like a, a, a, a AI more of an intern. Okay. So understand, when we talk about work, we do not talk about the output of the work. We talk about the outcome of the work. Which means that, let's say, I have some experience in education tech, some in fintech, and maybe most of the companies that I've worked with are B2C companies. Today, I go ahead and create a product strategy. And you go ahead, you have worked with fantastic companies, mostly in the, let's say, the B2B environments, as well you have worked for a fantastic gaming company, and you create a strategy for a B2B company. You understand that I can also generate the document. You can also generate the document. We both have availability of all the Google research or the Perplexity researches, but I am sure that you are going to do a much better job because you have that certain context, you have that certain understanding of the user or the market. So understand, don't just think about PRD or things that can be created with AI. Look at a scale. On a scale of zero to one, how helpful of a document can this AI create? Maybe AI will give you sometimes zero point two, sometimes zero point eight. But you have to set your own context and give your own knowledge, do your own research in order to make it a one. Is it a problem if you go ahead and create a zero point five PRD? No, if you don't care about it. But if you love your craft, you have to make sure that you are getting those extra brownie points. And what happens is, if you keep on doing this zero point five, zero point eight, zero point two work that AI has generated, eventually you will find yourself that you're not able to think very critically, and then you will lose your mojo as a product manager. So my recommendation is don't ask me if AI can generate wireframes. AI can just generate a complete product for you from one single prompt on, uh, Lovable. It can create the marketing strategy for you. It can run Facebook ads for you. You, you should just have, let's say, a pocket full of money. But it doesn't mean it should go ahead and do the same. Another example that I can give is Google, like OpenAI, has used all of its efforts, billions of dollars, in order to go ahead and create what? GPT. Now there is a com- a company came in China using the older Nvidia models, and then they were able to create DeepMind, DeepSeek. Understand all of these are AI companies, but what changed was the mindset. People understood that, yes, if we go ahead and focus on generating the use cases from GPT itself, training models from GPT itself, I should be able to do better. So understand, AI is going to level the field, and people who are going to go beyond AI are going to win in the market. So this framework has actually helped me out a lot in order to help people whether to use or not to use AI. You should definitely use AI, but you should understand that cannot create the best work. You have to get your insights in it.

   4. AGAakash Gupta34:55

      Yes. And also writing a PRD is often the process of thinking through things. So if you outsource your thinking, you're gonna lose that critical element.

   5. ASAnkit Shukla35:07

      Yes.

   6. AGAakash Gupta35:08

      And we talked about those stakeholders being so important, right? If the stakeholder management is the part of the job that is really creating the stickiness in the job, you need to make sure you do a good job. You don't want them to say, "Hey, I could just replace Aakash with an AI agent who is creating the PRD for me." So it's very important to maybe give a very good prompt to that AI agent, maybe have that AI agent or that LLM create your first draft, but then add in your own unique perspective, up-level that, so that they-- everybody can see the value you are adding and the craftsmanship that you are working with.

   7. ASAnkit Shukla35:46

      So that's completely true. And just that you have to understand where you can add value, and that is a very important thing that, uh, uh, like slightly digressing from the topic, but what is happening right now is that initially you understand that we as humans were actually going into fields and earning our own food and we were more strengthful. Now we have to go to a gym in order to keep ourself more fit. And that can start happening with AI because if your work is to think like a product manager, do product thinking, have some product sense, if you're just outsourcing it to AI, then

9. 36:17 – 40:52

   The AI PM checklist: from problem hypothesis to model choice, data, cost, UX, and evals

   1. ASAnkit Shukla36:17

      most of the effort that you're putting is into going ahead and making sure that you are able to think. And the best method to think is to go ahead and practice in your product, which is generate PRD with your own mind, go ahead and talk to the customers, and that connection should keep on happening. So now just to brief everything that we have done so far.In product management, we, we divi- we tend to divide everything into a problem space and a solution space. Now, problem space comes from four things. First is what are the business challenges that you are facing? User empathy, you should understand the users, what are the workflow optimizations that they can do? To give an example, let's say before Cursor, what used to happen was I'm stuck in, uh, in a line of code. I don't know how to do that. I'll go to Google, I'll search, I'll go to Stack Overflow, I'll have some opinionated answers, and if I go ahead and post a question, I'm going to get a thrash from people. And then I'll find a answer out there, and then I'll try to paste it. Now what, uh, Cursor does, you just go ahead and try to give it a simple prompt. Or even without prompt, you just give it a code base, it is going to generate it for you. Okay. So that is how you understand where people are stuck, which is workflow optimizations. And I use the term workflow because most of the AI use cases right now are in B2B space, most of the revenue-making companies. So workflow is an important work there. Understand the workflow which can be optimized and then work together for the same. Then there are market opportunities, and then there is data analytics. So now this is a problem space. Understand you cannot become a PM just by understanding the tools and build products and all. There are many people who come to me who say that, "I've given a-- I have created a product," and then they go ahead and give me a lovable link. I understand that you have created it from just a prompt, and that would not be even a PRD. So you have not done the critical thinking behind it. Understand you are a PM not because of this. This thing is more important. This is where everything should start, right? And then what happens is that then there is a second part, which is the solution space. Now, in solution space, there are multiple things that AI can do for you. I have already separated into predictive and generative AI, which is recommendations and ranking, clustering, categorization, forecasting and regression, anomaly detection, and contextual content creation, which is generative. Now, how do you connect both of these? You know what AI can do. You know what problem do you need to solve. So now the connecting part is this. Okay. This is what a AI product manager needs to do. If you are already an AI product manager, make sure that you create it as a checklist. If you are not an AI product manager, make sure that you understand it really, really well. These are the questions that are going to be asked to you, and this is the work that you have to exactly do in the job. So the first part is you need to have in-depth understanding of the problem. This is non-negotiable. You need to have clear product hypothesis, how the solution is going to look like, what this product is going to be, why it is going to run, why it is not going to run, what are the risks, what are the advantages. Then if it's a generative AI product or maybe even a predictive AI product, you have to understand few nuances which are AI product specific, which is choice of model. Is it a fine-tune model? Is it a thinking model? How do you contextualize it? So in terms of data and training, how do you context-contextualize it? Is it prompt engineering is enough? You have to do fine-tuning. You have to use RAG. Where will you get the data from? How the data pipeline works. Similarly, there are cost considerations. There is a cost that AI brings in terms of, uh, the token cost, hosting your own models and all. Infrastructure requirements, feedback and evaluation. Evaluations and iterations are a really important part because this is not a very predictive system. Then user experience and considerations. This is between both. It happens very importantly in traditional product management, but AI products also have their own, I would say, uh, uh, nuances of building the product. Like there is streaming happens, the output is unpredictable, so how do you go ahead and deal with them? And then there is launch and growth, and then tech, data, and design collaboration. This is everything is the same as in traditional PM, but these things are going to be some nuances that you need to understand as a product manager while implementing any kind of AI product.

   2. SPSpeaker40:21

      Brilliant.

   3. ASAnkit Shukla40:22

      Yes. So this I've already told you, we'll share the slide so that you are able to get a, a good look. I'm sure my handwriting was not correct there, so this will help you out. That is why I kept this. Yes. Now this is like an important part. Now understand a most common mistake that people make while getting into AI product management or any new field is they start with a long list of to-do items, and generally the initial two items are super, super difficult, and they don't go beyond

10. 40:52 – 46:26

    Don’t start with ML math: use Marty Cagan’s 4 risks (value, usability, feasibility, viability)

    1. ASAnkit Shukla40:52

       that. So as a product manager, my recommendation to everyone who wants to become an AI product manager is to go ahead and understand this framework which the mighty Marty Cagan has given. If you are building any product, you have to look at it from four angles. The first angle is easy, whether it is valuable for the people or not. If there is no reason to exist for this product in the life of the user, it should not exist. Second is usable. Can people use it? Which is a UI/UX part of it. Is it feasible? Can the tech create it? What models are you going to use? What is the tech infrastructure? What is the data science and ML behind the same? How is it scalable, structured? And then whether it makes sense for the business or not, which is viable. Generally, valuable and viable are PM responsibilities. Usable is a designer's responsibility if you are talking about a product with a GUI. Or if you are talking about an API product, then it is the responsibility of someone who's writing the documentation and all. And then there is feasibility, who is an engineering's job-- engineer's job. The most important mistake or the critical mistake that people are making these days is they just think about AI as this. I should straight away, if I want to become an AI product manager, I should straight away get into statistics. So they try to understand what AI and ML is. Then they get into neural networks, then they get into data science, then they get into understanding what are the mathematics behind it, then they go into statistics, and by that period of time, your motivation to go ahead and do something, build something, has already gone out, right? So if you go ahead and understand, uh, there's an Instagram story that, uh, the creators of Instagram did not know how to code. But they were very, like, they were very driven by that idea of going ahead and creating an app that is going to do something. Okay, it was not Instagram at that period of time, but they were very, uh, uh, very ambitious about it. They were very excited about it. So they carried that excitement. Understand that motivation is perishable.Your motivation, your inspiration is perishable. So if you just go ahead and try to do the wrong things at the right time, it will not-- it is not going to work. So make sure that you go out and start with a purpose. And your purpose as a product manager is understand, even if you are good with tech, even if you are good with tech, what will happen is the company has employed much better engineers than you. So let the best people do the job. I'm not saying that you should not understand tech. You should understand tech. Everyone should understand tech. But understand, you should understand your limitations, and you should play on your strengths. Basics is good, but do not try to get into the details prematurely. Once you join a company, you understand that these are the nuances I want to understand, then you can learn. But right now, as a product manager, you can contribute the most in the valuable part and the viable part, right? So if you look at products such as Perplexity, everyone talks about it's an AI product, LLM products and all, but boss, try to understand that if I divide Perplexity into these four parts, the first part is the product is usable. Even if it is using the most complicated AI in the world, most, uh, costly AI, but if it does not serve the purpose, it is not going to be there. So product manager has to define the same. So if I talk about the valuable, as a researcher user-- as a user who's a researcher, I want to be able to accelerate my research process. As a researcher, I want to check the sources of information in the LLM-generated content. As a researcher, I want to choose and try between different LLM models. These are the use cases. How do you do this? Same traditional PM thing. User personas, ideal customer profiles, customer journey maps, jobs-to-be-done, MOM Test. Understand these things and their valuable part is done. And also try to find out the use cases of AI, as in what AI is doing for other products. That's the first valuable part. Then for usable, you can look at the UI, understand where people are stuck, how you can make it better. Same problem if AI is solving that is-- that demands a different kind of interface. For example, both of them, both of Google and Perplexity, are solving the information problem, like finding the right information, but the UIs are entirely different. So you have to innovate on the same. Similarly, Cursor and, uh, uh, this Visual Studio Code and Sublime Text, although they might have the same kind of UI, but how it acts at the back end is going to be different. So you have to understand that UI part and all. And then usable part is there and done. In terms of feasibility, there are a few things that product managers can understand, but my advice generally is understand the basics. I'll tell you some basics in this, uh, class as well. You can understand more basics. But don't try to kill yourself if you cannot go out and understand that algorithm that AI is going to use at the back end, right? If you don't trust me, go ahead, go to Andrej Karpathy's videos around how he explains LLMs to individuals. Okay. Look at that video. I think that's a three, three and a half hour long video. You will get like a very good understanding of how things are explained and what you are supposed to know. You do not have to get into the trenches of deep tech, because then you are going to demotivate yourself. Do it once you have a purpose, you understand what you need to do, and then you can get into the trenches. Yes. And then there is a viability part as well, which is cost of tokens you have to consider as a PM, infrastructure cost. So cost of tokens is going to help you understand how to write your evaluations correctly so that it gets, let's say, least amount of, uh, tokens and saves you some cost. Pricing and monetization strategy, fitment with the company strategy, possibility of scale, business partnerships. Also, there are a few considerations. If an engineering is going out and developing the same, generally engineers have a tendency to over-optimize or pre-optimize, premature optimization you call it. So they'll focus on cost of tokens and all. But you as a PM will understand that in zero to one problems, you don't have to even focus on the scale right now. Just get the validation there, and then you can think about all of these things. So use the best model for zero to one, and then you can

11. 46:26 – 55:52

    Contextualization 101: prompt engineering vs. RAG vs. fine-tuning

    1. ASAnkit Shukla46:26

       see what you can go ahead and do. So understand this perspective, someone has to bring, and this is super critical for succeeding as a-- succeeding in making sure that the product is being successful. And this is being missed out if you focus on the wrong things as an AI PM. Yes. So this is just a recap. Now, this is a big slide. Here we can talk about the building blocks of AI. Understand that there is a concept in AI which is called as build, buy or borrow or steal. So here what happens is you, at every pi- point of time, so for example, these are infrastructure building blocks of an AI product, which is you have to use models, data layer, infrastructure, user interface, trust, security, feedback, prompt engineering, business layer, and multiple other things. And there is also something called as contextualization, which is how do you make your product contextual to the AI uses. I'll come to it. Now, understand that in all of these, because open source is developing, because so many people are building in AI, you have options, which is you can either build and you can borrow something open source, you can go ahead and take it on rent from OpenAI, or you can go ahead and just put a model and then you can tweak it as per your own understanding. So all of these things are, uh, available out there. And a very important thing in any of the product is that I'm going to come to is this. So now here is where the practical AI PM thing starts. Understand, I can divide the AI products into multiple parts. So the AI ecosystem is as follows. The first is we have the companies which are giving you the core bare metal infra. One of the winner companies here is NVIDIA, and other companies are trying to catch. Then we have something, people who are, let's say, cloud providers, where these things are hosted, or people who are, let's say, the model companies. And model companies and cloud providers have their, let's say, they have some kind of arrangement within themselves. These are OpenAI, these are Google's Vertex, these are Amazon SageMaker, and these kind of products, right? Now, there is a third and the biggest layer out there, which is the application. So now you have to think that what are the use cases where I can go ahead and build a product or improve my product. For example is, if you understand this slide, let's say if you ask Notion, if you ask a ChatGPT to create an essay or something, it will be able to create for you. But if you ask, "What is in my to-do list for tomorrow?" It won't be able to ask. It won't be able to answer. Why? Because it does not have your contextBut if you look at Notion, if you are a product manager at Notion, you'll be able to understand that people are stuck at few places, which is they cannot talk to their Notion in natural language. So now the power of LLM plus the context of Notion can create an amazing product called Notion, right? So now if I ask for what is in my project plan this day, it is going to go to my Notion, understand, and then it's going to give me the answer. Now, LLM has become more personalized to me, more helpful for me. Similarly, with Stripe, they were facing a problem that they have a very, uh, uh, long documentation, and when developers want to build their own products on the top of Stripe in order to, uh, integrate the payment gateway, they have to go through that documentation. So they created like a small chatbot where you can just enter the prompt, it will go ahead and generate everything for you. So power of LLM plus the documentation of Stripe, new powers unlocked for the developers. Similarly, this is a very interesting example, Atlassian Intelligence. So now not only it can search content for you, it can also do actions. For example, you have to find the right tickets, you have to use Jira query language. If you have to define the stories and tasks, you have to do it manually. Now, by reading through your PRD and everything and all of your context that is there in your Confluence or your Jira or Atlassian dashboard, it has all of your tickets and everything. Now imagine a powerful AI sitting on top of your Jira, your Confluence, or any other tool. So this is called as contextualization, and contextualization can happen in three ways. Okay, so this is an important part. We call this thing as contextualization, and contextualization right now can happen in three ways. Contextualization means how do you give your data to AI to help you go ahead and do a particular specific job. Now, contextualization can happen in three ways. One is very simple, your prompt engineering. Second is your, uh, what do we call it? RAG. And third is your fine-tuning. So I've created a detailed guide on, uh, uh, fi-- or prompt engineering. Uh, if you want it, I'll share it with Akash, and you guys can let me know-- let us know in the comments, we'll be able to share it with you as well. So prompt engineering, let's say if you are creating a product such as Chat PRD, where you give it a context that, "Go ahead, create a PRD for me." Let's say if I tell ChatGPT right now that, "Create a PRD for me," it is going to do maybe an average job. But if I give it a good template created by top product managers at top industries based on a particular domain, and then I can give it a context as a prompt, and then it will be able to go ahead and generate a better PRD. That is what products like Chat PRD do. They have most of the context. Because it's a simple context, it can be contained in a particular prompt. Second is RAG, which is a bit, uh, uh, uh, uh, a bit complex than prompt engineering, where what you do is... So this is how RAG happens. RAG, yes, RAG stands for Retrieval Augment and Generation. In RAG, what we do is we just go ahead, whatever information that we have, for example, your Notion database. Now, if you try to put all of your Notion base-- database into the prompt engineering context, either it will run out of context or it is going to cost you more dollars, so you cannot practically do it. And also it can keep on changing, so how will you change the prompt and everything every time? So what you do is you create your knowledge, you collect all of your knowledge, which is the Notion database. After that, you create its-- create, like, convert it into something called as embeddings or vector embeddings. There are APIs for the same. Your developers would know how to do this. And as people are adding the data to your Notion, the embeddings would be automatically created, right? Now these embeddings are created. Whenever you put any prompt, let's say you search for, uh, let's say you take the transcript of this video and you talk about-- and you talk to the LLM, say that, "What is the speaker talking about RAG in this video?" Right? So what it will do is the particular, uh, s-transcript that I have, this transcript is going to be broken down into chunks and embeddings. Every embedding would be somewhat meaningful in itself. It could be paragraph-based, it could be line-based. It depends on what model that you are using. Now, you have all the embeddings. Your LLM will understand where is the context. So now you are talking about RAG. So now you are going to put a system which is going to search everything in this knowledge source which contains RAG or similar kind of system. And it will not, let's say, search like a spelling search or an English search, it is going to search in the form of embedding. So embedding, if you can't understand embedding, embedding is very simple. It is like a position that is given to your content in a space. Things which have the same meaning are going to have similar kind of embeddings. They are going to have less distance from each other. Okay, so now you are going to go ahead and find what are the similar embeddings, things which are more contextually similar to each other, right? Then you will have some things which are going to match. So let's say right now we are, let's say at, uh, uh, one-hour mark. So if you go ahead and see it at forty minutes between one hour, you are going to get all the chunks which where we, I'm, I'm talking about RAG. So only that part of information is going to be gathered. It is going to become a part of the prompt. So now your prompt will become, in this video, in this, uh, in this, uh, context, what is being talked about in RAG. And then all of your important things that are a part of this knowledge source, which contains RAG, like top end things, are going to be embedded in that particular prompt. And then it will be going to an LLM, and then you have the answer. So RAG is very important. Why? Because it can save you a lot of time in, let's say, in money in terms of building the prompt engineering in context. It can save you money in context. Second is, it is very real-time. Always, like a Notion database is gr- increasing, your calendar is increasing, you can always go ahead and keep on adding more and more data to your knowledge sources. So most of the products these days, they are using some version of prompt engineering or RAG. The third thing is fine-tuning. So all the players such as OpenAI, Amazon, are going to give you a method to fine-tune your models on your own data. But there are two considerations that you should have. Fine-tuning generally makes sense when you have a lot of data to train on. You have to make your product more, let's say, specific to a domain. Second is it has a cost. So-You have to, like, think about that cost before you do that. And the third part is it is not real time. You have to again and again fine-tune if you have to make sure that it is containing the real-time information from the user. So I think that is a good mental model for people to have for contextualization. If it's easy, like start with prompt engineering, then go to RAG, and then most of the time you don't have to go to fine-tuning, but you can decide on the basis of the maturity of your product.

    2. AGAakash Gupta55:19

       Yeah, fine-tuning is really good for like a specialized use case. Like if you can give it a lot of really good examples, successful data, like here's one thousand input and one thousand output examples, then the model can really quickly fine-tune. Like if you wanna be able to create a regression based on natural text, for instance, fine-tuning is very powerful. RAG, very powerful for having access to updating data, updating knowledge base-

    3. ASAnkit Shukla55:47

       Yes

    4. AGAakash Gupta55:47

       ... updating calendars. But prompt engineering, good for eighty percent of use cases.

12. 55:52 – 58:25

    LLM limitations & why AI evals become a core PM skill

    1. ASAnkit Shukla55:52

       Yes, yes. Correct. At least for the start, yes. At least for the start, it's completely good. And now, because of, because now we know-- don't know how to move from prompt engineering to RAG and all, there is one more important concept which everyone should know, which is know the limitations of AI, which is it hallucinates, it has biases, it is indeterministic, and the reason is it gives you answers in full confidence, even if it is completely wrong. And then that is where most of the people get stuck. Understand, in traditional, uh, building traditional products, let's say software products built on CRUD and all, it's easier for us to test because they were very deterministic. But now in AI, you have to make sure that as a product manager, and most of the leaders have already told that, in AI product management, a very major role that a product manager has to play is not in just creating PRD. That is anyways you have to create. But it is in creating the right kind of evaluations. And consider evaluations like test, like a performance check, whether my LLM is doing correctly-- things correctly or not. Because you don't know what it is going to go ahead and generate in the real time for the users. So evaluation is a very important concept. Okay. And in evaluation, what you do is, understand it's not very difficult. It's like creating a PRD. I can tell you the template. But in order to fill that, you have to do a lot of research, you have to make it more contextual. So adding contextual evaluation is different, but I can tell you the process. The process is very simple. You just have to understand that what is the output of your particular LLM. Now, you have some considerations for that output, whether it is factually correct or not, whether it follows a particular structure. For example, if you are creating it for an application or a software to consume, you want it ideally to be in a particular structure, maybe a JSON format. So is it following the same or not? Is that some kind of biases or not? What is the time it is taking to generate? These kind of things. So what you can do is you can go ahead and evaluate that output. Okay, and there are two kinds of, uh, I would say evaluations out there. One is offline evaluation, other is online evaluation. Offline evaluation is that is done before your product has launched, so that you are able to consider it like a test. But in understanding LLMs, because things rapidly move, you all-- and you have to test things in real time, understand that online evaluations are not only close correction mechanisms, but they are also your beacons like analytics. They'll help you understand how your, uh, let's say, AI is performing. So just to give you a use case how this actually happened, because this is very abstract. Okay. Uh, so first of all, evaluations

13. 58:25 – 1:03:11

    Worked example: building an AI job site + using an eval model-in-the-loop

    1. ASAnkit Shukla58:25

       help you check for correct syntax and format, hallucination, biases, and other ne-necessary checks based on your use case. This is a case study that I can give, okay? Let's say we want to create a AI-first job website. And now this is what the product is. What I'm trying to do is I will first crawl all the jobs from the internet. I don't have to reinvent the wheel, I don't have to create a job website again. I can just crawl all the websites from the internet, all the jobs from the internet. After that, whatever is the job description that I get, I am able to find a problem that most of the HRs and hiring managers do a very lazy job at dividing the job description. So what I can do is I can take AI help-- AI's help in order to generate a good summary of the job description in better words. Maybe I can list some possible interview questions from the job description itself. I can also tell the skills that are needed. I can create a learning guide with so many resources. If you want, if you are preparing, if you have ten days, you can go ahead and do this. And then I can also give a small quiz for assessment. Now you see that this job description, small description, is much more powerful for, powerful for someone who's actually looking seriously for this job. Okay.

    2. AGAakash Gupta59:38

       I want this product. [chuckles]

    3. ASAnkit Shukla59:39

       Yeah. Correct. So, and now guys, this is the idea that you guys can go ahead and maybe, uh, start using Lovable or any product in order to build this, okay? You learn AI by not just consuming the information that we are giving you, but actually going ahead and implementing all of these things, right? So this can be a powerful product, but now this product has-- can have some issues. The issues could be, what if the summary of job description is all hallucinated? What if the possible interview questions are all over the place? What if the structure of the quiz is not good? Because quiz is going to be in-- is going to be a JSON, which is going to be fed into your system that can generate a quiz. What if the, uh, the system is not correct? Okay. So what we'll do is we will put an evaluation in the loop. So this will happen. This will get entered into your database. After this, you are going to create another prompt. Generally, you are going to give it a prompt. So let's say if you are testing for something very objective, is this a JSON format or not? Then you can use code. You can just give it to code, and code will tell you whether it is JSON or not. But most of the time you are dealing with indeterministic outputs, which is qualitative, which is subjective. So what you will do is you will put another model, you will give all of these things and your prompt to this model, which is a more intelligent model. Here you can use any kind of model that you want, but generally for evaluations, we generally use a more intelligent model, which, which is, let's say, a little bit better in thinking. And then you will againBased on the running from your evaluations, you are going to tweak. What you are going to tweak? You are either going to tweak whether, let's say, uh, uh, if the output was correct but the evaluation was wrong, then you are going to improve the evaluation. If the evaluation was, let's say, correct, and your output is, let's say, uh, not correct, then you are going to understand what is wrong there. If evaluation is wrong and the output is also wrong, which is good, then you understand that, boss, there is something wrong that I'm doing in my prompt engineering or my RAG. So you have to fine-tune that, right? So you have to make sure that you are able to understand. This will give you the alert that something is not right. Then as a product manager, you have to apply your problem-solving mindset in order to understand whether the data should improve or the prompt engineering should improve, or your RAG modeling and fetching of the data should improve, right? Now, what does the evaluation looks like? I'll show you the exact file. Yeah. So here we are writing this. If you are-- you are a content quality evaluator for product management job listings, and then given the job description, summary, interview questions, skill JSON, concept JSON, just everything, just go ahead and evaluate this on the following checklist. And then respond in this following format. So now what will happen is you will end up creating a dashboard for your human auditors or content editors who will check all of these things, who will get a dashboard where red flags and green flags will be mentioned, and they'll be able to check, and they'll be able to tell you whether we need to improve prompt or we need to do something else. So now your AI is not a magic box or a black box now. You are now able to see what is happening, and you are able to further go ahead and improve it. It will not remove the hallucination by 100%, but it will at least tell you to take a be-take a better path. Cool. So this is-- these are, like, the two important concept, which is RAG and which is, uh, uh, contextualization and also making sure that you are able to run evaluation. Understand evaluation is a very, very, very huge, large thing. We teach, let's say, a lot of details of evaluation in the, the different program that we have. And here what we are talking about is just to give you a backdrop of everything again. Here we are trying to connect these two things. We

14. 1:03:11 – 1:11:15

    From copilots to agents: action-taking systems, automation stacks, and MCP integration

    1. ASAnkit Shukla1:03:11

       are trying to connect these two things. We know what the problem is. We know what the solution is. Now we are understanding some AI nuances in order to make the product work. Cool. And then maybe I can briefly talk about AI agents, which are all popular these days. So- [chuckles] Yeah. So what happens with AI agents is understand that so far we have been only giving the thinking work to, uh, to the AI. But we don't need more people who can think. We need more people who can go ahead and do the job. Like, you have to take action. So you give the power of taking actions to your AI tools, and then you have an AI agent. So what an AI agent is, it is intelligence of the LLM, action based on the tools that you give access to, and then there is autonomy. Okay, I'll give you an example. Let's say I get a lot of invitations for podcast, for example, okay? But I also get a lot of invitations for, for let's say, for podcast which are not so good and maybe which do not have the right kind of, let's say, subscribers that I want to, okay. Now I want to understand, like, I don't want to waste time in order to sort everything out, so, and to call-- talk to people one-on-one. So I want to understand-- I want to make sure that I'm only sharing my Calendly link or scheduling time with people who actually are going to serve a purpose for my audience. So what I can do is I can build a small pipeline, okay? One is whenever I get a Gmail on my-- a mail on my Gmail, I should be able to categorize whether this is an invitation or this is, let's say, a general mail. Now, this is an invitation. This invitation is, let's say, for YouTube, and I want to understand how many subs they might have, right? Or maybe what is their audience, right? And then whether it is not in this particular category, right? And then if they are following all the criteria, I can just go ahead and send them a reply with my Calendly link, which is only for these slots. Because my general Calendly links are fifteen minutes, but these slots should be one hour, two hour long, because podcast is going to take a lot of time if you have to spend time on it, right? So now, right now it takes me or my VA some, let's say, some time in order to sort all of these things if we have to be on the top of an email. But this is a simple use case for an AI agent. So what is an AI agent? First of all, an AI agent is going to-- I can write an AI agent where Gmail will act as a tool. So I'll connect my AI agent with Gmail. It is going to read all the incoming emails, so it is using a tool. Then it is going to think. It is-- It will patch, fetch all the information in the form of text. It is going to categorize them into this or this. That particular kind of email, it is going to find the YouTube address, and then it is going to go to a UPI in order to understand what are the number of subscribers and audience that this person has. I think n8n has all of these things available. Uh, n8n, I'm going to come to it, what it is. And then I can go ahead and again give it a access to my Gmail API, Gmail access, so that it can go ahead and take action of sending an email. And then I can decide which Calendly link should be sent. So these are simple use case which can save a lot of time and hassle for me. So agents are nothing but brain, plus ability to take action and some kind of intelligence, which is ability to understand what they are doing, right? And there are so many tools. There is It, n8n, Zapier, and so many, but my recommendation is if you are a beginner, please try this agents.ai, right? Please try this agents.ai. It's very simple to use. You can create, like, models within, I think, ten, fifteen minutes if you are-- if you know what you are doing, right? Oh, this is new for me. Yes. So it is, it is good. So it's created by the founder of HubSpot. It was his pet project, and now it has got very popular. Very simple as compared to other players out there. Another recommendation that I have for Ev is-- uh, that I have for everyone is if you are, let's say, already a technical personWho are really serious about getting into this AI thing, uh, then also consider learning about LangGraph and LangChain. Because all of these things that you're seeing at the screen and it and all, they are just wrappers of the same. So if you are someone who are a bit more technical, wants to understand the trenches and all, my recommendation is at least learn about LangChain and LangGraph. You'll understand what is the possibilities that it can open up for you. Yes. So this is what we have already covered. And yes, now with new problems come new solutions, now we are going to run into a different set of problems altogether. So now the problems is, there are hundreds, if not thousands of tools in the world. How do you make sure that your LLM is able to go ahead and connect with them? Let's say Gmail has done it nicely, but what if you want to connect with Jira or Slack or GitHub? Do you want to depend on these companies to make it API simpler for you? No. So what happens is, because agent needs to connect to an external world, some amazing product managers at Anthropic came up with a great idea. That idea was model context protocol, which is everyone has APIs, but what if we can convert the API into something that model can understand with natural language? So a set of, uh, uh, a set of, I would say, structure or policies came from Anthropic. Now OpenAI has also generated the same, based on which you can create your APIs into some model context protocol. So what happens in model context protocol is, what you do is you have-- you are connecting with your LLM with an MCP. All these companies are either going to provide the MCP or you can also create their MCPs. What is an MCP? MCP is simple. There is an API which understand, get, post, request and these kind of things. Now, you can convert it by using the OpenAI specs in order to create an into an MCP. And then that MCP is going to rest in your computer or on your, like, besides your LLM, so that you can go ahead and model context protocol server, and then it should be able to convert these LLM calls into a unified thing. So you don't have to worry about integrating with everything. Just use your MCP and then it will be able to help you in the, uh, natural language, you should be able to have a conversation with them. Okay. And if you want to see an example what an MCP looks like, then you guys can go to this. Razorpay was the first fintech Indian company which was able to do so. Yes, Razorpay MCP server. Hmm, so this is Razorpay MCP server, and this is what they do. Okay. Right now, let's say if you want to generate a payment link on Razorpay, you want to collect payment from someone, they are a payment gateway. You have to go to their website, you have to click on something, right? Now you don't want to do this. You have a Claude that you are using, you have a ChatGPT that you are using, or you have some kind of agent that you are using. You wanted to create a payment link. You wanted to collect this kind of data. So what you can do is, these are the tools that the MCP provides you. You can capture payments, orders, and everything with natural language. Okay, so this is how it works, use cases and examples. Right, so this is they have taken example of Claude. Okay, so on Claude desktop, if you install the MCP server of this, you can just write this prompt: "Hello, please create a payment link for rupees ten send to Kushal on this." And then it will go ahead and generate it for you. It will call the API on its own, you-- it has your keys. And also understand, uh, people who have created MCPs, they have also taken care of security. So generally, it is recommended that if you are creating, like, using an MCP, if it is for important purposes such as fintech, health and all, it is recommended to use the official MCP that is given by these servers, because they have stopped some things that shall not-- they have stopped some accesses which are not-- which are-- which should not be exposed to the LLMs. For example, getting your money into your own account or your someone else's account, right? So that is a quick view on MCP. Yes. So this is it. So your AI host, such as Chat, GPT, Curzer, or, uh, Claude, is called as MCP client, which is going to consume the content from MCP. Then this is a connector MCP server, which connects, converts this large, like, this language into the APIs and all. Okay, and then the remote party could contains file database, APIs, or anything that you want, maybe an action, right? Yes,

15. 1:11:15 – 1:20:37

    How to actually land AI PM roles: job description mining → portfolio ladder → targeted outreach

    1. ASAnkit Shukla1:11:15

       and this is how you can create. Yes. Now talking about getting the whole circle back again, how you can go ahead and become an AI product manager. Okay, so you should have all the information that I have told you. You should practice it. Always remember that in these sessions or in this podcast, maximum what we guys can give you is information. But what you need is not information. What you need is knowledge. So how do you convert information into knowledge? You take action and you do a lot of reflection, right? So we have a policy that we focus on, we force people to go out and learn by doing things. That is how we have designed all the curriculums and all. So my suggestion for everyone is, if you have watched this video so far, please go ahead. If you really are interested into AIPM, okay, show people that you know AIPM. Okay, so look at this. People come to us, hiring managers, and they say, "I want to become an AIPM." But why do you want to become an AIPM? And people say that, "I'm passionate about it." But your passionate-- your passion does not converge to anything for that particular individual. So they want to understand whether you can do the job or not. How do they understand? They can either understand from your past experience, but most of you might not have it. Or even if you have it, that is not an AI. So people want to understand, people want to reduce their risk, that if I have this person, they should be able to do the job. That's it. So if you can showcase them that I have done the similar kind of work. So what you should do is, my recommendation to everyone is, and this is the most important part of this podcast is, please search for AI product manager jobs on LinkedIn or whatever site is popular in your region. Understand the job description really well. Okay, let me show you, I might have something. This is an exercise I keep on doing often to make sure that we are also talk-- taking, like te-- like teaching you all the relevant content because AI is moving too, too fast. So let's say I have created this spreadsheet. So what I have done is, I have searched for AI product manager roles.On LinkedIn. And this I've done just yesterday. Okay. Before creating the program also, we keep on doing this for every other cohort to make sure that we are giving you only the relevant information, right? So I understood that at Razorpay they are hiring a product manager too for AI. I have gone through the job description, the link is mentioned. And then I was able to understand line by line what is it that they are looking out for. This is a number one exercise that you need to do. If you want to be an AI PM, your roadmap is this. Go through at least 20 job descriptions for AI product managers. If there is no AI product manager, look for product manager which have some mention of AI. Understand what they are supposed to do, what do they need to do. And then try to question yourself that, let's say, if I have to showcase that I have these things, I know these things, how am I going to do this? How do I prove to this person that I know these things? And then you have the answer in your portfolio. So either you can create, let's say, uh, this is the ladder, this is the most easiest part, and this is the, this is the most easiest part, and this is the most difficult part. So if you are starting here, let's say if you don't have any context, you start with commentary or content. Write a substack, write it on LinkedIn what you have learned. It will help you reflect. Understand, as Akash told you, that writing is a form of thinking, improving your own thinking, right? So this is like a very beginner level thing that you can do. After that, you can also pick up AI products. This is a fantastic thing. If you want to learn AI product management, what you should do is look at the top products. Granula, Cursor, Grammarly, ChatGPT. Try to reverse engineer them, and then you'll understand a lot better, okay? So do product teardown case studies, then try to suggest some improvements, then build some side projects. There are-- Like, it is much easier today than ever to go ahead and [coughs] build a side project. But if you're creating like a project, side project as a product manager, please do not focus just on the implementation. Focus on the synthesis as well, as in how did you arrive to this? People are always going to ask you why this product is needed, right? And then you can also add, if you have, you can also add your past work. But I understand most people don't have the past work because either they have not worked as a PM, or they have nothing relevant, or they are bounded by the D- NDA agreements of their companies, okay. So this is the scale. As we grow ahead, go at the top, things become more difficult to do, but they become more effective. So if you do this, I am sure you'll have some proof of work to prove to people that you become product managers. Understand, guys, right now you have a big leverage. Your leverage is that AI is very, very new. Very new as in a lot of people do not know what is happening there. So now it is your advantage to go ahead and stand out by your hard work, okay? So make sure that before you desire, you deserve the role that you desire, right? And this is, like, something that I can give you to, for, to, for you to take action. Reverse engineer top AI products. Understand gaps, frustrations in your personal and professional workflow. Build AI tools or agents to solve those problems. I am specifically mentioned that you should go ahead and talk about your personal and professional workflow, because if you try to solve your own problems, you will also find people who are facing a similar kind of problems, okay? And as a product manager, if you're building a side projects, more important than what is the tech that you have used for product is important, how many times have you iterated on the basis of customer feedback? So if you are building a side project as a product manager, you definitely should have users. And you can only get users if you are solving some relatable problems, even if these are very, very small problems, right? And then whatever you are learning, start writing what you learn. Put yourself out there, right? Uh, you see two of us, most of, like, uh, uh, the success that we go ahead and, uh, attribute right now is to just going ahead and we have increased our surface area of luck by putting ourself out on Substack and LinkedIn. That focuses us to keep on learning always. And then keep a track of AI companies, right? You can go to Y Combinator, Slash Companies. You can go to Product Hunt. You can look for new AI companies. So there is also a, a website that less people are aware of, which is FinSMEs. I'm not sure if this is .in or .com or something, but it maintains a record of all the companies that are getting funded. So what you can do is you can just look at companies which are getting funded, if they are AI. Then what you can do is choose a list of 15, 20 companies, and this is the ultimate hack, right? You have created a portfolio, you have done anything, and maybe you are not getting the job. So this is the ultimate hack. Create a list of companies where you are interested in, maybe 15, 20 companies. They should not be billion-dollar mammoths, because HRs are going to do the gatekeeping. What you should do is find out companies which are small, which are medium-sized companies, maybe, uh, 100 to 100, 500 folks. And then imagine if you were the product manager at that company, what would you do in your first six to eight months? Build it. Like, if it's an AI company, go ahead and try to build something for them, help them build a roadmap, do something for them, and then make sure that you are able to follow up. Follow up as in make sure that you are following three times before calling it a quit. So if you do this, I'm sure you should be able to go ahead and become a product, AI product manager.

    2. AGAakash Gupta1:18:22

       Wow. That was easily the most thorough soup to nuts masterclass I have seen on AI product management. Go take action. Go follow Ankit's YouTube, go follow his LinkedIn, and go check out Hello PM. Is there anything else people should do if they want to get in touch with you?

    3. ASAnkit Shukla1:18:43

       Yeah. So what people can do is, my first recommendation is please go to our YouTube channel, okay? We have created playlist, like a new playlist called as, uh, Getting Into Product Management, right? Everyone, before you contact us or contact any other program or any other course, my recommendation is please go through the free content that will take you three, four hours. Once you understand, then you can go ahead and maybe go to our website called as hellopm.co. You can check it out, check out, check us out on LinkedIn as well. Then you can go ahead and look at the course curriculum from there, and then you can go ahead and apply from there. You can also, there is also an option for scheduling a one-on-one call with our team so that you are able to understand if the program is good for you or not. One important thing that I want to tell everyone is, understand, guys, product management is very popular these days. Maybe more popular than it should be. So I don't want you to just go ahead and join us or any other program just because it is very popular and you are having a sense of missing out. Make sure that you are taking a calculated decision. It is your important time, and if you commit to something, you have to commit it, like, really 100% to it. So my suggestion is, if you are just exploring whether you should get into product management, AI product management, yes or no, you are not decided. Before you consider any paid program, before you invest your money anywhere, go ahead and look at that free playlist. That will help you answer most of the questions, and then you are welcome to go ahead and explore most of the things on your own. I'm sure people are, uh, very intelligent to take those own calls.

    4. AGAakash Gupta1:20:06

       Yes. There are probably too many people interested in AI product management compared to the number of jobs. So realize that it's not gonna be that easy, but it is totally doable, and Ankit has just given you the exact roadmap. I hope you guys enjoyed that podcast with Ankit. If you want to check out the full deck that he shared, be sure to check out my newsletter issue where we walk through all of these and include all of the slides. And with that, we'll see you guys in the next episode.

Episode duration: 1:20:42