EVERY SPOKEN WORD
60 min read · 11,628 words- PHPawel Huryn
In today's episode, we cover everything, prompting, fine-tuning, RAG building that you need to know to become an AI product managers.
- AGAakash Gupta
Are all PMs gonna need to become AI PMs?
- PHPawel Huryn
The AI market is growing so fast that there is a high probability that we will need more AI product managers in the future.
- AGAakash Gupta
You can't just prompt like an average person, right? You need to be prompting at a very high level.
- PHPawel Huryn
And we instruct ChatGPT to think separately from different perspectives. I like adding that you will get $1,000 if you perform this task on a champion level or something like that.
- AGAakash Gupta
Why are AI PMs paid so much?
- PHPawel Huryn
One of the reasons is that they need to combine business skills with the technical knowledge. Then based on this, uh, generic information, it plans up to 11 separate tasks, and those tasks are distributed to different agents.
- AGAakash Gupta
It's not that everyone needs to become an AI PM, but this market is growing really fast, and we just gave you all the tools to become an AI product manager. Looks like that's ultimate list of product metrics. I'm a dummy, so I'm still trying to understand when do I use fine-tuning or when do I use RAG? Really quickly, I think a crazy stat is that more than 50% of you listening are not subscribed. If you can subscribe on YouTube, follow on Apple or Spotify podcasts, my commitment to you is that we'll continue to make this content better and better. And now on to today's episode. Pawel, thank you so much for being here again.
- PHPawel Huryn
Yeah, it is great to have, uh, to be here. Thank you, Aakash.
- AGAakash Gupta
So I wanna talk a little bit about AI PM. Why are AI PMs paid so much?
- PHPawel Huryn
That's a good question. [laughs] Yeah, they are definitely paid more than an average product manager across different industries and regardless of their experience. Uh, one of the reasons is that, uh, they need to combine business skills with the technical knowledge. So even though you do not have to, uh, like, just like with technical product managers, you do not need to code, and in this case, you will not fine-tune or create, uh, AI workflows, uh, but you need to understand the technology good enough, well enough to work with engineers, and that might be challenging. This is a new technology.
- AGAakash Gupta
Yeah, I think that's one thing, and then the other thing is it's just such a hot area, right? So all of the PMs wanna go into it, and I think that as a result, what we're seeing is that the best companies, they're trying to pick up the best talent. And to pick up the best talent, you might need to pay a little more. So today, I wanna break down for everybody how you can become an AI PM. And I wanna start at the very basics of AI PM-ing. You need to be able to prompt well. So can you break this down for us?
- PHPawel Huryn
Yeah, of course. You want me to demonstrate a prompt, share my screen?
- AGAakash Gupta
Yes, I would love to see how you prompt.
- PHPawel Huryn
So an example, an example prompt from my article where I describe the best practices is like this. So first, this is a prompt about identifying hi- hidden assumptions for, uh, the product trio which performs continuous product discovery. And the first thing we want to do is to explain the context. So just like when we are working with engineers or product teams and, uh, we want to communicate the context, so not only what is the task they are supposed to perform, but also, uh, why it matters and, uh, yeah, how it aligns with the broader organizational context. So in case of working with, uh, LLMs or AI in general, we also want to explain this context. So, uh, I informed ChatGPT that it is working in a product trio performing continuous product discovery. I explained what the goal is, what are the team objectives. Uh, the next thing, uh, that is in this prompt are, uh, identified opportunities related to that objective that are might be the result of interviewing customers. And only after providing all that context, we ask about assumptions that needs to be true for those ideas to work. Uh, so we have, sorry, we have idea, uh, that one of the ideas might be offering automated investment recommendations. Uh, and we instruct ChatGPT to think separately, uh, from different perspectives. So this is an perspective of a product manager, experienced product designer, and experienced software engineer. And we would like for each of those personas to identify assumptions related to value, usability, viability, and feasibility. Uh, and what we have done here is, so the first element is introducing the context, uh, and describing the goal, uh, the broader objectives, uh, what happened before performing this task. So we have identified some opportunities. We have came up with an, an idea. And then, uh, we instruct the, we explain the steps that LLM needs to take in order to perform this task well. So we have, um, iterating for different roles, and for each of those roles, there is a specific task to perform. Uh, let's try what and see what will be the result of the section. Mm. Okay, so we have, we have different assumptions related to the four areas that we mentioned. For product manager, product designer, and engineer, those will not be the same risk areas because each of those persons brings a different perspective to the table. Uh, and when being so specific and defining the steps that LLM needs to take, um, also providing the broader context, we can get much more reliable, much better results. Uh, in the post About, uh, top, uh, high ROI use cases for product managers about the prompting, I explain several, uh, hacks. So one of the hacks is to ask AI to play a role, like a product manager, product designer, or software engineer. Uh, the next hack, the next important thing is to clarify the context, so not only what is import-- what is required, but also why we need it. Uh, another one is, uh, talking like we talk to, to humans, even though LLMs do not have emotions. Uh, you get better results if you, uh, treat them well.
- AGAakash Gupta
[laughs]
- PHPawel Huryn
Maybe. [laughs]
- AGAakash Gupta
Yeah, I feel like it's like a one percent difference, but just being polite, even including a smiley face, helps.
- PHPawel Huryn
Yeah. Uh, another one is to set clear expectations, so it's just like when we delegate a task to a human. Uh, it's way better [laughs] when we specify what are the des- what are the desired outcomes and how the response should be formatted. For example, what are the success criteria? Uh, like those four risk areas for each of the persons that we, uh, selected. Uh, in other situations, so for example, if the prompt was about generating a user story, uh, it's very helpful to provide a template. So not just inform LLM that we want a user story, but if we have some user stories that we used in the past, we generated in the past, give them as an example so that, uh, the answers can be better aligned. Another one is providing step-by-step instructions. So for example, first iterate for this and this and this role, then, uh, identify assumptions related to first value, second usability, and so on. Uh, in reasoning models, so here, uh, this is one-shot prompting, so it doesn't actually do it, uh, sequentially. Uh, but in reasoning models, it will actually iterate on those, on those steps. Uh, another one is avoiding leading questions to avoid injecting biases. I have tested several times, and I get better results when I, uh, say AI that they will get a reward. So for example, if I have a prompt that, uh, I, I, I cannot get the s-satisfying answer, I like adding that you will get $1,000 if you perform this task on a champion level or something like that. And it, for some reason, it works, and definitely, LLMs tend to provide longer answers, more detailed answers if you, uh, mention the reward. Um, yeah, and, uh, the last two is, uh, just to iterate. So, uh, there is high chance that, um, if you try it the first time, um, the output might not be ideal. So in that case, we just iterate, try to, uh, improve, and, uh, inspect the outcomes. And in some cases, it also helps if you provide an example of the ideal output, like a user story, let's say, or a product requirement document, and you can ask, uh, LLM to reverse engineering. So what, uh, what should be the prompt, assuming this is the outcome? Uh, and that's a [laughs] very powerful technique.
- AGAakash Gupta
Okay. So that's skill one for AI product managers. You need to be able to prompt well. You can't just prompt like an average person, right? You need to be prompting at a very high level. The way I would think about prompting is try to think about it like chess, right? Hey, let me take a quick break to talk about something that's completely changed my product management workflow: Linear. As a PM, I was drowning in tools, one for planning, another for issue tracking, roadmaps and sheets, and jumping between Slack, Intercom, and app reviews just to piece together customer feedback. Sound familiar? I was spending more time keeping systems in sync than actually building product. Every time development kicked off, my carefully crafted plans would immediately need updating. I was the human API between all our teams, constantly chasing updates and translating between tools. That's why I love Linear. I can capture customer feedback, shape product ideas collaboratively, quarterback cross-functional teams, and monitor development progress in one place. It cuts through the maze of disconnected systems that were complicating my life. Product teams at OpenAI, Vercel, Brex, and Cash App all use Linear. If you're tired of spending your days keeping different tools and teams in sync, check out Linear at linear.app/partners/aakash. That's linear.app/partners/aakash. Today's episode is brought to you by Miro. Let me ask you something. How many tools are you juggling just to get a single project across the finish line? One for brainstorming, another for planning, something else for tracking tickets. That's where Miro comes in. It becomes an all-in-one collaboration workspace. Whether you're consolidating user research from several interviews, developing and synthesizing product briefs or a wireframe, or project managing development, Miro brings everyone into the same space. It's fast, intuitive, and fully loaded with features like project templates, two-way Jira sync, and integration with software like draw.io and PlantUML. Miro's AI features can be used to synthesize elements in a board to develop a ready-to-review product requirements document in seconds. If you're tired of tab overload and scattered workflows, try Miro. Head to miro.com and see why over ninety million users choose Miro to guide from idea to outcome. You can get pretty good at chess if you put in a couple hundred hours, but you're not gonna suddenly be Magnus Carlsen until you put in ten, twenty, thirty thousand hours. And so Pawel is sharing some of his hard-won tips and tricks after putting in those hundreds and thousands of hours. I wanna move to the next area for AI PMs.
- AGAakash Gupta
Which I think this is the one where it's now veering really into the PM skills, which is an AI PRD. How do you write a PRD in an AI context?
- PHPawel Huryn
For this one, we partner, we partnered with Mikdat Jaffer for, which is product lead at OpenAI. And so arguably PRD for, uh, AI features or AI products is not something completely unique. Uh, what is unique is a lot of hype around artificial intelligence, and this hype causes that, uh, in some cases, uh, product teams pursue features or products without, uh, uh, a justified business case. So what is included? There are two areas that are included in, in this PRD. One is ensuring that our initiative is aligned with our business strategy, uh, in case of AI features with product team objectives. And the second one is including AI-specific considerations, like how we will, uh, ensure that there are, there are specific, uh, guardrails implemented so that the model is aligned with the user. And, uh, there are several sections. Uh, this post is still free and can be downloaded from, from my newsletter or together with the template. Uh, but with, before we discuss the template, maybe it's worth mentioning that the first, uh, AI PRD, uh, is a tool for alignment, s- uh, building alignment, uh, in the organization. Uh, not... We do not necessarily need, and I actually don't like it, [chuckles] documenting every single detail like user stories, tasks, uh, very detailed deadlines, uh, and, uh, roadmaps. Uh, we would rather, uh, want to use PRD as a tool to, uh, highlight our assumptions and, um, yeah, uh, provide some evidence and also connect our initiative to, to the broader, uh, organizational context. And the second thing is that, uh, it shouldn't be a distinct phase at the beginning of the projec- product or project or initiative. Uh, we usually start with a draft, and as we build our feature or AI-powered product, uh, we iterate on, on our PRD. Uh, and the sections in the PRD, so the first one is executive summary. So just briefly, we briefly summarize what this initiative is about and how it will, uh, what are the success criteria, how it will benefit the organization. The second one is about market opportunity, and this is about AI specific, because we would like to explain why this is something that we should build right now, and maybe this is something that has just become possible. And also, what is the, uh, potential for the future? So is this market big enough, or will it become big enough, uh, in the mid or long term? Uh, another one is strategic alignment, and this is also inspired by this, uh, AI hype. So we want to ensure that for an AI product, it is aligned with our vision, our strategy, it supports company objectives. And for AI feature, uh, there is additional assumptions related to our team objective.
- AGAakash Gupta
Yeah, 'cause you don't wanna just build AI for AI's sake, even if your board or your investors are asking you. [laughs]
- PHPawel Huryn
Uh, customer and user needs, so this is quite straightforward. So what is, what is the problem we are trying to solve? And, uh, let's hope there is some problem and, uh, and the only problem because building, uh, an AI-powered product cannot be the problem in itself. So, uh, we want to identify market segments, um, which are clusters of customers with similar underserved needs. And for each of those segments, we would like to, uh, understand how important those needs are for the customers and how satisfied they are with what they already have. The next one is our value proposition, and the value proposition is about how we will address those, uh, needs for each of the segments. So, um, what is important here is that we do not want to focus only on features, but also we want to mention capabilities, um, and benefits. So what is the current state? What is the customer pain? Um, how we will address it. So, for example, we will introduce a, a specific feature, uh, and what will happen after. So what, what are the benefits for, for the customers, and also how it is different from what our competitors offer, and how we can communicate it. And for this, there is a value proposition template, also value curve, so that we can easily compare, uh, our value proposition to what other companies offer. Uh, the next section, competitive advantage, is about, uh, not just competitive advantage right now, but how we can sustain competitive advantage in the long term. So, um, what can we do so our competitors can't or won't copy our strategy? And this is a classical, uh, can't/won't test from Roger Martin, uh, work from his, uh-
- AGAakash Gupta
Yes.
- PHPawel Huryn
... work.
- AGAakash Gupta
Whose podcast I believe will be right before yours. [chuckles]
- PHPawel Huryn
Oh. [laughs] Yeah. That's nice.
- AGAakash Gupta
Yeah.
- PHPawel Huryn
And, uh, here, uh, yeah, uh, I'm, I'm a big fan, so I will [laughs] uh, watch it for sure. Okay. And, uh, product scope, so high-level assumptions, use cases, links to Figma prototypes, non-functional requirements, so general requirements and how we will measure them, and also AI specifics, so what are the key architectural choices, um, what are the-- how can we, um, assess, uh, our implementation. So for example, AI evaluation metrics or bias and fairness audits.
Episode duration: 1:36:30
Install uListen for AI-powered chat & search across the full episode — Get Full Transcript
Transcript of episode IfW1FMDkw4k
