AI and product management | Marily Nika (Meta, Google)

1. 0:00 – 3:20

   Marily’s background

   1. MNMarily Nika0:00

      There is something called the shiny object trap.

   2. NANarrator0:02

      Mm-hmm.

   3. MNMarily Nika0:02

      And I'm always telling people, "Hey, don't do AI for the sake of doing AI. Make sure there is a problem there, make sure there is a pain point that needs to be solved in a smart way. Once you have identified what that problem is and what that very, very high level solution is, then reach out and try to figure out how to actually implement it."

   4. LRLenny Rachitsky0:26

      (instrumental music) Welcome to Lenny's Podcast, where I interview world-class product leaders and growth experts to learn from their hard-won experiences building and scaling today's most successful companies. Today, my guest is Marilyne Nika. Marilyne teaches the most popular course on Maven, on AI and product management. She's currently product lead at Meta, focusing on metaverse, avatars, and identity. Prior to Meta, she was at Google for over eight years, working on Google Glass, computer vision, and machine learning around speech recognition. In our conversation, we touch on what PMs should be paying attention to when it comes to what's happening in AI. We talk about a bunch of resources that'll help you get started in the world of AI, how AI tools available today can already help you do your job better as a PM. We also get relatively technical into what exactly is a model, how are models trained, all kinds of fun stuff like that. Enjoy this conversation with Marilyne Nika after a short word from our wonderful sponsors. This episode is brought to you by Amplitude. If you're setting up your analytics stack but not using Amplitude, what are you doing? Anyone can sell you analytics, while Amplitude unlocks the power of your product and guides you every step of the way. Get the right data, ask the right questions, get the right answers, and make growth happen. To get started with Amplitude for free, visit amplitude.com. Amplitude, power to your products. This episode is brought to you by Eppo. Eppo is a next-generation A/B testing platform built by Airbnb alums for modern growth teams. Companies like Netlify, Contentful, and Cameo rely on Eppo to power their experiments. Wherever you work, running experiments is increasingly essential, but there are no commercial tools that integrate with a modern growth team stack. This leads to wasted time building internal tools or trying to run your experiments through a clunky marketing tool. When I was at Airbnb, one of the things that I loved about our experimentation platform was being able to easily slice results by device, by country, and by user stage. Eppo does all that and more, delivering results quickly, avoiding annoying prolonged analytic cycles, and helping you easily get to the root cause of any issue you discover. Eppo lets you go beyond basic click-through metrics and instead use your North Star metrics like activation, retention, subscriptions, and payments. And Eppo supports tests on the front end, the back end, email marketing, and even machine learning clients. Check out Eppo at geteppo.com, get, E-P-P-O.com, and 10X your experiment velocity. Marilyne, welcome to the podcast.

   5. MNMarily Nika3:08

      Thank you. Hello. Thank you for having me.

   6. LRLenny Rachitsky3:11

      It's very much my pleasure. We've interacted a little bit on Twitter. We've never actually talked before, just right now. I've seen your course just kind of all

2. 3:20 – 4:46

   How Marily stays informed about the latest developments in AI

   1. LRLenny Rachitsky3:20

      over the place, your course on AI and PM. And so I just thought it'd be really fun to have you on and help us all understand what the hell is happening in AI and especially AI and product. So thanks again for being here.

   2. MNMarily Nika3:34

      Yes, thank you. I'm really excited.

   3. LRLenny Rachitsky3:36

      I would love your help as a former full-time PM/everyone listening that is a current PM to help us understand what is going on i- with AI and product. Tech in general and tools in general move really fast. You know, if you're trying to pay attention to, like, what's happening, it's really hard to stay up to date on where things are going. And it feels especially hard in AI. It feels like there's just something coming out every day. And so I have a bunch of questions along these lines. The first is just, like, what media do you pay attention to to stay on top of what's happening and what's new and what's interesting in the world of AI and machine learning?

   4. MNMarily Nika4:11

      As you know very well, subscribing to newsletters is something that's really, really impactful. And of course, I sub-subscribed to your newsletter. But I am a big, big, big fan of The Download by MIT Technology Review, or TL;DR. And they're not necessarily AI centric, but what I'm advocating for and what I'm telling people is that in the future, everything will be AI by default. So even if you have something that's technology focused, you will see a lot of AI starting to get sprinkled in there.

   5. LRLenny Rachitsky4:40

      I want to follow up on what you just said there, but maybe we'll save it a little bit. Maybe going in different direction first, what do you think

3. 4:46 – 5:59

   What is overhyped and underhyped in AI right now

   1. LRLenny Rachitsky4:46

      is overhyped in the space of AI right now? What do you think is underhyped and undervalued?

   2. MNMarily Nika4:52

      I would like to discuss ChatGPT, which is both underhyped and overhyped at the same time. I was reading this article this morning where there are writers complaining, and they're very, very fearful, and they think, "Oh, writing online is gonna die. Everything we've been striving for is gonna be replaced. They're gonna take our jobs," and so on. And I'm just like, "No, no, no, no. ChatGPT and technology is enhancing our work. It's enhancing us. It does not steal from us." So that's what comes across right now. And then there are other things that are underhype. Like, obviously ChatGPT is amazing. I'm using it day to day. But there are other things AI can do in an amazing manner. Like, I was reading a research article the other day that said that AI can now detect lies. So lie detection, whether it is for security reasons, or at work, or anything like that, is now possible. So I encourage people to go through these newsletters and go through these online blogs, 10Crunch and so on, and just read what's happening. It's not all about ChatGPT. There is more. There is more about AI that you should read about.

4. 5:59 – 8:25

   How Marily uses ChatGPT for work

   1. MNMarily Nika5:59

   2. LRLenny Rachitsky5:59

      You mentioned that you use ChatGPT in your work life.Talk about that. What are you actually using it for?

   3. MNMarily Nika6:04

      Even when I'm at work, and I am trying to come up with a nice mission statement, right? When we're PMs, we come up with mission statements. It's just crucial part, and it's where the core begins. You wanna get people excited. You wanna get people inspired. There is nothing I can write that's gonna be as good as what ChatGPT would write. So what I do is I literally go to ChatGPT, and I say, "Rewrite this mission statement for me." And it just... The... Even first try, it produces something which is fantastic. So that. Number two, it helps me create user segments in a fantastic way. It will think of user segments that your mind wouldn't even go there. Like, it just wouldn't go there. And it will provide the motivations, it will provide the pain points, and you just come up with ideas as you s- as you breathe in. And then the last thing that it does is it provides ideas for you that are AI-enhanced. So, I just use it day-to-day, even improves my day-to-day workflow, but I'm not making it do my job for me. I'm asking it after I have already had a vision in my head and what it is I want to do.

   4. LRLenny Rachitsky7:10

      So is the way you're approaching it is you just put in, "Come up with a better mission statement than," and then you give it your version of the mission statement.

   5. MNMarily Nika7:19

      Exactly.

   6. LRLenny Rachitsky7:20

      Interesting. And you're saying that that comes up with a better mission statement than the one you had?

   7. MNMarily Nika7:25

      It's better because the mission statement is gonna be read by all disciplines. It's not just gonna be read by PMs that already have a lot of context and understand. It's gonna be read by leadership, by junior people, by stakeholders, by other departments, by competitors. And you need it to be on point and in the words that they're meant to be understood by everyone, even a kid could understand it. And they, they would, they get inspired by it as well.

   8. LRLenny Rachitsky7:49

      And then you also said to use it for personas. How do you actually frame that prompt with ChatGPT?

   9. MNMarily Nika7:55

      Let's say you're working for a specific product area, and you know you want to create some fitness band. So you would say something like, "Who would be interested in a fitness band that doesn't have a screen?" And it will provide a bulleted list of people. Like, hey, young professionals, if they're interested but don't have enough time, people that do not want to charge their wearables every day, then the list goes on. It's just fantastic.

   10. LRLenny Rachitsky8:24

       You were talking about

5. 8:25 – 11:16

   Why product managers will be AI product managers in the future

   1. LRLenny Rachitsky8:25

      how you think the future of AI is it's the default. And is what you mean there that it's basically baked into every product we use, and it helps the user do better things, it helps the product work better? Is that what you mean? Or is it something else?

   2. MNMarily Nika8:39

      I believe that all product managers will be AI product managers in the future.

   3. LRLenny Rachitsky8:43

      Mm-hmm.

   4. MNMarily Nika8:43

      And this is because we see all products needing to have personalized experience, a recommender system that is actually good. I mean, you cannot watch Netflix, you cannot even watch a movie without needing that. After you, you watch White Lotus or, like, Stranger Things, you will want something similar to watch. You're not gonna want, like, a romantic thing to be suggested or recommended to you, right? Also, automation is another thing. We need to keep improving as society. We need to keep making technological advancements. You're not gonna be able to do that if you don't have an AI-centric view in every sector that you're working on.

   5. LRLenny Rachitsky9:20

      When you say that every PM will be an AI PM, is your thinking that you'll be using AI tools in your job as a PM, or that you'll be building AI into everything you're building? Uh, how do you think about that?

   6. MNMarily Nika9:34

      I think it's that you will need to get comfortable with having a partner that's an research scientist.

   7. LRLenny Rachitsky9:42

      Hm.

   8. MNMarily Nika9:42

      And you will need to understand that these people can produce a smart model. They'll be able to do some automations, some personalizations, some recommendations, uh. And a lot of people feel uncomfortable with that. A lot of people don't know how to approach the researchers. A lot of people don't like the uncertainty that research has. A lot of PMs are very, very used to, "Okay, I'm gonna do this, I'm gonna lunch, I'm gonna do this, I'm gonna lunch." Whereas when you're working with research, it's more like, "We're gonna try this, and then in a year, if it doesn't work out, we're gonna shut everything down and pick up completely two weeks from now." So, I feel that if people get more used to uncertainty and research, things are gonna be good in the end for them.

   9. LRLenny Rachitsky10:28

      I thought you were comparing ChatGPT as, like, a researcher you're working with, but you're actually saying people will have PhD researchers on their teams, helping them build models into their product to make their product better. Is that, is that what you're saying?

   10. MNMarily Nika10:43

       Correct. This is exactly what I mean.

   11. LRLenny Rachitsky10:45

       Hm. Interesting.

   12. MNMarily Nika10:46

       And, um, and from a product perspective, like, I'm imagining, like, three bubbles in my head.

   13. LRLenny Rachitsky10:50

       Yeah.

   14. MNMarily Nika10:50

       So you want to find the intersection of something that's desirable by users, something that is gonna be a viable business, and something that is gonna be feasible from a research scientist and technical perspective. And then when you have that, it's just gonna be a fantastic product, fantastic launch that you came up with. So yeah, whenever I say researcher, I mean research scientist that can produce an AI machine learning model.

   15. LRLenny Rachitsky11:15

       Wow,

6. 11:16 – 14:12

   How to get started using AI

   1. LRLenny Rachitsky11:16

      didn't think about how every cross-functional team might end up with a research scientist. Interesting, interesting. For PMs who are curious about learning how to do this stuff, what are a couple things that PMs today, who are, have no experience with AI, what can they do to start learning how to build AI tooling into their products, understand what the hell's happening in the space of AI?

   2. MNMarily Nika11:40

      This is a good question, and I guess the, the message that I want to pass is you shouldn't be overwhelmed by these technologies if you don't have a technical background, because you can learn these things. And as a PM, you will never need to actually train a code. Also, even if you want to train, there are no-code approaches for training models. But to answer your question, if you're working on any product, you can always sprinkle in a smarter feature so you can make it more secure, you can personalize it.You can enhance it with, like, fraud detection. You can make it more ethical. If it's healthcare, you can make it faster, you can make it more accurate. If it's shopping, you can create better recommendations. Basically, anything where you can get data behind the behavior of your users can be improved with AI. So I guess it's all about changing the mindset of PMs, taking a step back, and just thinking about, "Okay, I have all this data that's just lying and sitting around. What is it that I can do with it?" I've been meeting PMs that said, "Oh, we don't have any ... We're not collecting any data. W- we don't have any dashboards." So even that is a huge first step towards AI. And then just start thinking about it, what you could do. Just hire, uh, get a data science intern and just see what they, they are gonna do. There's, there's just so much people can do.

   3. LRLenny Rachitsky13:00

      So say you want to start investing in some sort of model, some sort of AI within your team. You're saying maybe hire data scientists who can help you start to build something that you can start integrating. Is that your advice on the first step of once you start, you wanna start getting serious about building some sort of AI component?

   4. MNMarily Nika13:19

      There is something called the shiny object trap. And I'm always telling people, "Hey, don't do AI for the sake of doing AI." Make sure there is a problem there, make sure there is a pain point that needs to be solved in a smart way. Once you have identified what that problem is and what that very, very high level solution is, then reach out and try to figure out how to actually implement it. And there's a definition I like hearing. I usually say that a generalist PM helps their team and their company build and ship the right product. But the AI PM helps their team and company solve the right problem. So if you wanna get into AI PM, figure out what the problem is that you will get the data scientists to create a model for solving. But there needs to be a problem. There needs to be audience. There needs to be a user and a pain point for it.

   5. LRLenny Rachitsky14:11

      What are

7. 14:12 – 15:47

   When not to use AI

   1. LRLenny Rachitsky14:12

      signs that AI may not be a good approach to solving a problem? You said that, you know ... And this happened on a lot of my teams. "Oh, we were gonna build a really cool model that's gonna do something really smart in this case." And it often ended up being very low ROI investment, and took like six months to a year before you even knew what the hell (laughs) was, was happening.

   2. MNMarily Nika14:31

      Yeah.

   3. LRLenny Rachitsky14:32

      Do you have any thoughts on signs that maybe this isn't a place you should be putting a lot of time into AI, versus like, "This is definitely an opportunity. Yes, we should invest a lot of time into this."

   4. MNMarily Nika14:43

      Don't do it for your MVP. It makes zero sense. Do not waste time of data scientists that can train models with ... using powerful machines that are going to take weeks to train. This is because if you have an MVP and you just wanna get buy-in for an idea or feature that may use AI in the future, fake it. Create a little Figma prototype and just show it to some users, and just fake what the AI is going to be doing. So I have a lot of young early-stage entrepreneurs reach out to me and they say, "Oh, how would ... Should we train this model to do this and that? 'Cause we want to prove that there is a market." No. Do not use AI then. You should use AI where you think you already have some data, or data from an adjacent product that you feel you can leverage for your own product to create something that's meaningful. Recommendation automation, like we talked about. But not for an MVP. Please, people. This is, this is my advice.

8. 15:47 – 17:01

   How much data do you need for AI to work properly?

   1. LRLenny Rachitsky15:47

      How much data do you think you need for AI, ML, to have a chance to contribute? Do you have like a heuristic of, "If you have anything less than this, it's not gonna work at all"?

   2. MNMarily Nika15:59

      This is a good question, and it honestly depends on what you're trying to do. If you're trying to classify if the photo is a cat or a dog, obviously, even if you have, I don't know, like 15, 20 labeled photos, that's gonna work. But if you wanna create voice recognizers or complicated NLP applications, you're gonna need thousands and thousands of data. And, and this is what's making this not be easy, right? AI systems are not easy to develop. There is a life cycle of a machine learning, um, project. And after scoping, you need to figure out, "Oh God, how much data do I need? Where do I find this data as well?" Right? How much data? Sometimes I've seen people synthesizing their own fake data just so that they can have something to train with and test their models. But the exact amount is hard to be quantified, especially from a PM. Like I'm sure data scientists have a, a different opinion.

   3. LRLenny Rachitsky16:53

      Yeah, my guess is most startups are gonna have nowhere near enough data to build their own model and make it something really interesting. So do you

9. 17:01 – 18:35

   When should companies develop their own AI tools?

   1. LRLenny Rachitsky17:01

      have a thought on when it makes sense to try to build your own model, try to train your own GPT type thing, uh, versus use something that's already out there, like say GPT or Midjourney, or all those guys?

   2. MNMarily Nika17:17

      If you are a big tech company and you're offering a service that is gonna do, like speech recognition or that is gonna have like their own ChatGPT, you want to use more data and more diverse data to train and retrain and retrain. Because if you don't, then your quality is gonna be the same as every other company's. There are agencies that are selling data, packages of data that are ready so that you can get them and train your models. But the question is if everyone takes that exact dataset, then the quality that every single company is producing is gonna be the exact same. So you do wanna diversify, you do wanna collect your own data. And I guess a good question from a PM perspective is when is the quality of your product good enough to launch? And that is like a really interesting point because-It's totally your responsibility as a PM to decide, okay, the recognition of whether this photo is a cat or a dog is good enough for the users. It's like 70% accurate, 80% accurate. Where is the bar? Where do we want? And that's why I like the AIPM role is so cool, because you have problems like that to solve that no one else has kind of tackled before. So it's all on you.

   3. LRLenny Rachitsky18:34

      We've thrown around these

10. 18:35 – 21:25

    What an AI model is and how it is trained

    1. LRLenny Rachitsky18:35

       words model, and we talk about training models. Do you have a good succinct kind of explanation for what a model is for folks that haven't ... you know, that aren't that technical? And then just the general idea of training a model. Like, what is a simple way to think of here's what a model is?

    2. MNMarily Nika18:52

       So I have a three-year-old girl, and I'm teaching her about life and everything. So I was recently teaching her about animals. And, you know, you explain things to her once or twice, like what a mammoth is or an, a rhino and so on. But you will end up training your kid's brain by repeating the same information again. So you will say, "Hey, here's what a rhino looks like. Here's what an elephant looks like. Here's what a rhino looks like. Here's what an elephant looks like." And once you've done this enough times, then your kid will see an animal on the street, and they'll be able to recognize and say, "Oh, yeah. That's like the rhino we were talking about." This is exactly what a model is. A model is like a kid's brain. It has the ability to take an input, which means it has the ability to take an image and say, "Oh, I recognize what this is. That looks like a rhino, but I'm 70% sure about this." So it will output the probability as well of this identity.

    3. LRLenny Rachitsky19:51

       And you said image, but it could be text for, say, ChatGPT in the future. I imagine video. There's also voice, like Whisper. That's an awesome explanation. Basically, it's ... try to recreate the human brain, is a, is a nice way of thinking about it. And then training a model. Can you talk about what that means?

    4. MNMarily Nika20:08

       The purpose of training a model, for example, is providing a lot of images that are labeled-

    5. LRLenny Rachitsky20:13

       Mm-hmm.

    6. MNMarily Nika20:13

       ... and say, "Here's what a cat looks like. Here's what a dog looks like." And we're talking about thousands and thousands of data, data sets for this. And once you do this, there's a process where the model is just processing this information and it's learning. It's finding patterns through it. And the patterns are not in the form of, oh, if this is gray, then this means this. No. It just learns in a smart way how to identify specific things that we don't even understand. And then it's able to output the, the probability of whether a photo is gonna contain a cat or a dog.

    7. LRLenny Rachitsky20:49

       Just conceptually, what is the output of the training? Is it code that is autogenerated with these decision trees and weights and things like that? Is it a database of wei- like just conceptually, what is the output of a training that becomes a model? What's the simplest way to think about that?

    8. MNMarily Nika21:05

       So let's imagine speech. Speech is a great example. For example, I'm talking to a device with, which is like a home assistant, and I say, "Hey, what is the weather like today?" This is gonna take my voice and audio and it's gonna process it, and the output is gonna be a transcription. So it's literally gonna be text that corresponds to what I said to it.

    9. LRLenny Rachitsky21:24

       Thinking about

11. 21:25 – 23:02

    How Google demonstrated the ability of AI to translate a conversation in real time

    1. LRLenny Rachitsky21:25

       the stuff you've worked on at Google, at Meta, anywhere else you've worked on site projects even, what are some of the cooler applications of AI machine learning that you've worked on, contributed to, or even seen that, that you can talk about? I imagine there's a lot of sensitive stuff too going on.

    2. MNMarily Nika21:41

       One thing I want to talk about is the team I used to work for, for Google, which was the ARVR team, and they were working on an AR glass. And actually, they, they had a video on last year's Google I/O. They were able to have the Google Glass on someone that spoke one language, and then this other person will stand in front of them and spoke a different language. And the glass would take as an input the audio that came from that other person, and it would transcribe it, it would translate it, and show it on the screen for that person in their language. So we're talking about the ability for these devices to unlock the borders of communication. And that is not science fiction. This is, what, amazing and mind-blowing. There's no science fiction anymore. These things are real. The technology is here. It's just a matter of connecting the pieces to the puzzle in order to see them coming to life. So I think that one was the most, one of the most impactful things I've ever seen.

    3. LRLenny Rachitsky22:46

       I remember that demo.

    4. MNMarily Nika22:47

       Yeah.

    5. LRLenny Rachitsky22:47

       It was pretty incredible. Okay, so thinking a little more broadly, do you think ChatGPT or just, say, GPT-4 or JA- GPT-5, GPT-6, do you think at some point this will replace product managers? Something

12. 23:02 – 23:48

    Why AI will not replace PMs

    1. LRLenny Rachitsky23:02

       I see on Twitter a lot. People are like, "Oh, my God. Product management's dead. This thing made my product requirements document for me," or you talked about how it makes your mission statement better. Do you think there's a place where PMs aren't necessary anymore?

    2. MNMarily Nika23:14

       Oh, absolutely not. As I said, like it makes everything better. If anything, it's gonna free up time for me to, to do other things that are less tedious. For example, I am running so many projects and they all need their

    3. LRLenny Rachitsky23:26

       Yeah.

    4. MNMarily Nika23:27

       PRD, and the PRDs have all these areas that are common across, across all of them. If I had a system that can actually write the tedious stuff for me, so like I can focus on a more strategic side of things, that would be incredible. It will make us smarter. If anything, it will unlock new areas of product management that we haven't realized that, that are there.

13. 23:48 – 26:21

    A case for learning to code

    1. MNMarily Nika23:48

    2. LRLenny Rachitsky23:48

       Are there areas do you think ... With your kind of vision of all PMs will be AIPMs, are there areas that you think PMs should invest more skill-wise? Or areas they should less focus on invest, because, say, some machine learning model's gonna do that for them?

    3. MNMarily Nika24:05

       I'd like to see people being less overwhelmed, less intimidated, less afraid to start learning how to code, how to train a little model on their own. This is because even if, you know, ChatGPT or these no-code applications may be able to do this for us...It gives you a different approach, a different mindset, a different, if you want, confidence to know how things work. And here's a silly example. I was learning how to play the piano when I was young, and when my teacher came in, I was like, "Oh, I wanna learn how to play this cool song." There were some songs that I really liked. And she said, "No, you need to start with, uh, classical music," and I just hated it at the time. And I said, "Why do I have to do this?" 'Cause she said, "If you learn the fundamentals and how, you know, where things started and the meaning of music, it's gonna help you along the way to create music of your own, if you want to." And she was right. Like, I, I just loved it. So it's the same with coding. I encourage people to just take an online course, understand more, get your hands dirty, pair up with someone else that's in the same boat as you, because this is gonna give you the skillset to understand how that tool that's gonna help you<|a|><|agent|><|nolang|>

    4. LRLenny Rachitsky24:54

       It gives you a different approach, a different mindset, a different, if you want confidence to know how things work. And here's a silly example. I was learning how to play the piano when I was young, and when my teacher came in, I was like, "Oh, I wanna learn how to play this cool song," there were some songs that I really liked. And she said, "No, you need to start with, uh, classical music," and I just hated it at the time. And I said, "Why do I have to do this?" 'Cause she said, "If you learn the fundamentals and how, you know, where things started and the beginning of music, it's gonna help you along the way to create music on your own if you want to." And she went right. Like, I, I just loved it. So it's the same with coding. I encourage people to just take an online course, understand more, get your hands dirty, pair up with someone else that's in the same boat as you, because this is gonna give you the skillset to understand how that tool that's gonna help you in your day-to-day was even created in the first place. Instead of blindfolded, just trusted to do your job. This episode is brought to you by Pando, the always-on employee performance platform. How much do you love the performance review process? Mm, yeah. It's time-consuming, subjective, biased, and there's rarely any transparency. With the rapid shift to distributed work, it's a struggle to create the structure and transparency that you want to help your employees have the highest impact and growth in their careers. Pando is disrupting the old paradigm of performance management, including a continuous employee-centric approach so employees stay engaged, see their progression in real time, and know exactly when and how they can level up. With Pando, managers can leverage competency-based frameworks to effectively coach and develop their teams and align on consistent growth standards, resulting in higher quality feedback and higher performing teams. Visit pando.com/lenny for more info and get a special discount when you sign up and reference this podcast. That's pando.com/lenny.

14. 26:21 – 27:40

    Where to learn to code

    1. LRLenny Rachitsky26:21

       For someone that actually wants to do that and learn to code, which I love that advice, do you have any resources, places that you point people to for learning to code, getting started down that path?

    2. MNMarily Nika26:31

       It depends on what type of learner you are. There are some people that like to learn offline. So just go to Coursera. There's so many courses. There is an amazing one actually, Introduction to AI by Stanford, which I would encourage people to take a look at. But I know that a lot of people don't like, don't have the time, don't have the discipline to actually, you know, take time off or like after work, after they put their kids to sleep to just do it. So if you enjoy learning with others, if you enjoy being part of a team, if you enjoy going through a journey together, then I recommend these resources. So there is something called Career Foundry, which is a fantastic online coding school, General Assembly, and then Coding Dojo. I, I was, actually gave a talk, uh, ages ago at Coding Dojo about Python. And all it takes is just a few weeks of your time and passion and just for you to roll up your sleeves and just realize that this is not intimidating and realize the benefits you can get by learning how to code.

    3. LRLenny Rachitsky27:31

       Awesome. Thanks for sharing those. We'll include links in the show notes. Going back to a PM trying to become better in AI, if you think about a PM

15. 27:40 – 29:25

    How to become a strong AI PM

    1. LRLenny Rachitsky27:40

       that's kind of early in their career and wants to become a very strong AI PM, I know you have a whole course about this, which we can talk about now or later, whatever is easier. What should that PM be doing? We talked a bit about learn to code maybe, start playing with tools. What else do you suggest PMs that want to become really strong AI PMs do now and invest in?

    2. MNMarily Nika28:02

       So I do have a course that's coming out February 6th on Maven, which is for current and aspiring product managers that want to build AI products. But I also have offline recordings. I have the same course on an offline basis on my website. I'd be happy to, to talk to you if you're interested about this. What I feel people should understand is what it takes to manage an AI product. Of course, people are very familiar with the stages of product development in general, but AI product development is different. As I mentioned before, sometimes you're actually managing the problem and not the product, and you're trying to figure out if there is a problem that makes sense to be answered by a smart solution. So it's kind of a very interesting and more complicated process than regular product management. So number one, figure out how it differs from general product management. Number two, if you're already at the company that is actually having AI researchers and AI research scientists, I encourage people to just maybe chat to them and shadow them and spend an hour of their week just, just talking to them and experiencing what they're doing. This is going to open your mind. This is going to give you so much context as to what it is and, and the endless potential that you can identify there.

    3. LRLenny Rachitsky29:23

       Awesome. And

16. 29:25 – 31:16

    Challenges that AI PMs face

    1. LRLenny Rachitsky29:25

       is there anything else you wanna share from your course that you think might be interesting to folks?

    2. MNMarily Nika29:30

       So we talked about why it's awesome to be an AI PM, but I do want to call out that there are a few challenges that people need to be aware of. Number one, and I kind of mentioned it before, is the uncertainty. You may have been working on all these incredible research and ideas in hypothesis, but then when you actually train the model, the results you may be getting may not be optimal, may not be answering the questions or the hypothesis that you actually had in mind. So that's number one. You need to be able to encourage the team throughout this process because you're like the captain of the ship. You need to be the one that's kind of cheerleading the team, making sure everyone keeps going. Number two, you are gonna have to be like, "What do you want?" You are gonna have to change the direction, and managing this from a leadership perspective can be tricky and it can be challenging. Number three, we talked about data, but... getting good data is hard. Like you may need to be creative, figure out ways for data collection that you never thought you would do. You may get on the street and ask for people to actually contribute data for what it is you're doing. Like you need to be able to ... and willing to do everything. And the last thing is, from a career trajectory, usually product managers get ahead the more they launch. But if you're in, in a research org, you're not gonna launch as often. So you need to make sure to clarify with the hiring managers early on, "Hey, what does progress mean? How am I gonna get assessed in a research work which is different than what I've been doing so far?" So it's challenging, but I always encourage people to flex different muscles. And this is like the zero to one muscle that I think is, is just crucial when it comes to product management.

    3. LRLenny Rachitsky31:15

       This actually

17. 31:16 – 33:10

    Getting leadership on board with investing in AI

    1. LRLenny Rachitsky31:16

       is a great segue to a question I definitely wanted to ask, which is around getting buy-in for investment at a company for ML. So there's sometimes like all this energy for like a zero to one, "Let's just try something." Sometimes not, but that, maybe that's ... Maybe there's a two-part question here. Do you have any advice on just getting buy-in for, "We wanna try something with ML. It's gonna take us six months to figure out if it's worth the effort, but we think there's something here." And then sometimes there's like a lot of energy initially, and then you get some win, like your search ranking's smarter and it's great. But then maintaining that, having like all these really expensive people working on just tweaking this model and continuing to make it smarter and a little more efficient, often it's hard to continue to get buy-in for that sort of team. Do you have any advice on initial kind of buy-in, "Let's try something here," and then down the road, just like keeping a team going, trying to make this thing smarter and smarter?

    2. MNMarily Nika32:09

       People should know that there is an excellent source of inspiration and something that can kind of de-risk things, which is adjacent products. Maybe the company has already launched a product that has been successful that was AI first. And whenever I tried to convince leadership about something that I wanna do that's kind of a big bet, I always use examples and I'm like, "Hey, this seemed crazy at the time. Here's how it worked. What I'm proposing is very similar to this crazy thing." And then I propose a little contingency plan. Like, "Hey, if that doesn't work out, here's the rollback plan. Here's kind of the maximum impact it will have done in a negative way, which is not gonna be too much." And you kind of take it all on zero. And it's interesting because the more you work on this specific company, the more trust you get. And if the culture is such, then failing is gonna be welcomed. So I love companies that welcome failure because you can just go ahead and do this sort of thing.

    3. LRLenny Rachitsky33:09

       You tell me if I'm wrong, but

18. 33:10 – 35:29

    How PMs will work with data scientists and AI

    1. LRLenny Rachitsky33:10

       I feel like most investments in ML are not successes and often not great uses of time. I'm curious if that changes with more tooling and more kind of public models that people can plug into without having to build their own. I wonder if it becomes like, "Oh, okay. Look, we're, we'll put in three weeks, we'll get something really useful."

    2. MNMarily Nika33:27

       Exactly. And also, the other thing, and I wanted to add on the question you asked before about, "Hey, how do you keep updated about new niche tech?" We shouldn't underestimate academia and research blogs. And there's a website called arXiv where you can see new papers come out, because this is where ... I mean, ChatGPT and, and like used to be there for a long time. Like there, there was a lot of information on this sort of thing. But it's now recent where we see that research scientists and research orgs are kind of not as siloed as they used to be. So the more companies invest on staffing this layer between productionalizing and research, academic research, the more PMs you're gonna add there, and the more you're gonna see this bridge kind of creating good products that are created. So, sometimes you have amazing ideas by research scientists, but you need the PM to take it and actually figure out ways to also monetize it, right? That's the other thing. If you're a PM, you need to come up with ways to actually be able to monetize. And ChatGPT is now free for everyone, but I don't know if you, if you saw there was a, there was a sign-up forum that was kind of coming around saying, "Hey, would you pay for this? What would be the minimum you would pay? What would be the maximum you would pay? What would you like to see if you paid?" So having PMs bridge that gap is crucial for companies to be able to take the research and actually come up with meaningful use cases for users.

    3. LRLenny Rachitsky34:56

       I think they actually started charging the other day. I think it's like $40, $42 a, a month to start using it.

    4. MNMarily Nika35:02

       That's awesome.

    5. LRLenny Rachitsky35:02

       I think. People have been talking about it on Twitter. I don't know if that's live yet. And then you talked about research papers. When I think that, I always think of Tyler Cowen. He has this awesome blog, Marginal Revolution, and he's really good at sharing insights from research papers that he's reading. So, that's another place for folks to check out. He's just like this really smart dude. He's really excited about AI and ChatGPT in general, and so he shares a lot of really interesting insights about it all. Segueing a little

19. 35:29 – 39:12

    Marily’s AI course

    1. LRLenny Rachitsky35:29

       bit to your course, I have a couple questions about it. One is just like, can you just talk about like the broad framework of your course? Like how long is it? What do you learn? What are the workshops broadly? And then I have a couple follow-up questions.

    2. MNMarily Nika35:39

       My course is three weeks long. It's meant for people that are either aspiring or current PMs that want to understand how to sprinkle in AI solutions or they wanna become full-time AI PMs. Week one is more of an introduction what the product development life cycle is for regular products and how it differs for AI PMs specifically. And then we talk about idea creation. How on earth do you come up with ideas? And I love what Steve Jobs says, where he, he used to say, "Well, users don't know what they want until you show it to them." And that's exactly the mindset I want to imbue to people and say, "Hey, people don't know how on earth to use AI." People would never have imagined ChatGPT can do what it does. And then we take that and we dive deep and we talk about how on earth do you productionize something like this? What are the different partners you're working with? What is a research scientist and how on earth do you collaborate and how do you partner with them?How do you convince them of what you have in mind for their precious research to be converted into a product? How on earth do you convince them to trust you, and, and, how do you influence them? And then at the end, we're talking about how you actually will be able to pave your path with a PM, all the way from interviewing for this role, from what good resumes look like, and doing some mo- interviews, because the more you're practiced, the, the better it's gonna be.

    3. LRLenny Rachitsky37:03

       How many workshops are there through the course?

    4. MNMarily Nika37:05

       Nine workshops.

    5. LRLenny Rachitsky37:06

       Nine workshops? Okay. Of the nine workshops, which of them are you finding is the most exciting, game-changing for someone, most interesting?

    6. MNMarily Nika37:16

       So throughout the duration of all these workshops, people have homework, and they actually take home an exercise where they need to create and develop their own AI product end-to-end. And they can pair up with each other. By the way, there was this, this two student-spired app, and actually wherever the raise funding, which is mind-blowing to me, which is really, really great.

    7. LRLenny Rachitsky37:33

       That's awesome.

    8. MNMarily Nika37:34

       Um, but to continue, the most exciting part is when everyone at the very end are actually presenting their work, and they're actually asking questions and getting feedback, and they're just really excited and proud for what they've created.

    9. LRLenny Rachitsky37:49

       That's a good reminder of a lot of the learning that you do is just doing it, not just kind of reading about it and following Twitter. Can you share any examples of stuff people built after the course?

    10. MNMarily Nika37:57

        Someone was able to actually, and I kid you not, create a neural model that was able to take as an input x-rays that they found online, and was able to tell us what was wrong, if something was wrong with that patient. And it's just crazy to think that you can do that within three weeks. Obviously, it was just by photos we were able to crawl online for x-rays. But the concept is there, that you can build something like that, you can create it, and to take it a bit further, they wanted to create a little recommender system and say, "Hey, we think this is what's wrong with you. Here are the steps you should follow." Obviously, we're not trying to play doctors or to, to pretend that, you know, we're all medical in any way. But being able to see that actually functioning is just, it's very powerful.

    11. LRLenny Rachitsky38:48

        That's amazing. Did they already know how to code, th- this team that built this thing?

    12. MNMarily Nika38:51

        They did not. But part of the course is to teach people the basics that you are gonna need from a PM lens. And there are some no-code tools, as I mentioned, that are gonna allow you to drag and drop and train these models and input photos in it and be able to do it.

    13. LRLenny Rachitsky39:07

        Can you mention those tools again? 'Cause that is really interesting, and I know that was just like a peek at your course, but if someone

20. 39:12 – 40:31

    AutoML and how a renewable-energy company used it to improve its turbine maintenance procedure

    1. LRLenny Rachitsky39:12

       wanted to start building something like this-

    2. MNMarily Nika39:14

       Mm-hmm.

    3. LRLenny Rachitsky39:14

       ... what, what are some of these tools they could check out?

    4. MNMarily Nika39:17

       One of the tools I would like to recommend to people is actually AutoML. This is offered by Google Cloud, and essentially it allows you to train high-quality custom machine learning models with minimal effort. You don't need to be able to understand ........................ or anything like that. You need to have a lot of photos and images that you have already collected. Like, it's not gonna do the collection for you. And a great application I had to see, there's actually a YouTube video about this, is there was this company that actually had a lot of wind turbines. And what they did is, in order to maintain these, they would actually have people manually have huge ladders and go take a look and see if everything was okay. So eventually, they just got drones, and they had these drones fly on all of these machines and take photos and everything. And then they downloaded all these photos, and they uploaded on AutoML, and they were able to see which ones needed maintenance and which did not. And I think they reduced the time from like three weeks of work to like a few hours of knowing which need maintenance and just be able to send people there. So it's this type of thing that you can do on your own by applying these sort of tools.

    5. LRLenny Rachitsky40:26

       So, and that tool is called AutoML?

    6. MNMarily Nika40:28

       Yes, AutoML.

    7. LRLenny Rachitsky40:29

       Amazing. We'll link to that in the show notes. Coming

21. 40:31 – 42:53

    How Marily built her course and the modifications she has made

    1. LRLenny Rachitsky40:31

       back to your course, and maybe just a couple more questions.

    2. MNMarily Nika40:34

       Yeah.

    3. LRLenny Rachitsky40:34

       Can you just talk about what it takes to build a course like the course you built? Like, how much time did it take you? How much work did it take? Anything there you wanna share?

    4. MNMarily Nika40:43

       I treated creating my course like a product. Essentially what I did is I came up with some hypotheses as to who the audience was and as to what they were looking to get out of it. And I started reaching out to people, and I started saying, "Hey, first of all, would you like to learn from me? Second of all, what would you like to learn? What are the specific questions that you would need answered?" Because these are people that are working full-time, that have families, right? In order to take a break from all that, you need to provide something to them that is meaningful. And there were quite a few iterations. In the beginning, I was focusing the course more for software engineers that wanted to become AI product managers. But then I realized, no, there are a lot of PMs that want to become AI product managers. So I, I did a little, uh, mind shifting there. So what it takes is make sure you find the right audience. Make sure to figure out what that audience wants. Make sure to have the right duration. One week, I found it too short. Two weeks, it will still be rushed. Three weeks is excellent, because you give the opportunity to everyone to present and to get to know each other on like an offline Discord community, which is another important part. And then the last thing, you need to have a personal relationship with everyone. So I've messaged everyone. I've seen everyone's application. I met with some people as well, just to make sure to answer any questions and concerns, because I wanted to make sure that people were comfortable just trusting a stranger like me and paying them for, to provide knowledge for their course. So it was, it took quite a few iterations, but I was able to get there, and I'm very, very happy about it. And I recorded it offline as well for people.

    5. LRLenny Rachitsky42:27

       Has anything had to change in this course? Maybe this just as a last question, things are moving so fast. Is there anything you've had to, like, rethink, redo since you first built it?

    6. MNMarily Nika42:35

       I actually added bonus sections. And one bonus section was ChatGPT and how it was trained. This is because I started this new cohort in December. And on day one, the question I got is, "What is this? How did it start? What is going on? How did they train it?" So I added a dedicated section for it and I point people to it.

    7. LRLenny Rachitsky42:52

       Amazing.

22. 42:53 – 44:08

    Why you should create your own course

    1. LRLenny Rachitsky42:53

       Anything else that you'd like to share before we get to our very exciting lightning round?

    2. MNMarily Nika42:58

       It was someone that recommended I actually did a course. And in the beginning, I, it was, it was not... In the beginning, I laughed and I said, "Wait, people would wanna learn from me? Really?" And of course they did. And I'm teaching so many people. So what I wanna tell people is, don't underestimate this. Try creating your own courses as well. People may want to learn what you take for granted. For them, it may be game-changing. It can be life-changing. So building courses is an amazing thing. And you know, we're living in the whole content creation era now, so the course is content. So go try this.

    3. LRLenny Rachitsky43:33

       I find that teaching and at least crystallizing thoughts is one of the best ways to learn it yourself. I imagine you learned a lot about AI, much more than you even came into it with, just putting it together into a course.

    4. MNMarily Nika43:44

       Absolutely. And I, I got some, uh, uncomfortable questions that I had no idea how to tackle. Like people on day one were like, "How do I assess the trade-offs between these two different models?" And I had to figure out how to answer these things and how to incorporate it then in my course. So learning from the students, learning from the course, learning from explaining is just so valuable. It's all skills that you can get.

    5. LRLenny Rachitsky44:07

       Well, with that,

23. 44:08 – 48:01

    Lightning round

    1. LRLenny Rachitsky44:08

       we've reached our very, very exciting lightning round. I've got five questions for you. I'm gonna go through 'em pretty quick. Whatever comes to mind, share. We'll see how it all goes. Sound good?

    2. MNMarily Nika44:18

       Sounds good.

    3. LRLenny Rachitsky44:19

       Two or three books that you recommend most to other people?

    4. MNMarily Nika44:22

       Inspired. It taught me, it's, it's all about how to create tech products people love.

    5. LRLenny Rachitsky44:28

       By, uh, Marty Cagan, right?

    6. MNMarily Nika44:30

       Yes. That's the one. Yeah, yeah. Mm-hmm.

    7. LRLenny Rachitsky44:31

       Great. Cool. Anything else? Or that's the one? That's the one that comes to mind?

    8. MNMarily Nika44:35

       You Look Like a Thing and I Love You, and I have it right here. It's a great thing. Super, super cool. It's about how AI works and why it's making the world a weirder place. It's actually a very fun book. And there's one more, which is-

    9. LRLenny Rachitsky44:48

       Oh.

    10. MNMarily Nika44:48

        ... a book, a workbook I originally launched with Alana Carr. And it's about, um, it's a workbook for women in tech trying to navigate working in tech. It's called Adventures of Women in Tech workbook. So that's another thing that I want to shamelessly plug in.

    11. LRLenny Rachitsky45:02

        Uh, that's a great choice to plug. Where can folks find that? Is that on Amazon?

    12. MNMarily Nika45:06

        Yeah, Amazon.

    13. LRLenny Rachitsky45:07

        Amazing. What's a favorite other podcast that you like to listen to?

    14. MNMarily Nika45:11

        I like Boz's podcast. I don't know if you're aware of it. Boz is the CEO of Facebook. He has a great podcast.

    15. LRLenny Rachitsky45:19

        I have not heard it. I do know of Boz. I will check that out. I didn't know he had a podcast. He had some great writing over the years. Maybe that's why he doesn't write anymore, he has this podcast. What is a favorite recent movie or TV show that you've loved?

    16. MNMarily Nika45:30

        Oh my God, The White Lotus. People were talking about this thing. I, I ended up, you know, just trying it out and me and my husband, we just binge-watched the whole thing. It's just so different, so mind-blowing. Gets you excited about going to Hawaii again. It's just, (laughs) it's, it's really good.

    17. LRLenny Rachitsky45:45

        Have you seen the second season?

    18. MNMarily Nika45:47

        I've seen it and it's so much better than the first, which is rare. (laughs)

    19. LRLenny Rachitsky45:50

        Yeah. I agree. I agree. Awesome. Love that show. What is a favorite interview question you like to ask? And bonus point if it's AI related.

    20. MNMarily Nika45:57

        I love to ask people, "How would you explain a database to a three-year-old?" And I know it's, it's kind of an AI and not very relevant to AI. And, but I love asking it because people are kind of thinking back saying, "Wait, what did you just ask me?" But it's so important to be able to explain things in a simple way and have the storytelling to convince a kid and really explain technical terms to non-technical people.

    21. LRLenny Rachitsky46:23

        Favorite AI based tool that you think people should check out?

    22. MNMarily Nika46:27

        I mean, we can talk about ChatGPT now. My head is on ChatGPT. This is what comes to mind. Um, well, the Lensa was pretty cool too, right? We all uploaded our photos and were able to see what we would look like as fantastic heroes. I have to say, I tried being the male version 'cause it was so much cooler than the female version. So that's one I recommend to people. Try the male version.

    23. LRLenny Rachitsky46:48

        That's fun. And there's actually a, uh, they actually have pets now. That's what got me to download it and pay for it. You can take pictures of your pets and they look so fun. That's like a killer feature right there.

    24. MNMarily Nika46:58

        Oh my gosh.

    25. LRLenny Rachitsky46:58

        Good job, Lensa. And the app is Lensa, right?

    26. MNMarily Nika47:01

        Yeah. Lensa.

    27. LRLenny Rachitsky47:02

        Amazing. Marily, thank you so much for spending time with me, sharing your wisdom. Two final questions. Where can folks find you online if they wanna learn more and reach out? And how can listeners be useful to you?

    28. MNMarily Nika47:13

        Thank you so much. Uh, people can find me on Instagram. I also have a product channel on YouTube that you can check out. I just started it. I'm getting used to the whole process. I'm also kicking off a newsletter. Just any social, reach out and you'll see all my links.

    29. LRLenny Rachitsky47:28

        How do they find the YouTube channel? How do they find the newsletter?

    30. MNMarily Nika47:31

        Type in Marily Nika.

Episode duration: 48:01