This One Thing is Stopping You From $500K as an AI PM

1. 0:00 – 1:09

   Intro - Why AI System Design Interviews Matter

   1. AGAman Goyal0:00

      Product design is dead, and that's not really clickbait, but it's actually from some of my recent AI PM interview experiences.

   2. AGAakash Gupta0:07

      AI product managers are making millions of dollars. At OpenAI, the average stock grant per employee is $1.5 million. At companies like Google and Meta, employees are getting huge stock grants.

   3. AGAman Goyal0:24

      I was not really asked any of those conventional make a fridge for blind people kind of question. It has moved to AI system design, where they're not just testing product sense, but they're also trying to test your system design knowledge.

   4. AGAakash Gupta0:35

      If you wanna get one of the top AI product management roles at companies like OpenAI or Google or Meta, and you want it to be at the high end of the pay, the $1 million-plus roles, the roles that have $1.5 million in yearly stock grants. But not just anyone can land these high-paid AI product management roles.

   5. AGAman Goyal0:54

      So with Aakash, we'll walk you right through how a great response looks like, and we hope you benefit the most out of it.

   6. AGAakash Gupta1:00

      So without any further ado, let's get right into it. When it comes to the AI system design interview, they're looking for your ability to go deep on a technical topic.

2. 1:09 – 1:24

   Mock Question - Build a Churn Reduction Agent

   1. AGAakash Gupta1:09

      So we're gonna show you how to ace this with the mock interview question: Build the system design for a churn reduction agent.

   2. AGAman Goyal1:18

      So we, we, we need to build, uh, a churn reduction agent. Am I right? I would like

3. 1:24 – 4:47

   Clarifying Questions Begin

   1. AGAman Goyal1:24

      to-- I, uh, I think before we jump into solutioning, I have some clarifying questions. Is it fine if I ask some of them?

   2. AGAakash Gupta1:30

      Yeah, go for it.

   3. AGAman Goyal1:31

      Uh, I would actually like to understand, I know ch- what churn generally means, but in this specific scenario, do, does it have any specific, uh, meaning or any relation to any kind of product or anything which you would like to maybe give some detail about?

   4. AGAakash Gupta1:49

      Yeah. Churn, you can define it. I think that at the high level-

   5. AGAman Goyal1:54

      Mm-hmm

   6. AGAakash Gupta1:55

      ... there's a engagement churn that we're gonna see, which is they use the product, they've tried it, and then their engagement drops off. And then the lagging thing we'll see is they stop paying us. [chuckles]

   7. AGAman Goyal2:07

      All right.

   8. AGAakash Gupta2:08

      So we don't have, like, a price, more precise input for this interview.

   9. AGAman Goyal2:14

      Got it. Sounds good. Also, with, with regards to, like, with regards to the scope of this, are we talking about users of any specific domain, like, like mobile app versus desktop app, or is there anything there which needs to be understood here?

   10. AGAakash Gupta2:32

       I guess we wanna consider all platform behavior. The important thing to think about is what is gonna help us secure more revenue. And-

   11. AGAman Goyal2:45

       Mm

   12. AGAakash Gupta2:45

       ... if we can get signals about mobile versus desktop usage and it's relevant, then I think you should focus on it.

   13. AGAman Goyal2:51

       Got it. Cool. Are there any constraints regards to time or the effort or budget we have to launch this feature, or product rather?

   14. AGAakash Gupta3:00

       I would say think about the best case scenario. What would be the ideal thing to design, and don't worry too much about constraints for this interview.

   15. AGAman Goyal3:08

       Got it. Cool. Okay. So no constraints. I'll try to keep it to six months, if not more, but... Uh, finally, I know the goal is churn reduction. Do we have any other goals associated with the product apart from churn reduction?

   16. AGAakash Gupta3:24

       That's a good question. Um, I guess just to build a good system. Think about it as like its own code base, maybe its own repo. So how are we gonna build it specifically? The technical areas is what I'm most interested in.

   17. AGAman Goyal3:37

       Hmm. Got it. Okay, cool. So, so the reason I ask is because, uh, there are a lot of, uh, things which a company does because of its competitors. So if our competitors are trying to do XYZ, is that one of the reasons we are trying to do this? Or there is no independent, like, there is no, like, dependent correlation, is what I'm trying to deduce from it. But it looks like in your case, we just need to build an independent system, and we don't really have any other goals apart from churn, uh, in a nutshell.

   18. AGAakash Gupta4:07

       Yeah. I mean, just the normal reasons, right?

   19. AGAman Goyal4:09

       Okay.

   20. AGAakash Gupta4:10

       If we can identify churn, we can take actions in the product-

   21. AGAman Goyal4:14

       Yeah

   22. AGAakash Gupta4:14

       ... to improve engagement. We can secure more revenue. Generally tends to be more profitable for us than focusing on acquisition, so we have an overall push there.

   23. AGAman Goyal4:25

       Sounds good. Cool. And do you have any suggestions as to, uh, what kind of an, uh, industry product should I go towards, or ac- that's up to me to decide, uh, which product to pick for this?

   24. AGAakash Gupta4:40

       Just consider, like, um, software.

   25. AGAman Goyal4:42

       Okay, cool. Okay. Sounds good. So I'll just take a minute.

4. 4:47 – 6:26

   Defining the Product Vision

   1. AGAman Goyal4:47

      I'll try to define, uh, the product and the vision I have, and then I'll just get back to you in about, let's say, 30 seconds, if that's fine.

   2. AGAakash Gupta4:55

      Sure. Take your time.

   3. AGAman Goyal4:57

      Cool. Okay. So I think with regards to the vision, as you're speaking more about agentic AI, uh, customer care is something which we are trying to, you know, uh, as an industry, not just telecom industry, but even, uh, other industries, we're trying to use agentic AI to make our workforce more efficient. So I think what I'll do is I'll try to take, uh, an example from telecommunications in- industry because I think it will be easier for people to understand. And also, uh, in terms of the product, I'll try to take a mobile app, uh, which is, which will have the, uh, necessary agentic AI pipeline built in, which effectively should help us reduce churn. But this is my vision for the product. Is there any question before I jump into the next section, or what are your thoughts on this?

   4. AGAakash Gupta5:40

      Sure, yeah. Makes sense to kind of focus it in a specific real-life, more textured situation.

   5. AGAman Goyal5:48

      Yeah. Cool. Okay. So what I'll do now is I'll start talking about some of the users. I'll try to prioritize, uh, one of them, and I'll go towards, uh, the pain points. And, uh, finally, I'll move towards solutioning, where we'll discuss h- about the different AI solutions. And finally, we'll jump into system design in terms of the diagram, as well as some of the metrics we can track. Does that sound good? Cool.

   6. AGAakash Gupta6:14

      Yep, go for it.

   7. AGAman Goyal6:14

      Okay.Okay. So I'll just take 30 seconds. I'll get back with some of the users we can target. So I think with regards to users of any of, of not just this kind of product, but any kind of product,

5. 6:26 – 9:28

   User Segmentation and Prioritization

   1. AGAman Goyal6:26

      there are different ways how you can segment users. What I would like to potentially do is that I will actually, I'll try to stick to our, uh, users in terms of when did they join us. So there are a lot of new users. There are obviously power users. There are users who also, uh, focus from B2B perspective. A telecom company has B2B users as well. There are, uh... So I think, uh, I'll try to stick to different types of users. Broadly, I think because we want to, uh, prioritize to only one user, I will try sticking to power users, and the reason is because these are the users we want to take care of the most. I'm not saying n-not all users are important, but, uh, in any product, the users who are mostly engaging with you, who are most deriving value for you and out of your product, you want to protect them, and you want to ensure that their interests are well met. So, uh, we do not want to lose them. So from a churn per- uh, point of view, I, I think I want to prioritize this segment of power users and proceed from there. Uh, do you have any question, Aakash, uh, before I proceed to pain points?

   2. AGAakash Gupta7:35

      Yeah. I th- I think power users make sense with our broader strategy anyway, so great.

   3. AGAman Goyal7:40

      Um, before jumping into, uh, pain points, what I would actually-- what I like to do is also something called user journey. Pain points are often derived through the different phases of user journey. So I'm just going to spend a few seconds to define what a user journey of a power user might look like. So we can understand that this user already has been part of this network for a while and has been using our app and com- and customer care for a while now. And one thing I definitely know is that there are definitely going to be a, uh, a lot of, uh, there might be a lot of customer issues. So this user, uh, user calls or contact customer care through phone, tries to get a ticket for this user, uh, for the problem rather. Once that happens, the customer normally waits, reaches out again, if required, to the customer care through email maybe. Once issue is resolved, normal, normal, uh, system n- normal system, uh, power is restored, and the user effectively starts using the app as well now. Uh, now on the app, there-- Now on a telecom app, you can do a lot of things, but just to not take much time, uh, there are a lot of benefits provided on the app, uh, these days for, for our users. So he can browse benefits, and also he can actually try to, uh, get a Wi-Fi of the same company or, uh, or avail other services. So these are some of the user journey items which I feel is very, uh, common among our telecom, our top telecom providers, and there are a lot of pain points

6. 9:28 – 13:21

   Pain Points and User Journey Mapping

   1. AGAman Goyal9:28

      among these. Firstly, I think user call contact. I mean, everyone hates calling customer care. You do not want to spend your time calling customer care if especially you're a power user. So I think one, one pain point is definitely huge amounts of time and effort taken for, uh, each of these customer interactions, customer care interactions rather, which, uh, also ultimately causes anxiety, which also causes a lot of frustration, delay, and I think it overall, it doesn't really l-leave a good mark on the user's experience. But with regards to pain points, huge amount of time and effort for customer care. Apart from this, there is also the lot of customer pain points around the app experience. So when you open the app, you, uh, you might not be able to, uh, track, uh, sorry. You might not be able to track your, uh, request, uh, uh, across your, across customer call and app. So that kind of causes a lot of, uh, confusion as well. That, "Was my problem ever filed? What did I do wrong? Do I need to call the u- uh, customer care again?" Uh, and this is something which we definitely do not want to do. Finally, I think one pain point is also that regarding benefits. We-- There are a lot of users with different interests, and we do not want-- If you like dining, we do not want you to unnecessarily show maybe a benefit for a Wi-Fi. So I think there are a lot of irrelevance. Irrelevance becomes a huge problem in terms of benefits and services. So we are not able to tap in the, the right value for the right user, and that causes also, uh, friction, uh, loss of potential upsell which otherwise could have happened. So I think these are some of the pain points. I know there could be a lot, but these are some of the pain points I'm looking at. Before I prioritize, Aakash, do you have any questions on any of these maybe?

   2. AGAakash Gupta11:29

      Yeah. I think, um, these are good ones to focus on for this particular system design.

   3. AGAman Goyal11:34

      Okay. Uh, with regards to prioritization, I like to consider some of these things. Firstly, uh, alignment to, uh, uh, my vision, original vision, because there are a lot of problems. I do not want to focus on problems which do not align with my company's or with my product's vision. Apart from that, it is also frequencyUh, and impact. So what is the frequency of the pain point? How much impactful, uh, or, uh, how much, uh, problems it is causing for my consumer? I think with regards to that, I definitely would rate this as a very highly recurring, uh, problem with huge impact in terms of your experience. So I'll just try to, uh, maybe mark it as, uh, high for, uh, all the three domains. Apart from this, I do understand that incons-inconsistent ex-cus- experience is a big problem, but, uh, I feel that even if we do solve your problem, uh, at the end of the day, it might not matter as much, uh, in the short term. So we might go with a mid here, uh, and not a high for, uh, some of these because it doesn't directly align with our immediate vision, which is to create a highly engaging product and reduce churn. So I might just go medium here. Irrelevance in benefits causes friction, loss of potential and upsell. I feel this is again, very important that we surface relevant benefits and services, but on a-- but, uh, that is more to do towards, uh, making our user experience highly, highly engaging, which I do want to do. But when it comes to churn reduction, I feel that it might again not be our top, uh, lever in terms of, uh, reducing churn at this moment. So I'll again try to shift towards medium for this. So I'll, I'll try to prioritize

7. 13:21 – 16:53

   Brainstorming Agentic AI Solutions

   1. AGAman Goyal13:21

      only one pain point at this time, which is this one, and I want to try to devise an agentic solution just to resolve this kind of a pain point, uh, in our system. What do you think? Any thoughts there before I move into solutioning?

   2. AGAakash Gupta13:35

      How do we-- One of the things we wanna solve for in the system design is-

   3. AGAman Goyal13:39

      Yeah

   4. AGAakash Gupta13:40

      ... just giving our product teams the right signals that, hey, this user is about to churn, so we need to do something, take an intervention.

   5. AGAman Goyal13:47

      Yeah.

   6. AGAakash Gupta13:47

      So how do we get that early warning sign?

   7. AGAman Goyal13:50

      Exactly. So I think that that's a very highly proactive system which we wanna build so that we, before the user calls for you to churn, before three to four calls or before three to four, what do you say, uh, even types of interactions we are able to predict. So that is something which I'll be explaining more in my solution section. So if it's fine, I can-- Now I think that's a good segue to our solutions section. So I'll just take 30 seconds, and I'll try to jot down all the different solutions possible, if that's fine, Aakash.

   8. AGAakash Gupta14:21

      Sure.

   9. AGAman Goyal14:22

      So I think with regards to solutions, there are a lot of different solutions possible to this one problem. I think one of them is a bot, which is pretty common, where the system effectively tries to understand from your past calls his sc- transcription and accordingly helps you guide your solutions very quickly. But I feel that that might not be as engaging as possible because a human always prefers, uh, a more, uh, human-like touch. There is also something around voice bot, and I feel a voice bot might really be a very good scenario in this, where you're not just able to talk to the AI agent, but you're able to show your screen to it. You're able to actually help it exactly end-to-end guide you and not just, you know, just, uh, redirect you to a human agent. So I think an end-to-end AI voice bot assistance would be something amazing. In terms of, in terms of a more basic solution which might not require AI as such, uh, we can, uh, make our experience more gamified in the app to just show, uh, the features and, uh, benefits in a way that the user always tries to, uh, avoid churning just because they keep getting some of these benefits and features. But I feel that because at the core, we want to improve the user experience, I would like to go and try to, uh, build more of a voice bot agent, which tries to not just, not just speak to you, but actually tries to predict from your past data, l-like you mentioned, Aakash, when the user is going to maybe, uh, be in that red zone of churning and accordingly tries to, uh, activate that system of maybe sending more offers, maybe trying to send more benefits to them, and also trying to talk to the user end to end and solve those problems as soon as possible. So this is what I would like to prioritize, if that's fine, uh, Aakash, to you, or I would love to know your thoughts.

   10. AGAakash Gupta16:18

       It's gonna be a big project. I don't know, six months will depend really on the resources, right? But as I said, we can think about a best case scenario. [laughs]

   11. AGAman Goyal16:30

       [laughs] Yeah. I think, no, that's true. I think six months is a big time, but I feel that, uh, we can definitely map out an MVP version of it in six months. Uh, and from there, maybe craft a very good agent in maybe nine to ten months. But yeah.

   12. AGAakash Gupta16:50

       Sure.

   13. AGAman Goyal16:50

       What do you think?

   14. AGAakash Gupta16:50

       Let's see what it would look like.

   15. AGAman Goyal16:52

       Okay, cool.

8. 16:53 – 21:26

   AI System Pillars - Model, Data, Memory

   1. AGAman Goyal16:53

      So I think before jumping into our system design diagram, I would like to just, uh, talk about different layers of an AI system, uh, today and what this point might actually have. So, uh, okay, let's actually go to a new page. Uh, okay. So now that we are targeting this, maybe... Okay. So I think each, each AI agent or bot has potentially three things, I think.One is obviously model, data, memory. The-- Without-- These three things are the pillars of any AI system. And in terms of model, to be very honest, you already have some of the best AI models available at your disposal, and we could accordingly do a competitive analysis and, uh, uh, basic testing on an, uh, on an engineering level and see which fits our use case. For example, for a voice, uh, would you prefer a s- uh, maybe an Opus versus an, uh, like a GPT-5 is a question which you can always, uh, try to understand, do some benchmarking, and come to that conclusion. But overall, I think we don't need to spend much time on the model right now, and you can-- y- all the best models are at your disposal. So I'm just going to park this, uh, as, uh, as per your use case. In this case, it is voice bot, so I might actually go to, uh, uh, maybe Gemini because I feel that has a great voice experience. But, uh, again, it can be anything based on how your engineering team or ML team perceive, uh, perceives this. Now, I think coming to data, this is probably the core of it, right? Without data, your very powerful model is of completely, uh, like it's completely useless. In terms of data, you definitely need call transcription, right? That's the most basic thing you can provide. But you also need different, uh, signals. For example, app usage. For example, your network status through the past, let's say, twelve months. Your-- What has been your, uh, uh, what, uh, what has been your, not search history, but what has been your, uh, connectivity with regards to maybe the different, uh, competitors? Are you, uh, for example, if you're traveling a lot, are we, uh, uh, are we able to provide you those benefits of traveling which our competitors are providing? Or what kind of a user are you? So I think having a lot of different data points always helps you understand what kind of user are we dealing with. And accordingly, I think, uh, putting them, putting all these in a churn bucket, uh, would be helpful. For example, if you're a high-risk churner, then maybe put it in the red bucket, or maybe if you are not, maybe in a green bucket. So we can-- I think, uh, the model can effectively try to use these data points and then predict and put you in that bucket so that accordingly we can try to, uh, understand you. Also, also, I think we would also need some, some last-minute, uh, uh, retention offer. So let's say if you are in a red bucket and an AI agent w- might, uh, let's say, is now unable to resolve, we need some sort of, uh, a last-minute retention offer just for you so that we can actually send it, or the AI agent rather can send it to you. I think these are also highly, highly crucial. Finally, jumping into memory. Uh, I think in terms of memory, there are different types of memory available. Uh, there is episodic memory. There is, there is also session memory. So I think, uh, we need to, uh, we need to understand that, uh, we definitely can-- we do not want to store anything and everything, so we need to be, uh, really efficient here as well. So I, I would say that, uh, an episodic memory is highly important. What do I mean by episodic memory? Is basically in, in the, in the cases where the user has ever tried to contact customer care through chat or through call, what has the conversations looked like probably will be the most important set of transcriptions or data we can analyze from. Remaining data points are important, but if we cannot contextualize their previous conversations, it might be very difficult for us to provide a great solution. So I'll just stick to episodic memory for now. But overall, I think these are the three pillars of any AI bot. How-- What do you think, Akash, uh, before proceeding to a system design question? Is there anything else you would like me to also cover here?

   2. AGAakash Gupta21:25

      Well,

9. 21:26 – 22:25

   Latency and Performance Tradeoffs

   1. AGAakash Gupta21:26

      just make sure you think about latency of the bot and how quickly it's-

   2. AGAman Goyal21:30

      Hmm.

   3. AGAakash Gupta21:30

      -working. That'll be one of the key areas we'll need to p-pressure test it for.

   4. AGAman Goyal21:34

      Yeah. Well, uh, that's an amazing, uh, thing. Okay. Yeah, I think latency and overall performance, whether you are fast and also whether you're accurate, is something which we want because a lot of times there are wrong prompts which are injected, and a lot of times the, uh, bots are annoying and not really helpful. So I think doing-- I think I'll, I'll put in more metrics as to what metrics can we test later. But yeah, I think doing-- understanding, I think, response time, uh, from bot, uh, as well as maybe some sort of a basic customer satisfaction score, uh, or m-maybe NPS, maybe feedback, uh, might be important. But yeah, I think overall, uh, these are some areas, uh, which we could think of. So if it's fine, can I proceed to a system design

10. 22:25 – 26:10

    System Design Diagram Walkthrough

    1. AGAman Goyal22:25

       diagram, uh, to try to portray some of these in, uh, like, uh, in more pictures and words?

    2. AGAakash Gupta22:32

       Let's do it.

    3. AGAman Goyal22:33

       Cool. Okay. Okay. So, uh, uh, in the top we have our very basic interaction layer, which you can say is our mobile app in this case. Could be web app, whatever it is for your use case. Uh, it, uh, it-- We do need to connect this to our, uh, orchestration layer. Now, in any agentic AI system, you do need an orchestration layer which coordinates across different agents and then sends the response to the mobile app. So in this case, I think I have an orchestration layer. Now we do... Okay. Now, in terms of, uh, agents, there could be multiple agents in this scenario. I think you definitely need a data analyst for, uh, a data analyst agent for you who, whose sole understanding is to, uh, go deep into your data, get all the key details, get all the key points, and then serve it to you, right? Uh, apart from this, we also do need a bot which, uh, which actually, uh, tries to, uh,

    4. AGAakash Gupta23:39

       [audio glitching]

    5. AGAman Goyal23:40

       Right? So we need like a, a very solid customer voice agent in this case, uh, who responds to you based on the data. And finally, because we are supporting churn, we would also then need some-- we would also potentially, uh, n-need an action... Not action. Okay. Uh, maybe an execute-- executor agent who, whose sole task is to make those executions of maybe, let's say, the re- the retention offers. So in today's world, a customer care actually, uh, uh, has a lot of different... Uh, there, like, there are a lot of different departments within customer care and, uh, we want to try to replicate that entire system where we have a data analyst, we have a agent which talks to you. We also have an agent who, who ultimately executes some of those actions, whether it is retention offers or whether it is, uh, even maybe escalating to a human. And I think underneath, one, one basic thing which we do, uh, need to understand is that they are, uh, connected to... Now, it could be RAG, uh, which is probably the most common way you, uh, use, uh, your, uh, data extraction through the vector database. Uh, but yeah, this is where we could have our data and we effectively, our data analyst agent, customer voice agent, executor agent just uses this. So we need a data layer, and we obviously have our, uh-Yeah. Okay. We, we do have our, uh, model APIs here, uh, which m- which could be called on as and when, uh, we are using it for predicting, uh, the different data or services or data patterns, or even for trying to predict what the human will say, what, what are we trying to eventually arc-- uh, like, uh, create an offer for them. Let's say if they say that, "Hey, uh, I want to... I, I like Netflix." So maybe we create an, uh, we create an offer especially for them just because they, uh, do, uh, like Netflix or streaming. So I think these are the basic elements required. Uh, I know that there are a lot of other technical terms and a lot of other technical pieces to it, but on a very high level, I feel, uh, uh, if, uh, we can definitely rely on something like this. Uh, what do you think, Aakash? I'm happy to, uh, also consider any other suggestions or how do you-

    6. AGAakash Gupta26:08

       Yeah, I wanna dive a layer deeper. Like, how are

11. 26:10 – 27:25

    LLM vs ML Models - When to Use What

    1. AGAakash Gupta26:10

       you gonna... What ch- how are you gonna do the churn signals? Are you gonna use an LLM or ML models? What models? Those kind of thing.

    2. AGAman Goyal26:17

       Sure. I think in terms of churn signals, I think our data analyst agent will be strictly working towards it. So for example, uh, if we actually move back, I think we w- are trying to use different buckets. So each user already has been bucketed into different categories of churn. And in term-- and coming back to our diagram, I think data analyst, uh, holds all those patterns with regards to each and every user. So once you start interacting with a user, you effectively, uh, the customer voice agent and the executor agent actually know what exactly the percentage of the user is to churn, what, what kind of bucket they fall into, and accordingly, the voice agent will start taking, uh, actions. Like if the user is not really enjoying the experience, if and if the churn bucket is really high, it is red, we'll eventually quickly go to, uh, solve them with some retention offer. So I think having that kind of a framework or a system help your agent to ultimately move through those pieces.

    3. AGAakash Gupta27:21

       Okay.

    4. AGAman Goyal27:22

       Uh, yeah. So yep. Okay. I think in

12. 27:25 – 35:43

    Metrics and Evaluation Framework

    1. AGAman Goyal27:25

       t-- uh, now even though we have created some sort of a diagram and have a good solid system for MVP, I think metrics play a huge role, and there are different kind of metrics we can work towards. I think, uh, in this particular scenario, as Aakash did mention, like, uh, we, uh, uh, do need to focus obviously on latency and performance because those are the end, uh, performance indicators. But I, uh, but I think let's start with more regarding, I think, model as well. So from just the model perspective, I think like, uh, some of our metric like, uh, ROUGE, which is recall, which is basically, uh, recall, uh, over our, uh, intersection will really be helpful to understand if the model is able to, uh, uh, re- uh, use the data and is able to actually give you very accurate outcomes. Ultimately, uh, also gives us understanding of what's the hallucination of our model here. Uh, so I think ROUG, uh, BLEU is also a good metric, but we can stick to recall, uh, right now. I think apart from the model, uh, you also want to understand with regards to our, uh, latency, so, uh, what, uh, what's the response rate o- on an average per user. Also, apart from that, we do need to understand, uh, from an end user perspective, what is the, mm, overall NPS. Uh, is the user really liking the performance? Is the user... Is the problem getting resolved? So what is the percentage of issues are resolved by AI bot alone without escalations is a huge metric. If our AI bot is just escalating it to human, then it is, I think, not really effective. So that's something we need to understand. Also, I think our business metrics are hugely important, I think in terms of, uh, retention is one, definitely. Uh, revenue is a huge factor, I think again, driven by retention. But overall, I think, uh, a good understanding of what a model is doing, how is it performing, how is the end user actually seeing this, and what ultimately are the levers pushing for business impact is what I think could constitute a good set of metrics for this particular use case.

    2. AGAakash Gupta29:46

       All right. Where does this system fall over?

    3. AGAman Goyal29:48

       You mean like from the mobile app perspective or?

    4. AGAakash Gupta29:51

       What would be some of the failure scenarios you would design around?

    5. AGAman Goyal29:54

       Okay. Got it. Got it. Okay. Yeah. I think that's a good question. From, uh, weird cases, I could, uh, let's start from first. I think initially, uh, it initially it will be, uh, likeUh, let's say our, uh, our model, uh, is down or our, or our model is not working, uh, immediately we need to, uh, redirect to a human. So that's a fail-safe you definitely need to build. You do not want, uh, you know, that just because your model is down, uh, that the user actually just goes away. You need to redirect to human on that bot itself, uh, and not make them-- make the-- increase the churn percentage. I think apart from that, it's also important to understand, uh, the latency. So if, if model is maybe taking, let's say, more than thirty seconds to reply, uh, on an average, I understand that, uh, for some of the, uh, decisions for it to make, it might need more time. For example, retention offers might take time. But on an average, if it is taking more than thirty seconds, then I think it's important again for us to, uh, redirect to a human or try to understand how we can help the user, uh, reach that, uh, ultimate goal. Apart from that, I think I also feel that apart from model and latency, if the, uh, there is also case where the, uh, where a model is maybe, uh, repeating same response or, or a user is asking same questions and maybe getting frustrated overall, it's again important for us to understand when do we pass the baton to human. But then overall, in terms of failure cases, we, we understand that, uh, the AI agentic system will be the best case scenario, but we also want to prepare, as Aakash suggested, for some of our fail-safe and failure cases. These are the three I can think of right now, but I'm sure there could be much more failure cases we could prepare for.

    6. AGAakash Gupta31:57

       All right. And what about, let's say, how are you gonna build your system to handle this 10X spike in traffic?

    7. AGAman Goyal32:03

       So got it. So for example, let's say in six months we launch our MVP, we are successful, and now we want to scale it from, let's say, in a testing mode to, oops, uh, to a full-time, uh, production mode. I think for going to 10X, there are again multiple things I think I'll, uh... So scaling further, again, I'll break it down. I think you definitely from a model, uh, from a model perspective, you would, uh, now need to have that bandwidth to be able to call that model ten times or a hundred times more. So you definitely would need, I would say, like, uh, a very str-- like you, you would basically need a much more strong and dense server to basically handle all these models. A lot of companies host them on-prem as well to just to ensure that. So considering that it's a huge company, uh, we might then switch from just using a cloud to maybe going on-prem so that we can ensure that we are able to control and, uh, manage, uh, every, every decision or every call, API call or every, uh, interaction from model from our end much more quicker. Apart from the model, I feel because we are going 10X, latency suddenly becomes important. So I feel in my earlier response around on-prem, I'm, uh, I feel that a latency might also improve if we, uh, handle it, uh, on-premise. Uh, so, uh, I'm just going to stick to that, but we'll definitely need a much more thorough testing in new on-prem environment to ensure that our latency is very well. I think, uh, data becomes a huge, uh, bottleneck because now we suddenly have our, uh, our entire data from just maybe thousand users to, uh, maybe, uh, ten thousand or hundred K plus users. So we need to ensure that, uh, our vector, uh, like we need to ensure that we use a vector database hundred percent, uh, for MVP. If we are not using it, I think ensuring something very quick as a vector database, uh, would be, uh, would be highly, highly, uh, in required because it's much more faster and efficient than a normal database. I think from a memory layer perspective, I also, uh, there are a lot of intelligent... Sorry. From a memory layer perspective, there are a lot of intelligent memory, uh, solutions which actually, uh, decide, uh, whether at all we need to save, uh, uh, or process a certain kind of response. Uh, like, uh, uh, so this is not a sponsored post, but I, I do know that, uh, MemZero is one. There are a lot of other solutions as well. So I mean, I think, uh, y- un- having an intelligent layer over memory also helps you deal with so many requests much more quickly and ultimately improves latency. So maybe using anything, uh, which basically understands whether do we even need to save it, or whether we need to process it in this particular field or not, and just trying to deal with it is-- gets really important, I feel. I think these are some of the different factors, Aakash. Uh, do you feel we have covered the ground here, or am I missing something? Uh-

    8. AGAakash Gupta35:32

       That's all the time we have for unfortunately. But Aman, thank you so much for walking through this interview.

    9. AGAman Goyal35:38

       Thank you so much, Aakash.

    10. AGAakash Gupta35:40

        So now let's jump out of the interview, and let's

13. 35:43 – 36:42

    Feedback - What Went Well

    1. AGAakash Gupta35:43

       review, right? What did Aman do well? What could he have done better? Aman, why don't you go first, and I'll add on to it?

    2. AGAman Goyal35:52

       Yeah, I think asking clarifying questions always helps, right? So I think that was fine. I think with regards to interview process, uh, going deeper on users and problem or pro- pain points is very important. I think I could have spent more time on users, breaking down the user flow much more better, because a telecom company has so many different kinds of users, so it's a very good opportunity for everyone to spend time there. I think I slightly missed out there. Apart from that, I think I also, with regards to our solutioning, uh, I think I did decently well, but I feel that some of the areas which you also pointed out around latency system requirements, trade-offs, those are some of the things which I could have in- uh, just added by myself and just ensured that I build an end-to-end systems. And, but yeah, I think overall it was fine, but, uh, I would love to know your thoughts

14. 36:42 – 39:25

    Feedback - Technical Fluency and Delivery

    1. AGAman Goyal36:42

       as well.

    2. AGAakash Gupta36:42

       Yeah, so I think the big challenge with this type of interview is time management. And so-

    3. AGAman Goyal36:47

       Yeah

    4. AGAakash Gupta36:48

       ... overall, I wouldn't overly call out the amount of time you spent on users as too small, because you wanna get to that system design diagram, which you got to.

    5. AGAman Goyal36:57

       Sure.

    6. AGAakash Gupta36:57

       I think that ultimately where I think you could have improved in this interview is technical fluency and depth and delivery. So-

    7. AGAman Goyal37:05

       All right

    8. AGAakash Gupta37:05

       ... in terms of technical fluency and depth, sometimes when I'd ask a question like, "Are you gonna use an LLM or an ML model?" what I'm really hoping is for you to first confirm that you understand what I'm talking about, and then talk about the pros and cons of each option. So an answer might sound a little more tight if it sounded like this: "That's a great point. You know, we don't always wanna use an LLM when an ML model will do, because we know something like an XGBoost algorithm will also be cheaper and a little bit less black box. So if I think about the pros and cons, an LLM model may be able to adapt to new changes in new data sources a little better, and might be able to use an agentic search, but it would be much more expensive, and it would be more of a black box. So actually, I think for the data analyst part of the agent, we should use an XGBoost model." That might be an example of a strong response. And then the other area to improve is delivery. You have this crutch of "uh," which you basically, you don't have any pauses in your speech. Whenever you have a pause, you say, "Uh." And so-

    9. AGAman Goyal38:02

       Right

    10. AGAakash Gupta38:02

        ... those are the two big areas to improve in this interview for me, but I think that people watching are gonna get a lot out of this, because now you've seen how to present a structure, how to have clarifying questions, how to deal with lack of definition from an interviewer and make assumptions, how to share your screen and draw a system design diagram even if you are a PM. So those are the areas for people to take away. Anything else people should take away or focus on?

    11. AGAman Goyal38:28

        No, I think, uh, firstly, great, uh, feedback. I think one thing you should definitely focus a lot on is asking, uh, questions to the interviewer, uh, and may ensure that it is engaging and not just that you're talking, because a lot of times the interviewers also do not expect a one-side, uh, monologue or conversation. So ensure that you're keeping your interviewer very engaged and walking down what you're doing. So even if you share a screen, there is a tendency for people to just work through silently, which I do not, uh, suggest. So ensure that as you're typing, you're talking to your interviewer, you're trying to speak your mind loud so that they can assess what's your thought process. And do not really be very shy. Uh, just, uh, go, uh, go there and crush it.

    12. AGAakash Gupta39:09

        Yeah, and in terms of interviewers, I would say I was more on the easy side. Some interviewers will ask more pressing questions, and it'll mess up your time management, so be really careful about how you handle your interviewer and your time management. In this case, I modeled

15. 39:25 – 40:30

    Key Takeaways for Viewers

    1. AGAakash Gupta39:25

       a more easy interviewer. Sometimes you'll get more pushy ones. With that, we have other mock interviews on this YouTube channel, AI Product Design, AI Success Metrics, AI Execution. Go check out those videos and subscribe if you'd like more like this, and comment down below what interview mock you would like to see next. Until the next episode, I'll see you later. I hope you enjoyed that episode. If you could take a moment to double-check that you have followed on Apple and Spotify Podcasts, subscribed on YouTube, left a rating or review on Apple or Spotify, and commented on YouTube, all these things will help the algorithm distribute the show to more and more people. As we distribute the show to more people, we can grow the show, improve the quality of the content and the production to get you better insights to stay ahead in your career. Finally, do check out my bundle at bundle.aakashg.com to get access to nine AI products for an entire year for free. This includes Dovetail, Mobbin, Linear, Reforge Build, Descript, and many other amazing tools that will help you as an AI product manager or builder succeed. I'll see you in the next episode.

Episode duration: 40:40