You'll be left Behind as an AI PM If You Don't Use ChatGPT Apps

1. 0:00 – 3:09

   Intro

   1. CMColin Matthews0:00

      ChatGPT apps are a huge opportunity.

   2. AGAakash Gupta0:02

      How would you describe ChatGPT apps?

   3. CMColin Matthews0:04

      You can have this kind of built-in experience where you can interact with an application, uh, directly in your conversation.

   4. AGAakash Gupta0:09

      I brought in my friend Colin Matthews. He is one of my go-to sources for technical topics on product management.

   5. CMColin Matthews0:16

      This is a really underrated way to get more distribution.

   6. AGAakash Gupta0:19

      What's the case for other products, regular people-

   7. CMColin Matthews0:21

      One you can build is like a spreadsheet or, like, a to-do list app that gets pinned to the top.

   8. AGAakash Gupta0:25

      What is the architecture of a ChatGPT app?

   9. CMColin Matthews0:28

      So underlying ChatGPT apps is this protocol called MCP, or Model Context Protocol. I tried podcasting for a bit. I built, like, probably four or five different SaaS apps this year. This is actually my own prototyping tool.

   10. AGAakash Gupta0:40

       Wow.

   11. CMColin Matthews0:41

       I'm using Opus 4.5 for pretty much everything.

   12. AGAakash Gupta0:43

       What might be some interesting examples that we can build live?

   13. CMColin Matthews0:47

       There's kind of two different ways to invoke apps. The first way is to just type out the name. The second way is-

   14. AGAakash Gupta0:53

       'Cause there might have been some news pieces that you read about what the ChatGPT App Store is, but nobody has broke it down in terms of what it means for product builders. So that's what we're gonna do today. Before we get into today's episode, if you can do me a quick favor and check if you have a following on Apple and Spotify Podcasts and subscribed on YouTube, these are free actions you can take that really help the show grow. And if you become an annual subscriber to my newsletter, did you know that you get access to over $28,000 of premium products? That's right, Mobbin, Arise, Relay.app, Dovetail, Linear, Magic Patterns, DeepSky, Reforge Build, and Descript. They are all free for an entire year if you become an annual subscriber to my newsletter. So go take advantage at bundle.aakashg.com. And now into today's episode. Colin, thanks so much for being back on the pod.

   15. CMColin Matthews1:49

       Yeah. Super excited to be here.

   16. AGAakash Gupta1:50

       So what I wanted to do first was take a look at the announcement video for this because this was announced about a month ago at OpenAI's developer conference. Most people probably watched this video, but they've already forgotten about it, so let's refresh your memory. [electronic music]

2. 3:09 – 8:25

   What Are ChatGPT Apps?

   1. AGAakash Gupta3:10

      All right, Colin, so everybody's seen the cool use cases. We saw an Expedia example, a Figma example, Booking.com, all these different apps that are built into it. How would you describe ChatGPT apps? What are they, and what is the ChatGPT App Store?

   2. CMColin Matthews3:24

      Yeah. So ChatGPT apps are basically a way for companies to bring in, uh, their own designs, their own kind of way users should interact with it directly into ChatGPT. So rather than, you know, uh, maybe giving a text summary of something or you're recommending something from a web search, you can have this kind of built-in experience where you can interact with an application directly in your conversation.

   3. AGAakash Gupta3:44

      Why does this really matter? Because I guess it's kind of hard to discover. That's what I feel like when I gave the example of the iOS App Store, for instance. The discoverability was there. It was one of the few apps that people actually got pre-bundled with their new $1,000 phones. They're highly likely to open it up at least once or twice, and then when they open it up, it's showcasing different apps. It feels like ChatGPT apps are kind of hidden. I haven't heard about them.

   4. CMColin Matthews4:10

      Yeah, so I would agree. Uh, at the moment, and again, you know, we're recording this in 2025, so we expect it to change sometime soon, uh, they are kind of hidden. And so there's a couple of companies that partnered, uh, with the initial release, as you saw there. Some other ones coming soon, like Uber is on that list as well, planning to release an app, uh, sometime in the near future. But yeah, they are kind of hidden, uh, and there are definitely plans to bring together a full app store experience, so very similar to what you have in like iOS or Android. You'd be able to browse apps, find ones that you like, and then download them and use them inside of ChatGPT. Uh, there's one other mode of discovery that maybe we'll get into a bit more later, but one thing that ChatGPT promises is that if you put in a request that relates to an app, they might actually decide to sh- kind of service that app to you. So for example, if I say, "Without installing the app, I'm looking for a hotel," I might get the Expedia app kind of surfaced in line even though I didn't install it or ask for it to begin with.

   5. AGAakash Gupta5:01

      Okay. That would be super cool. So they might be building some sort of tool-calling system that automatically figures out, "Okay, here's a reliable app to help service this particular request." And I guess they've already done some of that, like, 'cause I've actually seen nowadays when I do search for a hotel, that they'll pull in some Expedia search results, although they don't always pull in from the app. It seems like they pull in from web search. So kind of how they're pulling in from web search, they'll start to pull in apps.

   6. CMColin Matthews5:25

      Yeah, exactly. Yeah, and again, it gives companies a little bit more control, right? There's a very large kind of, uh, maybe panic around getting your content into ChatGPT, right? 'Cause people know that it converts well. Uh, actually, there was some, some news I saw today that it's like 20 per- 6% increase in conversion when a user comes from an AI source 'cause they have higher intent, right?

   7. AGAakash Gupta5:45

      Yeah, I see that too in my own sites, that the LLM traffic is s- much, much smaller in volume to, like, SEO, but really high conversion rates.

   8. CMColin Matthews5:53

      Yeah, exactly. So companies want to be present, uh, but kind of playing this game of Whac-A-Mole with, like, the ChatGPT web search is hard. And so now you have, from an enterprise perspective, a deterministic way to show up in the application, right? Like, your app is gonna show up, especially if the user has it installed, but even, you know, if they're trying to do something and it's a relevant application, it can show up in their, in their chat. And you'll be able to control what that experience is like. It'll be branded, and you can even pop them back out. Uh, so a good example of this is Target. Target have recently announced that they're building an app, and you can't finish a, uh, like a checkout with them inside of ChatGPT. You can't actually purchase items, but you can build a cart.So you can say like, "Hey, help me find holiday items or Christmas, you know, uh, presents for, for my siblings." It'll build the cart for you, and then you click out, and it brings you into Target to complete the purchase. So I think that's another good example of, like, building this kind of deterministic, catered experience that feels really great inside of ChatGPT, uh, rather than relying on, like, web search to, to do the job.

   9. AGAakash Gupta6:46

      Makes sense. So in that example, at least, I highly grok it, kind of like the Expedia example as, okay, I'm offering some high-ticket service. I might have been getting a lot of traffic from search in many cases, so I need to get into ChatGPT, access its 900 million weekly active users. So I understand that case. What's the case for other products? Regular people like you and me, like, what sort of ChatGPT apps have you been building?

   10. CMColin Matthews7:13

       Yeah. So, uh, very similar to the App Store, you're probably gonna see, like, the eventual Ubers, right? Which didn't start at the very beginning. And then you'll see, like, the apps that are like the flashlight app, right? Like that-- If you remember back to the iOS beginning, it was like, that was an app you could download. Uh, and so I think we'll follow, like, similar ser- similar paradigms here. So for example, like, I've been messing around and building some apps, and one you can build is, like, a spreadsheet. So ChatGPT can, like, use spreadsheets as well as you, and you can kind of collaborate back and forth. It's kind of like what, you know, you'd expect out of the AI experience in Google Sheets, but it doesn't quite work correctly in Google Sheets. Uh, you can build a little spreadsheet app inside of ChatGPT or, like, a to-do list app that gets pinned to the top. Uh, so if you want to complete multiple tasks with ChatGPT, maybe you have like three things you want to do, it can check off those tasks for you so you get, like, a visual indicator. So those are, like, the little utility-type things you could build, and then you could build, like, more fully-featured experiences, right? So things like you saw inside the demo that's there with, like, apps that have maps, uh, navigation, um, search, and, like, integrations with whatever you want on, on the back end. Um, and I guess that's the last thing I'll mention here is that, like, there's actually no strict limitations in terms of what the apps can or can't do. It's just that they're so early that most companies are releasing, like, very, very bare bones versions of them to get into the marketplace. But I think we'll see, like, a lot more complex apps than what exists today.

3. 8:25 – 10:32

   Architecture & How They're Built

   1. AGAakash Gupta8:25

      I love that you've been building these. You've even built a platform to build these, so you really understand the technical details. What do we need to know about how ChatGPT apps are made and built?

   2. CMColin Matthews8:36

      Yeah. So underlying ChatGPT apps is this protocol called MCP or Model Context Protocol. This is invented by Anthropic. It's about a year old. Uh, basically what allows AI agents, so things like ChatGPT and Claude, as well as, like, Gemini, you know, Cursor, really like anywhere you would be talking to an AI. It allows those tools to reach out to other things over the internet, other tools, and use them for whatever purpose. So you can think about, like, web search as an example of this, like a common tool that would be built into a, uh, chat application. Any other tool that you might want to think of, right? So things like booking a stay with Expedia or getting Figma to maybe do some design work for you. Those could also be defined as tools that ChatGPT or Claude or any other, uh, AI chat could call over this protocol of MCP. So a quick diagram here. Uh, this one basically just shows us what it might look like to help book a short-term stay. So as the user, you would say something like that, you know, "I want to book a stay in New York for this time period." And ChatGPT is going to decide first, does this request need an app or would it be beneficial to use an app? And if it does want to use an app, then what are the available tools that we can use in order to, you know, help facilitate this request? And so the first thing it's going to do is actually gonna go ask for the list of tools that are currently available. And you can see that we have two tools available. We have, uh, book a listing and then browse listings. ChatGPT does cache this information, which, uh, basically just means that, like, it holds onto it and it doesn't refresh unless you kind of force it to refresh. But there is this kind of underlying, uh, need to know what tools are available before we actually go ahead and call the tool.

   3. AGAakash Gupta10:03

      If you're enjoying this episode, Colin literally teaches a course on this. The next cohort starts January 30th. It is on Maven. You can use my code to get a special discount off this course. I highly recommend Colin's content and courses. You all seem to love what he's doing, and I personally love reading it. I gained so many epiphanies just out of this one podcast recording, so think about what you could get if you were working with Colin extensively over a live cohort course. Check out his course, and now back on today's episode.

4. 10:32 – 11:24

   Ads

   1. AGAakash Gupta10:32

      Today's episode is brought to you by Vanta. As a founder, you're moving fast toward product market fit, your next round, or your first big enterprise deal. But with AI accelerating how quickly startups build and ship, security expectations are higher earlier than ever. Getting security and compliance right can unlock growth or stall it if you wait too long. With deep integrations and automated workflows built for fast-moving teams, Vanta gets you audit ready fast and keeps you secure with continuous monitoring as your models, infra, and customers evolve. Fast-growing startups like LangChain, Writer, and Cursor trusted Vanta to build a scalable foundation from the start. So go to vanta.com/aakash. That's V-A-N-T-A.com/A-A-K-A-S-H to save one thousand dollars and join over ten thousand ambitious companies already scaling with Vanta.

5. 11:24 – 19:52

   Building First App

   1. CMColin Matthews11:24

      And the last thing is ChatGPT is going to decide which tool to use for this request. So the browse listing one might make sense to start because we need to know what listings are available so we can show that back to the user. And so we could ask for New York on a specific date, and we get back a list of listings, uh, from our MCP server, and then ChatGPT would kind of describe that information. So this is the bare bones version of MCP, right? There's no actual UI or app being involved here. Uh, but you could build this if you wanted to and just, like, have it say, "Here are the top five, uh, short-term rentals in New York for that date," and it would literally just write it out as a description. The addition here on top of MCP is this thing that actually OpenAI kind of invented and is now being, uh, incorporated into the MCP s- MCP spec, uh, which is the idea of like widgets or these little interfaces. So in addition to the raw data, the listings, it can also return a position or a URL for a, uh, widget that we want to return. And so the last thing that ChatGPT is going to say is, "Okay, now that I know that there's a UI element or a widget that goes with this, let me go get that code and then render the code inside the chat." And so that's how we end up with, like, that code that shows up or your app that shows up inside the chat. And then it's still going to respond with something. So it can say, like, "Here are the best options," some small description. Uh, but really we're going to be interacting with the UI that you see here as, like, the main interface rather than the text.

   2. AGAakash Gupta12:41

      Makes sense. So how do we build one of these ourselves?

   3. CMColin Matthews12:45

      Yeah. So there's, uh, let's say the easy way and the hard way. Uh, so as you mentioned, I've been working on a platform to make this a little bit easier. It's called Chippy, and, uh, we can go ahead and take a look at maybe an example really quick before we hop into building it. But basically, what Chippy does is it spins up everything you would need in order to build a ChatGPT app for you. So it spins up an MCPs, MCP server for you, and when you prompt it, it's gonna basically be specialized at building tools. So not full stack web applications, but literally just what you need to build a ChatGPT app. And then there's some nice like UI/UX stuff that I built into it to help you, uh, build the app. So this example here, this is a, a coffee guide. So I kind of wanted something where like I can look at a map and see maybe a good place to get coffee. As you can see on the left-hand side, uh, inside of Chippy, I just asked it to make me a, a quick location guide, and this is what it decided to build. So on the right-hand side, we can see the component that it built. Uh, this is a tool by the way. I'll go through that in one second. But has this little left-hand pane, uh, you know, we can kind of click through on this, and it'll, uh, kind of pull up the right, right side for me, and I can get directions. So this is what it decided to build. The other nice thing about using this inside of Chippy is we can kind of get a preview of what it's gonna look like inside a chat experience. So I can say, "Where should I get coffee?" And there's an LLM working in the background that'll actually call that tool and then throw it into the UI for us. And there we go. So this is actually a full screen UI by default. Uh, we can see that's what that looks like there. And again, we can interact with it in this kind of... You can think of it like a simulated ChatGPT, right? So a quick way to test what you've built to see if you like it or not.

   4. AGAakash Gupta14:11

      Okay. Makes sense. So just to play that back, right, the easier way, what you're doing is you're bundling together an MCP server, which if we recall from the diagram, that's like how ChatGPT is gonna get connected. It's the universal USB-C plug for LLMs to call tools like this. This is a tool, and then it's got the right understanding of what needs to be built to create one of these tools. So it simplifies the tool process to basically just prompting with an LLM. That's the easy way. What's the hard way?

   5. CMColin Matthews14:40

      Yeah. So the hard way would be basically spinning up your own, uh, MCP server. That's the first thing. Getting that hosted on the internet somewhere, and then understanding kind of like how to write the code to build these tool definitions, as well as to build the UI. And there is like kind of something called bundling that has to happen, which is when it translates your UI code into something that ChatGPT can actually understand and render. 'Cause the code that you write doesn't just like get downloaded and rendered in the same format inside of ChatGPT. It has to go through this little process, bundling, and so you'd have to also bundle your c- your kind of UI code. And then the last thing is just understanding what the options are. So like, you know, how to interact with the full screen, uh, version of apps versus maybe like the, you know, picture-in-picture or inline, uh, all that kind of like guidelines that ChatGPT provides or OpenAI provides on how apps should be built. All that stuff is kind of built into this agent. Uh, but if you want to do it the hard way, you kind of have to learn some of that stuff and then host it, build it, and then eventually connect it to ChatGPT would be the last step.

   6. AGAakash Gupta15:32

      Okay. So you'd probably be using like a Cursor, Claude code sort of format versus here you have more of like a U- AI prototyping interface to build that.

   7. CMColin Matthews15:42

      Yeah, exactly. And you know, you can kind of get the experience of testing the tool that you built without going through all the steps of connecting it to ChatGPT. Uh, the reason I built this actually is 'cause I was working on a completely different app for ChatGPT, and it was such a pain to go through the iterations of like every time I wanted to make a small UI tweak, I had to rebundle the code, make the change, go back into ChatGPT, update it, and then see if I liked it. Whereas here I can at least visually see it and then kind of play with it in the UI without having to go through that whole process every single time.

   8. AGAakash Gupta16:09

      Cool. Awesome. So what might be some interesting examples that we can build live?

   9. CMColin Matthews16:14

      Yeah, sure. Um, so obviously we have this one here. Maybe I'll just quickly spin this one up inside of ChatGPT. Uh, so in order to do this, there is one last step, which is just connecting. So if we go up to Test here, we'll get a little URL that's generated for us. I'll copy that, and then we'll head over into, uh, ChatGPT here, and we'll go into our connections in our settings. And you'll see here that I, I mean, I have a bunch of enabled apps 'cause I, I play with these all the time. Uh, so you can see I have a few that are built elsewhere and then a few that are built by myself as well. But basically, the last step here is to click Create, paste in the URL in this MCP URL, uh, field, turn off the authentication unless you really want authentication, and then also give it a name of some kind. So this one we call like, uh, Coffee Map. And I think I already have one, so I'm gonna call this one Coffee Map too. And then finally, uh, click this little button. So it's a little bit involved when you're testing. Obviously, for installing apps, it's a lot easier, like the end kind of consumer experience. Even think about this more like the developer experience, right? Like I'm a developer, I want to build my own app. Those would be the steps that I go through to test it before I release it to everyone else. Obviously, installing like, you know, Canva's app isn't as involved. You just go in, click the button, and click Install, and that's pretty much it.

   10. AGAakash Gupta17:23

       Mm-hmm. Makes sense.

   11. CMColin Matthews17:25

       Yeah. And then we'll give this one a try. So, uh, just to show you, there's kind of two different ways to invoke apps. Uh, the first way is to just type out the name. So if I say Coffee Map, you'll see that it pops up automatically, that it knows that there's an app and I want to use this app. The second way is to actually tag it manually. So if I go into my apps here, I can click Coffee Map, and again, it comes up. And the last way, theoretically, we'll-- we can give it a try afterwards, is if I don't even... If I say something alluded, alluding to it, like, "I want to get a coffee. Where's a good location?" ChatGPT may decide to use my app, and that's a lot of like where the kind of finesse comes in, in terms of getting it to be better, is you want your app to show up on relevant queries.

   12. AGAakash Gupta18:00

       Yeah.

   13. CMColin Matthews18:01

       And so you're gonna have to play with that and, and actually go through like an eval process very similar to, you know, other AI tools.

   14. AGAakash Gupta18:06

       Hmm. It's an eval process. Say a little bit more about that. I guess I was thinking it was almost like another AEO process. Like you need to somehow develop the reputation through queries over time that ChatGPT's feeling like you're a good tool amongst the millions of other tools trying to get called for this query.

   15. CMColin Matthews18:23

       Yeah, that might become the case. Uh, you know, there might even be ads for, for, you know, tools and stuff like that. Uh, but for now, really what it is, is like when you type something in, um, is ChatGPT gonna do a good job of calling your tool? And that's just based on like the very limited set of tools that even exist, right? I mean, there's less than 20 right now, different apps.

   16. AGAakash Gupta18:42

       Okay.

   17. CMColin Matthews18:42

       So-

   18. AGAakash Gupta18:43

       And can anyone, and anyone can get access to publish a public tool?

   19. CMColin Matthews18:47

       Yep. So right now there's no marketplace, uh, where you can kind of publish them publicly. You kind of have to be part of the launch partners, so that's some of these large companies. But, um, in the very near future, then there'll be this kind of public marketplace where you can launch your own apps directly, uh, very similar to what we're doing here.

   20. AGAakash Gupta19:03

       Okay. So right now, if you build one of these, you can't launch it.

   21. CMColin Matthews19:07

       Yeah, correct. Not to the public. I mean, you can always do what I'm doing here, right, which is like give someone a URL that they can, they can play with.

   22. AGAakash Gupta19:13

       Yeah.

   23. CMColin Matthews19:13

       Um, but, uh, yeah, ChatGPT has said or OpenAI said by the end of the year, so we're getting there. It's December now, uh, so we'll see if that comes through or not. But yeah, around the end of the year.

   24. AGAakash Gupta19:22

       So we're, we're basically learning how to build for a platform that's about to become available, and so you're kind of just on the bleeding edge of the distribution of this and making a bet that OpenAI will support it.

   25. CMColin Matthews19:36

       Yeah, exactly. Yeah, you got it.

   26. AGAakash Gupta19:38

       Cool.

   27. CMColin Matthews19:39

       Cool. So yeah, here is our little coffee map. Again, you know, it's a nice little demo application, doesn't do too much. Uh, but yeah, why don't we go ahead and build something new. So we'll flip back over to Chippie here and, uh, yeah. Any, any thoughts on which- what might be interesting?

6. 19:52 – 30:18

   Live Demo: Healthcare App

   1. CMColin Matthews19:52

      What do you want to try?

   2. AGAakash Gupta19:53

      I feel like I want to do something in the healthcare space.

   3. CMColin Matthews19:56

      Okay.

   4. AGAakash Gupta19:56

      Healthcare or legal. I feel like those two spaces are just like infinite value for me on ChatGPT. Doesn't really have anything to do with product management, but I could imagine like, let's say like you're a healthcare product manager, I guess. So if you're a healthcare product manager, let's say you're a healthcare product manager at a hospital, and you want to be able to give access to your customers, like some information about your hospital system through ChatGPT. How would we think about it? What would be a good unit for an app for that product manager?

   5. CMColin Matthews20:27

      Yeah. So like, actually like kind of hospital reviews and surgeon reviews are a really big thing. Um, there's actually like SaaS companies that help, you know, hospitals and, and surgeons manage this because it's related to revenue, obviously.

   6. AGAakash Gupta20:38

      Yeah.

   7. CMColin Matthews20:38

      Like, if you have really g- garbage reviews, uh, you won't have as many customers. So maybe what we'll say is something like, uh, "Build a solution that helps hospitals manage and share their, uh, Google reviews." Something like this. And I'm actually gonna turn on plan mode so that we can kind of see what we get back before we kick it off, just so that, uh, you know, we don't end up building something that's completely unrelated.

   8. AGAakash Gupta21:04

      Cool. So plan mode's gonna give us that thinking, reasoning model that gives us the plan first before it executes.

   9. CMColin Matthews21:11

      Yeah, exactly. And under the hood here, I'm using Opus 4.5 for pretty much everything. Uh, so that's a brand new model that literally just came out like last week. Um, the nice thing about it is it has this effort parameter that you can turn down, so you get a really high-quality model, but you can kind of reduce the amount of time that it spends on a task. And so for things like this, I ha- I'm using a higher quality model, so better thinking, uh, but kind of low effort so that it doesn't spend forever like spinning its own wheels. It gives back the response pretty quickly.

   10. AGAakash Gupta21:37

       Yeah.

   11. CMColin Matthews21:38

       Cool. So here's what we have. It pr- proposed three different tools to build. So one is viewing reviews, one is sharing reviews, and then one is review analytics. Uh, so we'll be able to like see a dashboard of our reviews of some kind, a shareable card that we can share with other people, and then do some, some summary stats and so on. What do you think about that? Does that sound good?

   12. AGAakash Gupta21:58

       Yeah, I love it.

   13. CMColin Matthews21:59

       Okay, cool.

   14. AGAakash Gupta22:00

       This is what I was trying to figure out, like what is the, what is the takeaway for PMs? One thing that's just kind of on my mind, though, is the PM really gonna be building it? The PM's mainly probably gonna create the spec for this. So this could be like they could create their prototype here.

   15. CMColin Matthews22:13

       Yeah, exactly. I think it's a little bit hard to really understand, like if you're a PM, how would you spec this out without ever like using one of these or even like testing how they might work?

   16. AGAakash Gupta22:23

       Yeah.

   17. CMColin Matthews22:23

       So yeah, I think, uh, using this as kind of like a prototyping tool is, is a great use case. And then in the long run, I think that like there's an opportunity for solo builders to also build apps and distribute it. This is the exact same way as the iOS App Store, right? Where like-

   18. AGAakash Gupta22:34

       Yeah

   19. CMColin Matthews22:35

       ... um, they'll be building their own apps. So that's kind of the, the way I'm thinking about this platform is, um, prototyping primarily for PMs and then like for solopreneurs or people who want to build their own apps, that you could do the whole end-to-end of like hosting your application on here as well.

   20. AGAakash Gupta22:49

       Makes sense. So there's a really big opportunity here, I think, for anybody who wants as a PM to create a side project or something like that, a portfolio project to improve their AI PM credentials. But in terms of actually coding up the production version of your ChatGPT app, you're probably not gonna be doing that. You're gonna be creating the prototype here, and then your engineering team is gonna take that, and they're gonna kind of create the real version.

   21. CMColin Matthews23:15

       Yeah, exactly. And actually kind of funny, so a lot of the things that you would do in a normal AI project, as we mentioned, you have to do those here as well. So things like running evals on the prompts that are triggering your tools to make sure that the right kind of phrases are triggering the right tools. And you might have to tweak the tool descriptions a little bit, uh, to try to improve that, you know, this type of request should trigger this tool. Or even like have it where someone writes a request and it doesn't trigger your tools at all because it's not relevant. So you need to kind of go through the very similar process of like what you might choose to do to improve a regular AI application or AI agent, uh, you would do in the same, same case here.

   22. AGAakash Gupta23:51

       Hmm. Makes sense.

   23. CMColin Matthews23:53

       Cool. So it built us our three different tools. Um, again, I'm gonna just for fun share a little bit about behind the scenes here. So you can see it actually viewed some examples of code that's I've built in the, in the background. So this agent that I built, what it does is it can choose to look at relevant files to kind of get inspiration for what it should be doing to build the thing that we've asked it for. And so it take a look-- I took a look at some, like kind of technical stuff, as well as a list example that kind of covers like some UI/UX for lists, and then it decided to build those different tools for us. Um, so here's a little preview, right? I don't know if I love the UX. I'll have to kind of see how it looks like in line, like when it actually has some data in it. It's hard to tell when there's no data in here. Um, but anyway, we have view reviews, we have share, nothing in there, and then we have review. And yeah, so we'll give this a try in a second. Uh, I'm wondering, though, like if this has data that's gonna be passed in by the model or if it needs better mock data. So I'm gonna go ahead and ask that same question. We'll say, "Does this rely on ChatGPT passing in the data?"Or should we have mock data? And the nice thing about this is because the agent has like... You can think about it being like an expert in ChatGPT apps. It can answer questions like this one, right? Where like maybe you're unsure about the best pattern for how this should work. You can just ask the agent. It'll kind of read through, read your code, take a look at, uh, how everything works, and then make a decision for you. So here it's saying that we-- it already has some built-in mock data, and if it's using the correct pattern, that it should be passing in data from ChatGPT, and then fall- falls back to the mock data. Um, so I'm just gonna tell it is that like the mock data is not very good if there's mock data. And so I'm gonna say, uh, "Please improve the mock data," just to get a little bit more, uh, information in here so we can take a look at these components. Cool. And then, yeah, the only other thing I'll mention while we're waiting here, this will just take a second, is you can kind of see that the UX for this building experience is a little bit different, right? So I can see each individual tool. I can also modify the parameters, but these would be the same parameters that ChatGPT, ChatGPT would be using, right? When it decides to call this tool, it's gonna be passing in data for these. So like it would be deciding what the filter is, what the sort is, or any other parameters that are necessary. And so, um, yeah, you can kind of mess around with the prompts directly in here or the, or the parameters in order to get different, like UX or different experiences, uh, that might be rendered inside of ChatGPT, depending what's g- gets passed in.

   24. AGAakash Gupta26:15

       Okay. I'm keen to see this with the real data because then I wanna actually, like you mentioned, like you gave us that teaser that really, uh, really perked up my ears around was the evals for the prompts calling it, 'cause I think that part is really interesting.

   25. CMColin Matthews26:29

       Cool. So we'll hop over into ChatGPT, and then we'll get this hooked up. So first thing we need to do is just go back over to our settings, our apps and connectors, and then create a new connector here or a new app. We'll paste in the URL. Again, we'll turn off authentication for now, just keep things simple, and we'll call this one Healthcare Reviews. Cool, and connect. Uh, so this will just take one second to connect, and then we'll give it a try, and we'll see if it works. And then after that, as mentioned, we'll go back and look at the, kind of the logs. We'll create an eval really quick and see how that performs. Uh, so we're gonna spin up a new chat. You don't have to, but I just kind of don't like to see the old ones, and I'm gonna tag it this time. So we'll say Healthcare Reviews. We'll say, uh, "How are my reviews doing for St. Mercy, uh, Healthcare?" And we'll see what happens here. I'm actually unsure. The, the tool's a little bit janky in that like that one didn't, uh, didn't have any mock data, it didn't look like. But okay. So ChatGPT is actually generating the mock data and then filled it in. We saw it for a second there. Uh, it popped back out for some reason. [laughs] Um, so-

   26. AGAakash Gupta27:36

       Yeah, I swear I saw it.

   27. CMColin Matthews27:38

       Yeah, yeah. So I think what happened maybe is that like there was some underlying mock data that tried to override it there. Um, probably have to do a little bit of iteration on, on this one. But let's see if we can try to call one of the other ones. So let's say like, uh, "I want to share a review," and this should call the other tool, uh, maybe, hopefully, right? So the tool that kind of generates... Yeah, there you go, our reviews here. So we can see our Mercy Hospital. Uh, there's a little problem with the underlying data there. You saw it for a second, and then it disappeared.

   28. AGAakash Gupta28:06

       Got it. Yeah.

   29. CMColin Matthews28:06

       I'll just be able to clean that up. But, uh, but yeah, so you can see the different tool calls kind of in action. And then one last thing I'll show you is if we go back over into the connector here, we can actually see those different tool calls directly inside the connector, right? So we have our review analytics tool, we have our share review tool, and then we have our view reviews tool. So those three different tools that we've set up.

   30. AGAakash Gupta28:25

       Oh, wow. Okay.
7. 30:18 – 33:12

   Ads

   1. CMColin Matthews30:18

      you can do that.

   2. AGAakash Gupta30:19

      Are you looking to land your next product management job? I am accepting a group of just 30 product managers into a 12-week cohort led by me, where every Monday for 90 minutes, I help you through your job search, creating your candidate market fit, updating your LinkedIn, updating your base resume. You're gonna get personalized feedback and one-on-one mentorship sessions with my co-teachers, Ankit Virmani, who is an AI PM at Atlassian and was a group product manager at Meta; Prasad Reddy, who is a CPO and has been in product for over 26 years; as well as my other live instructor, Bart Jaworski, who's gonna run another 90-minute session per week. So if you want coaching from me to land a PM job, this cohort is a no-brainer. It is a premium-priced product. It is more expensive than the average product out there, but the return is huge. Most people who join the cohort see a salary raise anywhere from $10,000 to $100,000 in the first year, and so the ROI will be there within a year. And we guarantee two-plus interviews. So if you don't get two interviews after completing the 12-week program and following all the steps, we will refund the money to you. So it's a no-brainer. Check it out at landpmjob.com. And now back in today's episode.Before we dive deeper, let's talk about something every PM faces: getting alignment on product decisions. You know that feeling when you're trying to explain a user flow to engineering or justify a design choice to leadership and you're just describing it with your hands? That's where Mobbin comes in. Mobbin is the world's largest library of real-world mobile and web app designs from industry-leading apps like Airbnb, Uber, and Pinterest. Instead of spending hours taking screenshots or hunting for inspiration, you can instantly find exactly how successful products handle onboarding, paywalls, checkout flows, whatever you're facing. Over 1.7 million product builders use Mobbin to benchmark against best-in-class products and show their teams proven solutions. Whether you need to convince stakeholders there's a better way to handle user activation or research how top apps approach feature discovery, Mobbin gives you the visual proof to back up your product decisions. Check out mobbin.com/aakash. That's M-O-B-B-I-N.com/A-A-K-A-S-H, and get 20% off your first year. Today's episode is brought to you by Naya.one. In tech buying, speed is survival. How fast you can get a product in front of customers decides if you will win. If it takes you nine months to buy one piece of tech, you're dead in the water. Right now, financial services are under pressure to get AI live. But in a regulated industry, the roadblocks are real. Naya.one changes that. Their air-gapped, cloud-agnostic sandbox lets you find, test, and validate new AI tools much faster, from months to weeks, from stuck to shift. If you're ready to accelerate AI adoption, check out Naya.one at nayaone.com/aakash. That's N-A-Y-A-O-N-E.com/A-A-K-A-S-H.

   3. CMColin Matthews33:11

      And the

8. 33:12 – 40:19

   Improving with Evals

   1. CMColin Matthews33:12

      second thing is kind of start to build up your evals or your golden set. So let's say, for example, that this was correct. We want this prompt to trigger this tool. I can add that to my set of evals, and I just click this button here, and I have three different types of evals, and this comes directly from OpenAI's guidance. So there's a direct, an indirect, and a negative. So direct meaning that the user actually typed in the name of the product. So this would be like, "Canva, can you do X for me?" That'd be a direct request. Indirect would be they typed in something that's relevant to the tool. So what we did here, "I want to share a review," right? That's not naming the tool or naming the application. And then a negative, uh, eval would be where the user types in something completely unrelated like, uh, "I want to go shopping this weekend." And if it called your tool, that'd be a bad thing, right? 'Cause you don't want that to happen. And so in this case, we'd say that this one is an indirect. The user describes the outcome without naming the tool, and we'll go ahead and add that to our evals.

   2. AGAakash Gupta34:00

      Cool.

   3. CMColin Matthews34:01

      And then last thing we have is the actual eval. So now we have one eval that we can run. Uh, we have this one set up as an indirect. And the way that I built this is there's two ways to run the eval. So the first way is if we open it up here, you can run it on auto. So what this does is it literally sends the same prompt and the same set of tools to an LLM and asks the LLM to decide which tool to call, and it can also decide to call no tools. And so this is a very quick way to test a bunch of different prompts at the same time. You can basically run your whole eval set through auto, and you'll see what will happen. And so, for example, this one failed when we passed it back over to GPT-5, and we'll take a look at the reason why. So let's take a look at, uh, cancel out of here. Okay. So when it said, "I want to share a review," we had the expected of view reviews because that's what happened inside of ChatGPT, but this is telling us that the, the r- correct tool to use probably would've been share review, right? Which makes sense, right? So we have these two different ones. One, one is called view, one is called share. Inside of ChatGPT, what happened is it called this one, uh, but what, what we would have wanted is for it to call share review, and that, that's kind of picking up on that issue there for us automatically. And so this is a good example of like, okay, in order to fix this, we have to go back into our tools and probably modify the description of the tools to be more accurate so ChatGPT has a better idea about when to use this view re- reviews tool, uh, 'cause it accidentally used this in the case where the prompt was, "I want to share a review."

   4. AGAakash Gupta35:20

      Yep. So let's-- what does that look like? Maybe we can just look at the full cycle of improving performance-

   5. CMColin Matthews35:25

      Yeah, absolutely

   6. AGAakash Gupta35:26

      ... against the eval.

   7. CMColin Matthews35:26

      So now that we've run the eval, I'll, I'll show you one more thing in here. Uh, so that was an auto eval. The auto evals are like a great way to get a quick kind of directional input, but it's not necessarily gonna match one-to-one with what ChatGPT provides. And so if you wanna actually manually run your evals, you can basically build an eval set and then go through and just type in what, like type in the prompt to ChatGPT, "I wanna share a review," and just log what happened, right? So that's what this is here for, is to just literally go through one at a time and log what happens with each one.

   8. AGAakash Gupta35:53

      Nice.

   9. CMColin Matthews35:54

      Uh, but yeah. So let's say we want to make that change. What we do is we go back into our application, our hospital reviews manager, and we can either prompt our way through this or we can just edit it manually. Uh, so in this case, we're gonna edit manually. I'm gonna go into the config. I'm gonna find view reviews. That's my tool call. And we can see that this is the description that my LLM or my agent decided to write for this, which is "Display hospital reviews with filtering, sorting, and sharing options." And so what likely happened here is because it has the word "sharing" in the tool description, when I said, "I want to share a review," it decided to call this one, uh, by accident. And so we'll just get rid of the word "sharing" just to kind of clean that up. So we'll say, "Filtering and sorting options." And then-

   10. AGAakash Gupta36:32

       So basically improving the metadata, if we think about it, like from like a SEO sort of standpoint, like they type in a keyword, they're using the title and the metadata to match it. Kind of ChatGPT is doing the same thing with these MCP tools it has available to it. So we're trying to give it the right metadata here.

   11. CMColin Matthews36:49

       Yeah, exactly. And these, these descriptions can be pretty verbose. I mean, there are character limits, but you can put in things like examples of how to use the tool, right? Uh, for example, I built a spreadsheet tool before, and it supported formulas, but I needed to tell ChatGPT what those formulas were so that it knew how to use those formulas inside the tool, right? If it was gonna be, you know, writing any data to that spreadsheet. So it's not just about necessarily SEO. It's really just like how should ChatGPT use or behave with, with that tool.

   12. AGAakash Gupta37:15

       Makes sense.

   13. CMColin Matthews37:16

       Yeah. So clean that up a little bit, and then I, I'll probably just add something here like, "This is intended to fetch existing reviews for the purpose of showing... the user. It's probably not the best description, but trying to get more at the idea that like this is not for sharing, this is for retrieving information.

   14. AGAakash Gupta37:40

       Yep, makes sense. Modifying that metadata to just get it called at the right time, and then we're gonna keep iterating on that and that's how we have this end-to-end cycle around evals and we can run those manual evals as you said. So, is there another category of evals then about like how effective the reviews were if we pulled the right reviews? How would we write that category of evals?

   15. CMColin Matthews38:03

       Yeah. So that's less about like did the tool get called based on the prompt, but more around like did the user get the ex- expected result or like was the behavior good.

   16. AGAakash Gupta38:11

       Yeah.

   17. CMColin Matthews38:12

       Um, so very similarly, if we go back over into our observability, we can take a look at the logs and that's really the best way to get a good ide- idea of like what's happening is we can see again what the user requested, what tool got called, and then what some of the data was, uh, that basically got filled in. And using these logs we can kind of get an idea for what happened. Again, you kind of have to have more context on like reading through these logs, what did you want to happen? And it's very same th- the same thing with any type of eval, right? Like you have to have an idea of what ground truth is. Like what do we want to occur when a user types something in? And so again, you, you kind of look at this data to get a feel for what did happen, but separately from that you'll have to decide what you wanted to happen, uh, in, in whatever case.

   18. AGAakash Gupta38:53

       This is kind of expanding my conception around what a good AI prototype is, you know? I think some people might have the tendency to want to like ship the AI prototype when we did right at the beginning like, "All right, two prompts in, we're good to go. Let's ship it. You know, we had our initial prompt, we changed it to add in some dummy data. Good to go. Let's see it." But it seems like actually going through this eval process along the major categories of evals. Here, the major category was like discoverability and then good result. Tweaking it and improving it. This is gonna help you really understand the corner edge cases, like some of the things we used to have in a deeper PRD help you understand what's gonna move the needle in this feature's success or not.

   19. CMColin Matthews39:33

       Yeah, and a lot of these things, honestly, you can't really necessarily predict like how is ChatGPT going to interpret the way that you wrote your tool description, right? And so you could, you could spend a very long time trying to figure out what the best thing is, but really you should just test it and run evals against it and see what works rather than like thinking about it for a long time. And so I totally agree, like getting into the process of this, this type of like iteration provides a lot more information than thinking about it or like, you know, s- writing it in a PRD and then handing off to engineering team, because eventually your team's gonna go through this iteration anyway. It's just a matter of like, are you getting through some of that on your own quickly? Uh, or, you know, you can obviously bring in your counterparts, but you have a, a mechanism to do it quickly versus like the full handoff between teams back and forth, back and forth, right? Uh, which can take like weeks or months or even

9. 40:19 – 52:01

   PM Role & Prototyping Debate

   1. CMColin Matthews40:19

      longer.

   2. AGAakash Gupta40:19

      So I consider you one of the leading experts on AI prototyping, and since we're talking about it, I wanted to bring up this alternative view that I saw from Itamar Gilad a couple months ago. Went pretty viral on LinkedIn, where he talked about, well, what are all the other things a PM could be doing? Researching the market, talking to customers, talking to stakeholders, talking to partners, looking at user and business data, identifying opportunities and threats, setting goals, evaluating ideas. Sometimes I wonder like, are we just endlessly expanding the PM role? What is the right way to think about the prioritization of this work that we've been going over so far versus some of the other work that Itamar has listed here?

   3. CMColin Matthews40:58

      Yeah. So I think to start, just we'll skip on the ChatGPT app side of things and just address this first. I think it's a skill the same way that like a PM who knows how to use Figma is probably more useful in certain contexts, such as like talking to design stakeholders or even like spinning something up really quick, uh, to show to a customer, to someone like that. Like you're not dependent on other people to do every single kind of touch point or element for you. And so I wouldn't say that like using Figma should be an extra line item in here. Using Figma is a skill that supports talking to customers and talking to stakeholders. You know what I mean? So it's not, they're not independent things. It is, yes, these are the responsibilities of a PM, and then they have a way to do that. Like the same way that talking to customers involves some type of skill around interviewing, and they need to learn that skill, right? Talking to stakeholders involves a lot of skill around like stakeholder management and, you know, even managing up and stuff like that. I would say using some of these tools is complementary. So I personally wouldn't advocate for like AI prototyping to be an extra line item or vibe coding to be an extra line item on here. I think these are tools that we can use to support these ideas or these kind of tasks and that, to me at least, it's obvious that if it's very difficult for you to visually communicate something, that that's a great use case for AI prototyping. Like if I want to kind of explain, "Hey, this is h- how I think our AI product should work." Uh, you know, I'm building some type of ag- agent that's gonna do some task. It could be hard for me to kind of explain that to my stakeholders or to my customers. And so spinning up a quick prototype in whatever prototyping tool you like is an easy way for me to start to have that conversation and improve like the kind of the fidelity of the information that I'm sharing. Like, I can be like, "This is kind of what I was thinking. Does this resonate?" Uh, and so that's, that, that's to me like how I would kind of, I guess, have a rebuttal to this, is it's not an extra line item. Vibe coding is not something a PM should do for no purpose. Uh, it has, should be related to like some reason that they're, they're building that prototype.

   4. AGAakash Gupta42:51

      So AI prototyping enhances some of the activities on this list, is the way you should think about it and you shouldn't necessarily think about not doing the stuff on this list. This stuff is important, but how can AI prototyping help you do some of this stuff better?

   5. CMColin Matthews43:06

      Yeah, exactly. And this isn't really new. Like PMs have been trying to brainstorm and communicate ideas forever, right? So like Balsamiq was popular for a long time. It's the same thing, right? It's just like helping if someone who's not a designer communicate something visually to get the idea out of their head and kind of like onto some form of, of paper that people can see. And so yeah, I would say it's, it's literally for the, the exact same purpose. Um, and so I, a- again, if someone's vibe coding first, like if you're a PM and you don't have these other skills and the only thing you know how to do is vibe coding, I don't think that that will be like a, a way to be super successful in the long run. Uh, maybe there's like some short-term gain because it's popular at the moment, but these skills are, are super critical. It's, and vibe coding can support some of those or, or AI prototyping can support some of those.

   6. AGAakash Gupta43:49

      Amazing. So I wanna do some mind mapping together. What are the benefits for creating a ChatGPT app?... for your product. What would you put those major groups as?

   7. CMColin Matthews44:01

      Yeah, I think we'll kind of classify this into two categories. I think there's some benefits from, like, the perspective of learning how to, like, build agents basically or build, build tools that agents are interacting with. So there's kind of like a career benefit or a skills benefit for an individual person, right? A PM, a designer, an engineer. That, that's one kind of classification. I think the main one is enterprise-focused, to be honest. I think the vast majority of early adopters of this is actually gonna be large enterprises, not small companies. And the main thing is, uh, getting clicks or views. It's basically growth, right? So I want people to see my app, uh, see my product, and ChatGPT has hundreds of millions of active users per week, and the intent when a user comes from ChatGPT is higher than the intent when they come in from SEO or another channel. And so every company on Earth, given, like, the proper tools to capitalize on that will, will do so, I think. Uh, and so I think that's really, like, the main benefit is what is the right form factor for us to get in front of customers, get in front of users, and help them interact with our brand, interact with our company, uh, so that we can kind of pull them into our ecosystem.

   8. AGAakash Gupta45:07

      All right. And then the next part of this mind map I wanna understand is who should be building a ChatGPT app? How would you create the major buckets or, like, if I'm a PM, how should I understand if I should be?

   9. CMColin Matthews45:17

      Yeah. So in, in a typical enterprise setting, like we think about the Canva app or, you know, any of the ones that exist today, I would expect it to be, like, a pod, to be honest. So you'll probably have a designer who needs to understand, like, what are the form factors. And it's actually, for a designer, I think an exciting place because it's very unique. You have these kind of like little micro apps that you can build, and each one can do a very small amount of things. You can build more than one if you want to. They can communicate back and forth. Uh, so yeah, understanding the form factor is, is pretty critical. The second to that would be the engineering team, so, like, how do we actually ship this thing? Uh, there's a couple of technical complexities, like authentication is very complicated as compared to regular authentication, so you need to make sure you get that right. And so engineering is gonna figure out, like, how do we actually get this into the world? Uh, you know, how do we support these different types of tool calls that are coming in? And then lastly would be the PM and, like, as a PM, the reason I might choose to do this, like, what is the kind of guiding light? It is growth, right? I would decide, like, is a ChatGPT app a good method for us to drive, you know, higher conversions from AI, basically AI search or A-AI, you know, chat. Uh, and maybe this is a priority that we have that, like, we want to capture more market from, you know, OpenAI, uh, from that type of search. And so the PM would prioritize this as, you know, something that is relevant, and then would also hopefully be involved in this process of, like, building evals, um, you know, shipping small incremental changes to the application, understanding how users are using that application, and then sharing back with anyone who cares about it internally what's happening with that application. So it's really, like, it's kind of its own form factor of software. It's not like it belongs to one persona or group. In my mind, it would be a pod, and then they're gonna ship this together, and each one should have some skill kind of around this new form factor.

   10. AGAakash Gupta47:02

       So if you're a PM deciding whether this is an important opportunity, how do you decide that?

   11. CMColin Matthews47:07

       Yeah. I think for now it is, like, definitely, uh, give it a try, first of all, so that you have some familiarity with, like, what the options are. So building, you know, full-screen applications, how that differs from building, like, just a quick inline card. And then second thing I would say is really, like, pay attention to what other people are doing. Um, so, you know, larger companies like Target and Uber bringing kind of in- coming into this space. When you interact with ChatGPT, like, is it pulling up Target for you? Is it pulling up Uber for you? Is it pulling up Coursera for you? And if it is, you can see it happening, like, in real time, like, what the benefit is of having these apps. Um, and then lastly, the thing I would think about is, like, is this an opportunity to re-engage customers off of your product? So for example, if X percent of your customers are using ChatGPT, they might not be logging into your application like Coursera, but they could be using your kind of micro app, right, and still getting benefit from your product and still feeling like they're connected to your product. And so I think there might be a, uh, kind of a value add here around... You can kind of think about it like retention, right? You'd have to think about how it affects retention a little bit more, but something around that space of consumers or users interacting with your brand and your product without necessarily having to go directly into your app experience.

   12. AGAakash Gupta48:18

       Makes sense. All right. I think I can play with that arrow infinitely. So we got a little bit of this package here of ChatGPT app if you're building it for an existing product here, right, on the right. Now I wanna go to the other side on the left and talk about what are the good ChatGPT app ideas to build if you are a solopreneur or a side project person.

   13. CMColin Matthews48:40

       Yeah. I think, so to start with, I would start thinking about, like, unique ways that ChatGPT can interact with your application. So we saw a brief demo of that here. Uh, you know, we had a little bit of a data issue in the background, but ChatGPT actually can fill in the data for you, right? So it is the one who's deciding what to call your tool with. And so a good example of this, uh, I'll kind of go back to one I, I referred to earlier, is, uh, a spreadsheet application where ChatGPT can kind of partner with me on it. So for example, a spreadsheet app that has financial modeling support. Some, you know, person drops in some, some financial data into ChatGPT and says like, "Hey, uh, can you help me model this?" And it pulls up the spreadsheet app, puts in the relevant formulas, you know, generates some nice charts and graphs, like all that kind of stuff that's deterministic so the user can go back and actually change the data in the spreadsheet and say like, "Oh, you know, you got that number wrong. Let me just quickly fix it." Uh, like, that'd be a small example of, like, a utility or an application that is embedded with ChatGPT. It's not just, like, showing you stuff. It's not a search tool. It actually has the ability to collaborate directly with ChatGPT in some form factor like a spreadsheet or a task list or a whiteboard or whatever, right? Like, you can imagine this ChatGPT that has memory of you and knows you really well and has access to all the tools that you want to use, then you don't need to hop into like Miro or Google Sheets. Like, you can do a lot of work very quickly directly with these embedded applications. So I think there's a lot of, like, uh, potential for these types of embedded apps to, to kind of take over smaller use cases of, like, where ChatGPT kind of falls short right now. Um, but it'd be useful for-ChatGPT to help you with these types of tasks.

   14. AGAakash Gupta50:14

       Okay. So maybe like a domain that ChatGPT is interesting in, like healthcare or legal or productivity or writing, but maybe a use case within that that's neglected.

   15. CMColin Matthews50:25

       Yeah, exactly. Yeah, yeah. And, you know, you can think about, like, any, any example you can find out in the world where there's an AI company building a product for this. So a g- other good example of this is Gamma, right? So Gamma's a very large company building, like, a sp- uh, presentation tool, an AI presentation tool. Theoretically, we could build a ChatGPT app that also makes presentations, right? And so you could, like, provide a really great experience in building presentation or software inside of ChatGPT. You maybe won't be as good as Gamma. I f- I su- expect probably not. But you also don't have to be. You just have to be good enough that, like, it is a good alternative that someone who's already inside ChatGPT goes like, "Ah, yeah, this presentation's a good starting point," right?

   16. AGAakash Gupta51:03

       Yeah.

   17. CMColin Matthews51:03

       And so the, really, like, the strength of the distribution with these kind of embedded applications, um, is, is what I think will kind of win the day there.

   18. AGAakash Gupta51:12

       Yeah. Anything that might benefit from an embedded distribution. What else? It seems like there was a lot of, like, e-commerce examples, right?

   19. CMColin Matthews51:18

       Yeah, I think... Well, I mean, there's a ton of work b- going on right now in terms of, like, shopping, right? So I think it'll be pretty common that people like Target or, you know, uh, you know, other kind of consumer-facing companies want to be in this space if people are searching for products inside of ChatGPT, right? And so they want you to be able to, to build a cart, they want you to be able to check out, uh, because they want to be in front of you the same way that they're in front of you in, on Google or any oth- any other product, like, you know, so on. It'll be interesting to see if Amazon does this, 'cause, like, Amazon has their own LLM activities going on. Uh, but I think Amazon would be an obvious example of, like, can you imagine just hopping in ChatGPT, you say like, "Hey, reorder me the Thursday order," it fills out your cart for you from, like, whatever you ordered last Thursday or whatever you re- currently get, and then you just

10. 52:01 – 54:38

    Ideas for Solo Builders

    1. CMColin Matthews52:01

       buy it. Like, there's a lot of good examples in the e-commerce space, uh, that I think would be consumer friendly.

    2. AGAakash Gupta52:06

       And then we saw Figma and Canva in there, so I guess those are like if you have any sort of media or content creation tool, right?

    3. CMColin Matthews52:13

       Yeah. These are still early days, so, like, I think Canva is the best example from a functionality perspective, so I'd encourage you to, to give it a try. But basically, the idea here is you can use some, like, kind of mini version of the Canva app directly inside of ChatGPT, so it's more fully featured than, you know, just, like, showing you information. You can actually interact with the application, move stuff around, like, like I said, a, a mini version of Canva. Uh, and again, like, you're reliant on ChatGPT to, like, help you do that. So rather than me, you know, clicking through everything or all that, like, I'd use ChatGPT as an agent that understands Canva. And so in some ways, like, we're moving towards a future where ChatGPT is, like, this operating system more. It's like the, the u- universal agent, and then these are all different applications that it can call or use, uh, as needed rather than every company building their own agents.

    4. AGAakash Gupta52:59

       Yep. And all this is built on MCP, so theoretically, if Claude or Gemini win, they could also pull into these or would it-

    5. CMColin Matthews53:07

       Yeah

    6. AGAakash Gupta53:07

       ... what about that?

    7. CMColin Matthews53:08

       Yeah, exactly. So this is the... This is probably the best part, kind of icing on the cake to some degree, is that this started as an OpenAI initiative, this kind of, like, apps inside of, uh, chat. But using MCP, which Anthropic is responsible for, uh, they pretty quickly amended the MCP kind of protocol or standards. So now Claude actually is, like, today working on the same thing. Uh, you can see screenshots of it if you look around on, on, you know, uh, Twitter or LinkedIn of the team sharing, like, how these apps are gonna work inside of Claude. And so you're not just building for one distribution channel. You're actually building for any distribution channel that supports MCP, which right now primarily is Claude in ChatGPT, but Lovable actually supports MCP, uh, Cursor supports MCP. Like, there's a lot of tooling that supports MCP already. Uh, Gemini does not, interestingly enough, uh, but maybe they, they will at some point in time. Maybe it's just not their focus. But, uh, but yeah, I think that, like, there's a potential future here where you can get yourself plugged into multiple different chat applications through one app that you've built on top of MCP.

    8. AGAakash Gupta54:10

       Okay. So it currently isn't in Gemini, so that is one sort of downside here, but it is in Claude.

    9. CMColin Matthews54:16

       They're working on it. It's not released yet. Maybe by the time the podcast goes live it will be.

    10. AGAakash Gupta54:19

        Uh-huh.

    11. CMColin Matthews54:20

        Um, but yeah, there's basically, like, you know, the engineering builds of it. Uh, they're working on it. It has been approved, like, it's part of the MC- MCP spec. It just needs to go through, like, the actual development process at this point in time.

    12. AGAakash Gupta54:30

        Okay. Is there anything else people need to know or add to this mind map to get a good understanding of ChatGPT apps?

    13. CMColin Matthews54:37

        I think this is

11. 54:38 – 1:01:26

    Colin's Solopreneur Year

    1. CMColin Matthews54:38

       pretty, pretty comprehensive. Uh, obviously, like, as we talk today, it's, it's still early days. I think maybe that's, that's the last thing I'll mention is I don't wanna try to hype it up too much. Like, it, it's a cool form factor. It gives you a great experience in terms of, like, testing and building AI apps without having all the infrastructure yourself, so it's a great way to learn. Uh, and but obviously there's only, like, l- less than a dozen companies that, that are currently partnered with ChatGPT, and so I think a lot of this is gonna depend on OpenAI's ability to execute. Like, can they actually get this marketplace over the line? Uh, you know, do people start to use these apps? What does the discovery experience really feel like and look like? Uh, and so yeah, just I guess with a grain of salt, I'm super excited about this space obviously. I think there's a ton of potential, uh, but it does depend on kind of a couple of things getting across the finish line. But I'd say we're, like, 70% to 80% of the way there, and I would guess by March, like, we'll know one way or the other if, if this is the case.

    2. AGAakash Gupta55:29

       So I think this is what you're highlighting, that critical PM skill. Is this an important opportunity? That's what you guys all need to think about for yourself in your unique situation. That is our masterclass on ChatGPT apps for PMs. Hopefully the best guide on YouTube that you have seen yet. Colin, I want to talk a little bit about you because you're-

    3. CMColin Matthews55:47

       Sure

    4. AGAakash Gupta55:48

       ... you're one of the most interesting men in the PM content-

    5. CMColin Matthews55:51

       Appreciate that

    6. AGAakash Gupta55:52

       ... tech space. You're, you just finished a year as a solopreneur, so you were a PM for a long time. We all know you have very highly ranked on the Maven leaderboard, so that could be one thing you did with all your time, I imagine, and you-

    7. CMColin Matthews56:05

       Yeah

    8. AGAakash Gupta56:05

       ... did financially fine. But you didn't really stop there. You did some experiments this year. You launched a podcast. You launched a couple SaaS apps. This obviously, Chippy, you've built. What is, what is the pie chart of Colin's-... time and attention and focus these days?

    9. CMColin Matthews56:19

       Yeah. So as mentioned, one year, uh, almost like maybe a week ago, two weeks ago, uh, which is really exciting 'cause I've actually wanted to work for myself for literally my entire career. Uh, so dream come true for me. But, um, yeah, in terms of time and attention, I would say it's like maybe 40-ish percent keeping, you know, the whole thing running, and I have some help with that, which is great. Like my w- my wife actually helps me a ton with operations. Uh, I have like a TA named Paulo, uh, who helps me with like some of the core stuff, and so I have some support, uh, with that. And then the other like f- you know, 60-ish per- maybe 50% of my time is spent on like new bets. And when I think about bets now, it's actually changed over time. So partially it's like, is this gonna work commercially? And then the second part is like, am, do I actually wanna do it? And sometimes I don't know if I wanna do it until I try. So podcast is a good example of that. Like, I tried podcasting for a bit. I think I was creating interesting things, but like the thing I think about is, am I gonna be like in the top, you know, 5 to 10% of this thing? And if I'm not, I kind of drop it and I think for, of something else that I wanna do. And so for podcasting, I don't think I'm gonna be in the top 5 or 10%. Maybe I'll come back to it one day. Um, but for now, it's not really like a bet that I'm, that I have the same amount of conviction about. And so yeah, I tried podcasting for a bit. I've built like probably four or five different SaaS apps this year. Uh, this is actually my, my, my second AI prototyping tool that I've built, like my own prototyping tool. Uh, this one obviously catered to the, to this use case, but I built one earlier in the year as well. Uh, a lot of learning, like obviously knowing how to build these tools takes a little bit of time, so like learning how to build agents, learning how to like run these systems. And, uh, yeah, like, I don't know. Testing other stuff, to be honest. Like, I spent time this year on, uh, a RAG app for a little while that I built from scratch. Um, what else? I don't know, a bunch of different stuff. I just try different stuff. Obviously, I have a Substack. Um, so it's been a little bit scattered, to be honest, which I don't like, but it is nice to try different things, fail quickly, and then move on, um, rather than just like doing the one thing forever. Uh, and then I think, j- I know you didn't ask this, but my, my kind of end goal is to have a software product and continue teaching and kind of like balance those two things. Um, but I'd love to have like a software product that's, that's super cool, that's valuable, uh, that people wanna use. So that's, that's what I'm trying to shoot for at this time.

    10. AGAakash Gupta58:32

        Got it. So you're like a Mark Luyuan at, plus course instructor.

    11. CMColin Matthews58:37

        Sure, something like that, yeah. I just, I just mess around with stuff a lot. I write about stuff when I find it interesting mostly, and then that's pretty much it. I do have to get better at marketing. This is an aside, but just being transparent. Uh, that's like one muscle that in the next year I'm hoping that I, I improve on, 'cause my marketing activities are very haphazard at the moment. Uh, and so I need to get more consistent at, at marketing and just showing up so people know I exist. But, but yeah.

    12. AGAakash Gupta58:59

        And what's your stack for building these SaaS apps? How'd you build, um, the app we just saw today?

    13. CMColin Matthews59:04

        Yeah. Actually, this might be really interesting for you all. Uh, I built the UX entirely on Replit. So like, uh, just the Gemini 3, uh, the new s- the design mode they re-released recently. Uh, but in terms of like actually shipping stuff, so I use VS Code and Cloud Code for the vast majority of like code gen. I'm obviously using Git, uh, for like version control. I use a database provider called Neon. Um, I use a hosting platform called Render, and then I have lots of different libraries depending on what I'm doing, right? So like for the RAG app that I built, RAG is a whole world on its own that's kind of complicated and hard to optimize. And so like I was using this, um, this vendor called Voyage for the embedding models and different, different vendors for different things. So like you end up with this whole stack of like random stuff that you learn about and try to build and then, you know, maybe it works, maybe it doesn't. But anyway, uh, my core tools are like Cloud Code, VS Code, GitHub, and Render for, for building stuff.

    14. AGAakash Gupta59:57

        Why VS Code and not Cursor?

    15. CMColin Matthews59:59

        Yeah. So I don't use Cursor. I use Cloud Code predominantly. I use the, uh, Codex tool as well sometimes. And I just find like the integration inside of VS Code's a little bit nicer for those tools than it is inside of Cursor. Uh, Cursor has its own AI obviously, and it tries to like use that AI, and I don't want to. I want to use Cloud Code or Codex. And so yeah, I don't, I don't really use Cursor. I find Cursor at times kind of like tempts me back 'cause they release new features. So for example, they, they release embedded websites, so like you can interact with your webs- your, the, whatever you're building inside of Cursor, and the AI has some context on it, can actually debug for you. But more often than not, like those things don't really move the needle for me. What really moves the needle is like quality of code gen. That's like 95% of what I need and care about, and all the other stuff is just bells and whistles. And so right now for me, Cloud Code is like the highest quality code gen, uh, at, with the fastest pace. And so I just, that's like my, my daily driver.

    16. AGAakash Gupta1:00:48

        Fascinating stuff, man. We're gonna have to have you back again. Maybe you can show us this stack of how people can build stuff. I'm sure there's a million different other episode ideas we could come up with. You guys leave a comment below. Should we have Colin back on for a third episode? By the way, for those who don't know, he used to have our number one episode of all time when he did our top five AI prototyping tools. Now, since then, we've managed to release some episodes better, thank God for us as a podcasting team. But hopefully, we can break the records with this one. Drop a comment below what you liked about this episode, whether we should have Colin back on. Colin, thank you so much for dropping all this sauce.

    17. CMColin Matthews1:01:23

        Yeah. Yeah, happy to be here.

    18. AGAakash Gupta1:01:24

        All right, everyone, see you later.

12. 1:01:26 – 1:02:12

    Outro

    1. AGAakash Gupta1:01:26

       I hope you enjoyed that episode. If you could take a moment to double-check that you have followed on Apple and Spotify Podcasts, subscribed on YouTube, left a rating or review on Apple or Spotify, and commented on YouTube, all these things will help the algorithm distribute the show to more and more people. As we distribute the show to more people, we can grow the show, improve the quality of the content and the production to get you better insights to stay ahead in your career. Finally, do check out my bundle at bundle.aakashg.com to get access to nine AI products for an entire year for free. This includes Dovetail, Mobbin, Linear, Reforge, Build, Descript, and many other amazing tools that will help you as an AI product manager or builder succeed. I'll see you in the next episode.

Episode duration: 1:02:22