Skip to content
Aakash GuptaAakash Gupta

You'll be left Behind as an AI PM If You Don't Use ChatGPT Apps

Colin Matthews is back with the definitive guide to ChatGPT apps. MCP protocol explained, live app building demo, and eval strategies. Plus: why every PM should learn this new distribution channel. Full Writeup: https://www.news.aakashg.com/p/colin-matthews-podcast Transcript: https://www.aakashg.com/chatgpt-apps-guide-colin-matthews/ Chippy: https://chippy.build/ ---- Timestamps: 0:00 - Intro 3:09 - What Are ChatGPT Apps? 8:25 - Architecture & How They're Built 10:32 - Ads 11:24 - Building First App 19:52 - Live Demo: Healthcare App 30:18 - Ads 33:12 - Improving with Evals 40:19 - PM Role & Prototyping Debate 52:01 - Ideas for Solo Builders 54:38 - Colin's Solopreneur Year 1:01:26 - Outro ---- 🏆 Thanks to our sponsors: 1. Colin's ChatGPT Apps Course: Next cohort starts February 13th on Maven - https://bit.ly/4qd2ryx 2. Vanta: Automate compliance, Get $1,000 with my link: https://www.vanta.com/lp/demo-1k?utm_campaign=1k_offer&utm_source=product-growth&utm_medium=podcast 3. Land PM Job: 12-week experience to master getting a PM job - https://www.landpmjob.com/ 4. Naya One: Accelerate AI adoption in financial services - https://nayaone.com/ 5. Mobbin: The world's largest mobile & web design library - Get 20% off: https://mobbin.com/?via=aakash ---- Key Takeaways: 1. ChatGPT apps = MCP + widgets - The Model Context Protocol (invented by Anthropic) lets AI agents call external tools. OpenAI added UI widgets on top to create embedded app experiences directly in chat. 2. 900M weekly active users = massive distribution opportunity - This is the new SEO. Early data shows 26% higher conversion from AI traffic vs traditional search. Every enterprise will eventually build here. 3. You're building for multiple platforms - MCP works across ChatGPT, Claude (coming soon), Cursor, and other AI tools. Build once, distribute everywhere. Gemini doesn't support it yet. 4. Apps get called based on tool descriptions - Your metadata matters. Like SEO but for LLMs. Run evals to test if correct prompts trigger your tools. Iterate on descriptions to improve discovery. 5. Three eval categories: direct, indirect, negative - Direct: user names your app. Indirect: user describes outcome. Negative: irrelevant request shouldn't trigger your tool. Test all three systematically. 6. PMs should prototype but engineers ship production - Use tools like Chippy to prototype quickly and test concepts. Show stakeholders real interactions. Engineering team builds the production version. 7. Enterprise-first, solo builders second - Large companies (Target, Uber, Canva) are early adopters chasing distribution. But huge opportunity for indie builders once public marketplace launches. 8. Best opportunities: embedded collaboration tools - Spreadsheets, task lists, whiteboards where ChatGPT can partner with you. Not just search results—actual interactive experiences. 9. Error analysis on observability logs is critical - Track what prompts triggered which tools with what parameters. Look for mismatches between expected and actual behavior. Iterate tool descriptions. 10. Marketplace launching by end of 2024/early 2025 - Currently only launch partners can publish. Public marketplace coming soon means anyone can ship apps and reach ChatGPT's massive user base. ---- 👨‍💻 Where to find Colin Matthews: LinkedIn: https://www.linkedin.com/in/colinmatthews-pm/?originalSubdomain=ca Newsletter: https://blog.techforproduct.com/ 👨‍💻 Where to find Aakash: Twitter: https://www.x.com/aakashg0 LinkedIn: https://www.linkedin.com/in/aagupta/ Newsletter: https://www.news.aakashg.com #chatgptapps #aipm ---- 🧠 About Product Growth: The world's largest podcast focused solely on product + growth, with over 200K+ listeners. 🔔 Subscribe and turn on notifications to get more videos like this.

Colin MatthewsguestAakash Guptahost
Jan 21, 20261h 2mWatch on YouTube ↗

At a glance

WHAT IT’S REALLY ABOUT

ChatGPT apps: MCP-powered embedded tools PMs must learn now

  1. ChatGPT apps embed interactive, branded mini-app UIs directly inside a chat, giving companies more deterministic control than relying on AI web search links alone.
  2. The underlying architecture uses MCP (Model Context Protocol) for tool calling plus UI “widgets,” enabling ChatGPT to fetch tool schemas, invoke functions with parameters, and render app interfaces in-chat.
  3. Colin demonstrates building a healthcare reviews manager app rapidly with his tool Chippy, then connecting it to ChatGPT via an MCP URL to test real tool invocation behavior.
  4. The episode emphasizes evaluation workflows—direct, indirect, and negative trigger tests—to improve discoverability and correctness by iterating on tool descriptions and metadata.
  5. They argue AI prototyping is a complementary PM skill (like basic Figma fluency) that accelerates stakeholder alignment and learning rather than replacing core PM responsibilities.

IDEAS WORTH REMEMBERING

5 ideas

ChatGPT apps are a new distribution channel with higher-intent traffic.

They can be surfaced inline during relevant user queries, and companies can deliver branded experiences that may convert better than traditional SEO traffic.

MCP is the core “USB-C” connector for agent-to-tool interoperability.

ChatGPT (or other agents) requests available tool definitions, calls the appropriate tool with parameters, and uses returned data/UI instructions to render an embedded experience.

UI widgets turn tool-calling from text output into interactive software.

Instead of only returning raw data, apps can return a widget/UI reference so users interact visually (maps, lists, dashboards) inside chat.

Prototyping speed matters because iteration friction kills learning.

Colin built Chippy to avoid the slow loop of rebundling UI, redeploying, reconnecting, and retesting inside ChatGPT for every small change.

Tool descriptions are product-critical metadata, not documentation fluff.

ChatGPT’s tool selection can hinge on wording; a single word like “sharing” in a “view reviews” description caused the wrong tool to be invoked.

WORDS WORTH SAVING

5 quotes

ChatGPT apps are basically a way for companies to bring in their own designs... directly into ChatGPT.

— Colin Matthews

Underlying ChatGPT apps is this protocol called MCP, or Model Context Protocol.

— Colin Matthews

This is a really underrated way to get more distribution.

— Colin Matthews

Theoretically… if I say something alluding to it… ChatGPT may decide to use my app.

— Colin Matthews

You can’t really necessarily predict how ChatGPT is going to interpret the way that you wrote your tool description… you should just test it and run evals.

— Colin Matthews

What ChatGPT apps are and why they matterDiscovery and distribution mechanics (current and future app store)MCP architecture: tools, schemas, caching, and widget UIsBuilding: easy path (Chippy) vs hard path (DIY MCP server + bundling)Invoking apps: named, tagged, and auto-surfaced tool callingEvals and observability: direct/indirect/negative, logs, iteration loopPM role, enterprise adoption, and solopreneur opportunitiesCross-platform potential: MCP across ChatGPT, Claude, Cursor, etc.Risks and platform dependency on OpenAI execution

High quality AI-generated summary created from speaker-labeled transcript.

Get more out of YouTube videos.

High quality summaries for YouTube videos. Accurate transcripts to search & find moments. Powered by ChatGPT & Claude AI.

Add to Chrome