Inside Figma’s PM OS: skills, agents, and Auto XP

1. 0:00 – 0:51

   Episode preview

   1. EMEzra Mechaber0:00

      I, as a growth PM, I feel like the biggest sin you can commit is wasting traffic. What if we could just automate myself out of a job and, like, create a little growth PM who could just, like, run and run and run? I call it AutoXP, and right now one of our engineers is running it and has, like, five different experiments going that I have not taken a look at, and it found a huge win on its very first experiment. AutoXP has its own channel where it can communicate with itself. "Hey, AutoXP, go find an experiment to run on our settings page. See what you can do. Go crazy." And it will trigger a data scientist to go find drop-off and, like, look at the funnel and look at mix shift and all of those things.

   2. CVClaire Vo0:37

      The whole hypothesis was, like, how do you exploit all this traffic and maximize the number of experiments, and then, like, the time to be alive. It is here. You can just, like, five code it there. This is amazing. [upbeat music]

2. 0:51 – 2:04

   Introduction to Ezra Mechaber

   1. CVClaire Vo0:51

      Welcome back to How I AI. I'm Claire Vo, product leader and AI obsessive, here on a mission to help you build better with these new tools. Today, I have Ezra Mechaber, product manager at Figma, and he's gonna show us Figma's internal PM OS, the 31 skills that every product manager at Figma uses to make their life a little bit easier. We're also gonna talk about, of course, my boy Jev, and how you can build a real-time AI editor with that little cheap, fast model. Let's get to it. Optimizely Agent Platform for Marketers is built on a bigger idea, a whole directory of agents now joined by a newly revealed squad of virtual teammates, each with a defined role and a personality of their own. This isn't about faster drafts. It's about handling the coordination, too, the research, the approvals, the handoffs, the routing, the back and forth that quietly eats your week and never makes it into anyone's highlight reel. Get the story behind it all. Hear the confessions from marketers who have handed off the busy work. Meet the squad, and understand what virtual teammates can do for you at optimizely.com/howiai.

3. 2:04 – 6:30

   The origin story of Figma’s PM OS

   1. CVClaire Vo2:05

      Ezra, welcome to How I AI.

   2. EMEzra Mechaber2:08

      Thanks so much.

   3. CVClaire Vo2:09

      One of the things I love about this podcast in particular, 'cause I know we have a lot of crossover listeners to Lenny's podcast, is PMs are really leaning into AI, but they're all doing it in completely different ways, right? I started like a jillion AI years ago with, like, ChatPRD and, like, how do we make documentation and go really fast. We've seen so many different product flows. I think everybody's sort of crafting their, like, artisanal product platform now because they can. And so when you said you had a PM OS, I was just so excited to have you come on and demo it. So tell me, what was the origin story of what we're gonna see today, and why did you build it?

   4. EMEzra Mechaber2:50

      I like to tell my engineers that I... my goal is to be lazy in the right places. And so for me, a lot of it was I walked into Figma a couple years ago, and I thought everyone was gonna be using AI. It was gonna be crazy AI pilled. Um, I was just gonna... I was already feeling behind. Like, I just, I felt behind 'cause we're all on Twitter, and we all see, like, you know, Tebow just posting about-

   5. CVClaire Vo3:14

      [chuckles]

   6. EMEzra Mechaber3:14

      ... like, the future of the universe. And I walked in feeling really behind, and I realized basically no one was using these tools outside of engineering to, like, get real stuff done. And so I started just trying to automate some of my own productivity. Like, I was the person who was trying to do too much in Zapier, like, many years ago. And the more time I spent with it, one, Claude Code had just come out, and I started to realize, like, there were things I could do in the terminal that were, like, really interesting. All of a sudden, I could, like, build CLI. You know, we were in that CLI phase. And, like, no other PMs who I was working with were doing it, and they were all sort of curious. And we had this really interesting challenge from Yuki, who was our CPO at the time. She was basically like, "What would it take to get all of our PMs kinda AI pilled? Like, what would that look like?" And, um, working with a couple of other PMs, um, shout out to Jen and, and Lawrence, started thinking about, like, what would it look like to turn this thing that I was doing, just, like, hunched over my laptop like a little creep, into this kind of, like, reusable thing that lots of other PMs could, could build on top of.

   7. CVClaire Vo4:26

      Great. And so what, what were, like, the problems that you thought were most important for PMs to think they could solve with AI?

   8. EMEzra Mechaber4:35

      First of all, I feel like the, the, like, really trite thing about being a PM is that you're doing too many jobs, and the question is, like, which jobs do you either do really poorly or not do at all? And so for me, it was like, okay, how do I do... For the jobs I'm not doing and the jobs we are not doing as a team, how do we do them at, like, the minimum allowable level of, like, missing things? Like, my... I, our support team is incredible, and the thing that I forget to do on every launch is to loop them in at the right time. And so one of my questions was, like, how could I do this to make sure that I don't forget Katie, one of our support ops people? Or meeting notes, not my friend. I'm really bad at them, but, like, we have transcription tools. How do I just get, like, good enough so that I'm not blocking other people? So a lot of them started from my jobs of what I wanted to do, find someone in the org who works on such and such thing. We're like 2,000 people now, so just understanding who the code owner is is really useful, um, and understand who we are in relationship to each other. And then also just, like, how do I get fun productivity out of all of our other PMs? So, um, one of the skills that's in there was this, like, really detailed spec skill that started to walk through how does our code base work if you wanted to do this really intensely. Like, if you wanted to do this the right way, what would that look like?

   9. CVClaire Vo5:57

      Amazing. And so I, I love this idea that you had a couple tasks on the PM plate that you were like, "We need minimum viable PM- ... doing of these tasks. Like, it just needs to-

   10. EMEzra Mechaber6:10

       Yeah, just like making not bad at it.

   11. CVClaire Vo6:11

       Yeah, and I talk a lot about you have to, like, both raise the floor, like we have to do just a little bit better on our baseline, and take off the ceiling. And I think-

   12. EMEzra Mechaber6:19

       Yeah

   13. CVClaire Vo6:20

       ... AI can help you do both. And often you have, like, one set of people taking off the ceiling and saying, "Hey, everybody else, like, take off the ceiling," instead of like-

   14. EMEzra Mechaber6:29

       Yeah

   15. CVClaire Vo6:29

       ... "This is how I can help

4. 6:30 – 10:28

   PM OS demo: the “chief of staff” setup

   1. CVClaire Vo6:30

      you raise the floor." Okay. So-

   2. EMEzra Mechaber6:32

      Yeah

   3. CVClaire Vo6:32

      ... show us what PM OS is, its skills, what it does. How are you relying on this day-to-day?

   4. EMEzra Mechaber6:40

      Cool. Okay. So this is the rough structure. It started just as, like, a really simple repo inside of Claude Code as a TUI. So it's got this, you're, uh, you're the PM's chief of staff, you know their work. I hate the term chief of staff, but like-

   5. CVClaire Vo6:57

      [laughs]

   6. EMEzra Mechaber6:57

      ... it, it's, to me, it's really useful for like teaching.

   7. CVClaire Vo7:00

      It's good shorthand.

   8. EMEzra Mechaber7:00

      It's really good shorthand, um, which makes me die a little bit every time I read this. Um, when the session kicks off, it reads some information about, like, who the, um, who the PM is, who they work with. It's got a memory file. And then it's got a couple of different skills that we can walk through. So one of them was just, like, we had no good data on, like, which set of people work on X thing. Like, we have several different org chart tools, and, like, none of them are stitched together in useful ways. And so one of them is just, like, literally who works on this team, who owns X, and it calls it when it writes docs. So if you need review on a doc, you, the PM, do not need to remember to say, "My p- my engineering manager is Aman." It just adds all of the relevant people as, like Charlotte is my designer, and Aman is my engineering manager, and it adds them as the relevant people and remembers who your skips are. So that's skill number one, which was just, like, a nice way of reducing some of the, like, repetitive work, uh, inside of new docs. I use it a lot as, like, a meta Slack. So our S- our Slack is insane. I feel like anyone who's in a large enough org in tech, just, like, Slack becomes really unruly. I archive roughly 10 channels a day. It's bananas. Um, and so one of the things that we were really focused on, uh, was how do you help PMs know where to focus and what to do, and, like, go run down all the things that are important. And so I run this skill what's up, like, six times a day. It pulls all of my relevant Slacks and projects together. It pulls all of the things that I like, who I owe stuff to, what decisions are outstanding. I have a version of this that runs through all of my live experiments on the platform and, like, gives me a s- not that I peek at my, you know-

   9. CVClaire Vo8:52

      [laughs]

   10. EMEzra Mechaber8:52

       Data scientists, stop listening.

   11. CVClaire Vo8:53

       Wait, hold on.

   12. EMEzra Mechaber8:54

       Like, not that I peek at my experiments.

   13. CVClaire Vo8:56

       Say it. Say directional. Say directional, I dare you.

   14. EMEzra Mechaber8:58

       Right. Right.

   15. CVClaire Vo8:59

       [laughs]

   16. EMEzra Mechaber9:01

       Um, and so, uh, I, I find this thing to be incredibly useful, and let's see if we can find anything. So it will look at, like, certain channels that we've identified as useful, it will read the whole thread. This is probably a little overspecified. Like, I feel like we're all moving towards, like, the models are just good enough now.

   17. CVClaire Vo9:19

       Mm-hmm.

   18. EMEzra Mechaber9:20

       But there's a lot of stuff in here about how to find exactly the right stuff for the way that we are all set up in Figma, um, which is just enormously helpful for how I run my day-to-day as someone who's just in way too many meetings and trying to do too many projects.

   19. CVClaire Vo9:36

       What I like about what you're showing, because so many of these, like, PM OS's that we've seen skip all this, like, foundational piece, which is i- in large organization, it is actually really hard to know, like, who should have an opinion on this product? Is somebody else working on it?

   20. EMEzra Mechaber9:51

       Yeah.

   21. CVClaire Vo9:51

       And so I like that you invested in sort of, like, the baseline context gathering as skills, and then I'm presuming that helps, like, l- level up some of the more, like, complex or strategic workflows that come out of PM OS.

   22. EMEzra Mechaber10:06

       Yeah. It's been a long ti- I've, like, tried to build a benchmark sort of unsuccessfully, and I'm curious how you-

   23. CVClaire Vo10:12

       Mm

   24. EMEzra Mechaber10:12

       ... think about it for-

   25. CVClaire Vo10:13

       Yeah

   26. EMEzra Mechaber10:13

       ... all the stuff you're doing now. But, like, I would be really curious how this performs relative to, like, totally vanilla Codex these days.

   27. CVClaire Vo10:20

       Yep. Yep.

   28. EMEzra Mechaber10:20

       Um, but I do think, like, in general, especially in that early, like, I don't know, like, last October through, like,

5. 10:28 – 12:00

   How the repo works: skills, hooks, and private sub-repos

   1. EMEzra Mechaber10:28

      May, as, like, the models were getting better but were not, like, super powered yet, felt like it was really, really helpful for how it navigated all the weird parts of, like, Figma's IA.

   2. CVClaire Vo10:38

      Yeah. And one thing, again, we actually didn't specify this for folks, but product team OS is a repo, right? So, like-

   3. EMEzra Mechaber10:48

      Yes

   4. CVClaire Vo10:48

      ... take a step back and tell us, like, what is it? Is it a repo of skills?

   5. EMEzra Mechaber10:53

      Yeah.

   6. CVClaire Vo10:53

      Is there more to it? Like, what is actually the thing?

   7. EMEzra Mechaber10:56

      Yeah. So it's a really, really basic repo. Um, and so it's got an agents.md, which just tells it how to behave. It's got a bunch of related skills that tell it when to fire and what to do. Uh, it's got a couple of command hooks, um, so things like on boot, go find out if there's a new version of PM OS, because we wanted a good way to, like, keep our 60 to 70 PMs kind of roughly in sync. Um, we've also got a couple of built-in folder structures. So the base template when you onboard is this, like, it's a dot folder, so it's invisible and you can't mess with the context, but then it duplicates and it sort of builds your version out. And then it's technically two Git repos. It's the parent repo that everyone can download, and then their workspace is a sub-repo, so that if you wanna write stuff back, you can write it back privately. Um-

   8. CVClaire Vo11:52

      Amazing

   9. EMEzra Mechaber11:52

      ... again, 'cause, like, none of this was in a UI at the time, and so we were all just trying to figure out ways to, like, make GitHub work for PMs who had never touched GitHub before.

6. 12:00 – 14:15

   How the skills got built, roasted, and whittled down

   1. CVClaire Vo12:00

      And there, was there j- a small group that sort of, like, brainstormed what these initial skills should be? Was it like, "Hey, everybody, just put in-

   2. EMEzra Mechaber12:08

      Yeah

   3. CVClaire Vo12:08

      ... your favorites"? How did that work?

   4. EMEzra Mechaber12:10

      Yeah. So Jenning and Lawrence I think talked about this a lot at Config this year, 'cause they did a, a little presentation on this. But basically, we, we had a really small Slack channel of just the three of us hacking away and firing off as many PRs as we could for a little bit. We had that sort of like Cambrian explosion of like a thousand skills-

   5. CVClaire Vo12:29

      Mm-hmm

   6. EMEzra Mechaber12:29

      ... and then started to like whittle stuff down. And then we did an offsite with every PM at Figma this spring, where we had everyone install it, use it, hack on it, come up with like what was good, what was bad, just like roast it. Um, and so some of the skills that are in here came out of that. Some of the skills either got whittled down or whittled away.

   7. CVClaire Vo12:48

      Mm.

   8. EMEzra Mechaber12:48

      And I would say as the models have gotten better, I've been trying to basically take stuff out that's no longer useful.

   9. CVClaire Vo12:53

      That's, that's great. And so I have a question. Do you have, as a team, any telemetry on this, like how it's being used? How-

   10. EMEzra Mechaber13:02

       Yeah

   11. CVClaire Vo13:02

       ... is it instrumented?

   12. EMEzra Mechaber13:04

       No. I thought about doing this, and I d- I decided, one, I like didn't want anyone to feel creeped, and-

   13. CVClaire Vo13:10

       [laughs]

   14. EMEzra Mechaber13:11

       ... uh, and like I wanted it to be fun, and I also just didn't have the time for it, 'cause all of this was, you know, like sort of 20% time. Um, the biggest thing for us was just we, like PMs at Slack are chronically online, and so we got a lot of feedback. And then we had a public channel, and so we started getting like, all of the engineering managers started using it and asking us if we could like come help them talk about how to do knowledge work for engineering. Um, our data scientists have one that is like essentially a fork of this. Um, and so we were getting a lot of feedback about what it was good at and what it wasn't good at. And I think for us, the, the big next thing, and we can talk about the evolution into what it is today, if that's helpful, but like mostly for us it was we managed to get a bunch of PMs comfortable in the terminal, but like surely that's like not the right end state. Like, especially if you're doing a lot of writing, it like sucks to read monospace font all the time. And so for us, like that was where we wanted to spend our time is like, okay, what, what would be next? How do we get people out of the terminal now that we're, we've got them comfortable with the concept?

7. 14:15 – 17:33

   Build artifact, FigJam MCP, and coaching skills

   1. CVClaire Vo14:15

      Well, before we go to what's next, can you walk us through like just a couple more of your favorite skills in here, maybe ones that teams haven't thought of or you think are particularly unique or kind of helpful?

   2. EMEzra Mechaber14:26

      Yeah, sure. So I do think for us, like with a busy Slack, the, the kind of like meta what's up that is tuned to just like literally what do you have to do every day and how do you judge-

   3. CVClaire Vo14:35

      Mm-hmm

   4. EMEzra Mechaber14:35

      ... whether you're doing a good job is really useful. Build artifact is, um, one of my favorites at the moment, which is if I have something like, actually this, um, this demo like dashboard that I made for this podcast is just like a really simple HTML artifact, and then we have the ability to publish it. We've got a like little hosting tool that we use internally. And so it takes it through... It's got, um, in this case it has a template HTML, and I've got a couple... I don't know where they're sitting. I've like totally lost track of my folder structure at this point. But, um, there's like a couple of different templates of little CSS that I like and I've sort of grabbed from different places. Um, and so it builds stuff like this when I wanna show stuff off. FigJam, the MCP for Figma is insane. I don't know if you get to use it on a regular basis, but I feel like I use the MCP all the time, and one of our PMs built this amazing FigJam builder that produces these like wild workshop templates very quickly. And so just figuring out like how to load it up with, like what do you think looks good or how do you think this should work? And so there's a lot of taste in here on how it's designed. There's, you can also tell there's like, we've got a lot of Claude slop in here from earlier, right? Like the-

   5. CVClaire Vo15:55

      I was gonna say, that is such a Claude slop sentence right there.

   6. EMEzra Mechaber15:59

      Yep. Um, but it's a mix. Like there's a lot of stuff in here that like if you look at this output compared to some of the FigJams that we would put together natively-

   7. CVClaire Vo16:07

      Yeah

   8. EMEzra Mechaber16:07

      ... internally, like it's, it's just a really good fit over time. And then maybe my favorite other one is coaching, which I use, um, as someone who used to manage people and is now IC'ing again. Like I spend a lot of time-

   9. CVClaire Vo16:24

      Living the life.

   10. EMEzra Mechaber16:25

       Oh, it's so nice. It's so good.

   11. CVClaire Vo16:28

       [laughs]

   12. EMEzra Mechaber16:28

       Um, I used to spend a lot of time thinking about how to coach people and, uh, you know, I feel like managers are very busy, especially now. And so thinking about like, I always wanted to make sure I was doing the first pass of my own coaching before taking it elsewhere.

   13. CVClaire Vo16:45

       Mm-hmm.

   14. EMEzra Mechaber16:46

       And it's just really good at saying like, "Hey, here's our career ladder. Here's where you are. Here's all the feedback that I have about you. I'm gonna listen to your problem, or I'm gonna listen to this thing that you're working through, and I'm gonna tell you basically like get over yourself and don't be like-

   15. CVClaire Vo17:01

       [laughs]

   16. EMEzra Mechaber17:02

       ... don't worry about it and just like keep working on this thing. Or like learn how to communicate with this person, 'cause I remember that the last time you had an experience with this person, like you misunderstood them, and when you had a conversation about it, you understood that you had just misunderstood." So I really like it for coaching. I think it's really, really strong on that one. Um-

   17. CVClaire Vo17:19

       And you have about 30, 31. And like just looking through the list, they seem like both a combination of very tactical-

   18. EMEzra Mechaber17:26

       Yeah

   19. CVClaire Vo17:26

       ... interpersonal, and then like generative in some ways.

   20. EMEzra Mechaber17:29

       Totally.

   21. CVClaire Vo17:29

       Like that seems to kind of be the, the, the,

8. 17:33 – 19:30

   Transitioning from the terminal to a local web front end

   1. CVClaire Vo17:33

      um, the catalog of them. Okay, so how do people actually use this? Is this a kind of like app that you build?

   2. EMEzra Mechaber17:41

      Yeah.

   3. CVClaire Vo17:41

      Like how are people using this?

   4. EMEzra Mechaber17:43

      Yeah. So I think it, it's been a little bit of a journey. So we started in Claude with the like literal TUI in terminal. Um, we had like a lot of debates about like the best terminal app 'cause people were coming over from UIs. Then we built a little front end for it, which runs locally, and tried to make it really fun. It's got like all of the like LeetCode-type ASCII art in it. And the goal here was just bring it into the web browser so that people could type. Like we're doing a lot of writing. It's hard to read monospace. It's got some nice markdown formatting. We built it, it works on mobile. I, I actually like right after, this was, this was pre-Astra, but like I spent a bunch of tokens building like a local native version just to like see what it would look like. Now I'm basically in Codex 100% of the time. One, I think browser use has gotten really good.

   5. CVClaire Vo18:37

      So good.

   6. EMEzra Mechaber18:39

      Really good. Um, we have... Uh, we don't have permanent desks in the New York City office, and I don't like thinking about booking a desk every day. And so every once in a while, I have it running on a schedule to just, like, book my desk one day out. Um, but it's also... It's a total beast at bug bashing. And so, um, like, in Codex, I have a little bug bash. I have, like, a threaded conversation that just lives here all the time, and when something new comes up for our team where we're about to launch an experiment or a redesign or something, we have these FigJam files that explain, "Here's, here's how to bug bash it. Like, here's the test account to take, here's the path to follow, here are the actions to do, and, like, here are the things to look for." And I have started basically snagging one account for me and one account for Astra,

9. 19:30 – 22:08

   Bug bashing with Codex

   1. EMEzra Mechaber19:30

      and just having it run through. So in this case, we are trying to make some of our, uh, request modals inside of Figma just, like, much more straightforward and much more streamlined. It's all UX stuff. It's all UX writing. It's fairly straightforward, but the, like, logic behind it is really complex. And so Astra is just much, much more diligent about following directions than I am. And so I will run it through, and I will say, like, "Keep doing all of these specifically," and just, like, it will run through all of these scenarios, and it will use the browser, and, uh, it's a dream. And then when it's done, it writes out back to the FigJam board of all of the things that it has done. Uh, and I feel like we are still tuning it so that it is not, like, annoying in the comments that it drops, but in terms of, like, what it finds for me, like, it, it found for... Two things it found in the last month. One, it discovered that if you put 1,000 emojis in a text box, it would break, which, like, weird, weird edge case. Great. Good to harden. Two, it dis- it found that the way we had you request a seat for a given team inside of Figma, and when you do it, uh, it shows your team name. And we had A, like, A "team name" seat.

   2. CVClaire Vo20:49

      Mm-hmm.

   3. EMEzra Mechaber20:49

      And if your team name starts with A, you need to use an.

   4. CVClaire Vo20:53

      An. [laughs]

   5. EMEzra Mechaber20:53

      And, like, it had just not shown up in our earlier bug bashes because it really depends on what the team name is. And so it was just... It wound up being so diligent that it caught really small knits like that that I would never have caught on my own.

   6. CVClaire Vo21:06

      This episode is brought to you by Jira Product Discovery. With AI, a single PM can summarize research, draft a PRD, and build a working prototype, but multiplayer mode, that's broken. You can't get everyone aligned on what should actually ship with just a markdown file. Jira Product Discovery gives your team one place to capture insights, prioritize with evidence, and build living roadmaps everyone can rally around. And because it's powered by Atlassian's teamwork graph, it has the context you need to decide what to build. Those decisions flow straight into Jira so your developers or coding agents can start building with the context behind every decision. More than 25,000 companies already use Jira Product Discovery. Try it free at atlassian.com/howiai. That's atlassian.com/howiai. Jira Product Discovery, better product decisions in the AI

10. 22:08 – 26:26

    AutoXP: an automated growth PM

    1. CVClaire Vo22:08

       era. I, I love that so much. And so I think this, like, bug bashing workflow is a really great one. I highly recommend people do this, and then keeping track of it in a FigJam or some other sort of, like, kinda asset is helpful for you to just track what the feedback is. You had another automation that you're using in the team around experiments, right?

    2. EMEzra Mechaber22:29

       Yeah. So one of the things that I had been thinking about for a while as our models have gotten better is, like, what are all of the different ways that we can improve cadence? And so one of... Like, I, as a growth PM, I feel like the biggest sin you can commit is wasting traffic, right? To, like, have unallocated traffic that is not running towards an experiment at any given time is like, why are you wasting that traffic? And so thinking about ways that we could run an experiment on a surface we would never have spent time doing normally was really, really interesting to me. And then the models were just getting good enough across a little bit of engineering, a little bit of data science. The user res- like, the user experience writing was not so bad. The ability to come up with, like, within a really bounded sandbox was getting better. And so I started thinking about, like, what if we could just automate myself out of a job, and, like, create a little growth PM who could just, like, run and run and run. Um, and there's, there's been a couple of other folks who have done this. I think some of the folks at both the labs have done it, um, which is where a bunch of this inspiration came from, but I realized that internally we finally had all the tools, and so I call it AutoXP. And right now, one of our engineers is running it and has, like, five different experiments going that I have not taken a look at and are just r- and it found a huge win on its very first experiment. And basically, the way that it works is this... You can see this is, like, one of the fancy schmancy artifacts that we made for this. Um, is it works as a cursor automation. So there's, it, there are three cursor automations that trigger based on stuff in Slack. AutoXP has its own channel where it can communicate with itself. Someone can either tee up an experiment, uh, in the Slack channel, or they can, or it can just, like, run on a loop. So one of the things you can do is just say, "Hey, AutoXP, go find an experiment to run on our settings page. See what you can do. Go crazy." And it will trigger a data scientist to go find, or a data scientist agent-

    3. CVClaire Vo24:37

       Agent

    4. EMEzra Mechaber24:37

       ... to go find, uh, to go find drop-off, and, like, look at the funnel, and look at makeshift, and all of those things. It will trigger a UX writing agent that will come up with some ideas for copy in a given place. Um, we're, like, moving towards more front-end design now. Um, and then, uh, lastly, it'll trigger an engineer that'll pick up a PR and run it. And then, again, with all of these steps, it writes back to the Slack channel in the same thread, so you can follow over the course of the experiment basically, like, what's happening. Um, and there are two things that I think it does here that are really amazing. One of which is we are able to run experiments in places we would never have bothered to dedicate time or traffic because it's way faster to get from, "I have an idea," to, "This experiment is launched." And then two, um, when it's done, uh, when... And an experiment finishes, it triggers, uh, essentially a web hook back to the original experiment to say, "Hey, it's time to analyze these results." And it has a bunch of instructions from all of the time that I have spent as a growth PM to figure out, like, do we want to explore more? Do we wanna try to exploit? Is this a win? Is this a loss? And the goal for all of this is the only time you're allowed to look is before you launch the experiment. So, you review the data science findings, the hypotheses, the UX writing, and the code all at once, so, like, one human touch. And then it launches, and then you can review whether or not it's a win or a loss, but it has already started iterating on the next rounds. I would say, like, where we are in theory is pretty, like, is great. Where we are in practice is almost there. And so one of our engineers, Marianne, has been taking this really, really far. Um, but we're now at, like, five live concurrent experiments on things that we would not have been doing otherwise.

11. 26:26 – 26:58

    Claire’s Experiment Engine backstory

    1. CVClaire Vo26:26

       So, this is a little bit of Claire Vo lore, but for those that are, are new to me, I have worked at every experimentation platform company in history, and-

    2. EMEzra Mechaber26:37

       [laughs]

    3. CVClaire Vo26:37

       ... did a startup literally, oh, 13 years ago, something like that, called Experiment Engine. And the whole hypothesis was like, how do you get... How do you exploit all this traffic and maximize the number of experiments? And then, like, what a time to be alive.

    4. EMEzra Mechaber26:54

       Oh, it's the best.

    5. CVClaire Vo26:54

       It is here. You can just, like, vibe code, vibe code it there.

    6. EMEzra Mechaber26:56

       It's the best.

    7. CVClaire Vo26:57

       This is amazing.

12. 26:58 – 33:15

    Building a real-time writing assistant with Jev

    1. CVClaire Vo26:58

       Okay. I want to show one last thing, which is my boy Jev.

    2. EMEzra Mechaber27:04

       Yeah.

    3. CVClaire Vo27:04

       And we have been doing a lot of Jev episodes.

    4. EMEzra Mechaber27:06

       Oh, poor little Jev.

    5. CVClaire Vo27:08

       Poor... I, we, uh, I don't know. I think J- Jev's got a, got a real shot at some lon- longevity here.

    6. EMEzra Mechaber27:13

       Jev has some juice.

    7. CVClaire Vo27:14

       But you... [laughs] Juice. But you built something really cool with Jev that I want you to show, 'cause it's a use case I haven't seen yet.

    8. EMEzra Mechaber27:20

       Yeah. Yeah, totally. So, right. So, for people who don't know, though how could you not know, I feel like, if you're listening to How, How I AI right now. Jev is this model where you can ask it really weird, complex questions in text, and you get, like, essentially a number back, right? So, like, how good is this on a scale of one to 10? Or is this true or false? Or, like, what is the meaning of life? Is it A, B or C? And it's really fast, and it's really cheap. And so I started thinking about all of the different places we could shove this that would really change the way that people work, right? So, if you've used... Figma has an agent on the design canvas. I adore it. It's really, really good, but it is still subject to all of the latency that you get with all of these top-tier models, which is, like, it's slow in places where you wanna have a coworker, and it doesn't feel like a coworker. And so as we work on that, Jev feels, like, really, really interesting for some of those possibilities. And one of the first things I was thinking about was this idea of, like, what if it could run questions about things you were writing as you were doing it? Like, what if it could know that you needed to... If you've ever written a blog post and had a section where you aren't sure what's gonna go there, maybe you write TK or you leave some brackets. And sure, you could do that, like, totally programmatically of just, like, look for and identify. But also, maybe you have a claim that you're not sure, like, you have data for, or you need to go look up in your notes later. And so I basically, I built this prototype. I'm gonna forget the name of it, but there's, uh, I think it's called Marker, uh, which is this beautiful-looking writing app from, uh, this guy John Steinbeck, which is an incredible name for a guy who runs a writing company.

    9. CVClaire Vo29:05

       [laughs]

    10. EMEzra Mechaber29:06

        They have some of these ideas around it. And so as I was just playing with, like, what could be possible with Jev, that was kind of one of the inspirations here. And I wanted to have Jev basically make a call to Jev anytime I stopped typing. And so the thing that it does is when you stop typing, it will look for, uh, places where you've left blanks or claims worth supporting, and it will use stuff in your library to go ahead and actually, like, substantiate those claims. So, in this case, like I said, most people find it easier to keep writing than to stop and look something up. And it's making a call here as soon as I stopped writing it into, uh, the library where it has all of these interviews about, like, what these people actually did. Like, if I go hunting for a number, the sen- it will... I forget the sentence I was trying to write. We recently put the API behind our own internal firewall, so let me see if I can actually run it live or if it's gonna get mad. Uh, let's... Okay.

    11. CVClaire Vo30:13

        Oh.

    12. EMEzra Mechaber30:13

        And then-

    13. CVClaire Vo30:14

        Hot dog.

    14. EMEzra Mechaber30:14

        Oh, no. Hold on. But it doesn't actually wanna... Is it blocking it? Oh, it is blocking it.

    15. CVClaire Vo30:21

        I don't know.

    16. EMEzra Mechaber30:21

        It wants to run movies but it won't.

    17. CVClaire Vo30:22

        It, it impressed me. It did something fast-

    18. EMEzra Mechaber30:24

        Right

    19. CVClaire Vo30:24

        ... in my eyes. [laughs]

    20. EMEzra Mechaber30:25

        Right.

    21. CVClaire Vo30:26

        Okay. So I just wanna, like, reiterate this for people-

    22. EMEzra Mechaber30:29

        Yeah

    23. CVClaire Vo30:29

        ... because it is mind-blowing, which is as you write, as soon as you stop for a minute, it runs a quick Jev check, which is very, very fast, and says, like, "Does something need to be filled in? Does something need to be supported?" Like, what questions do I have about what you're writing, and then, like, hooks up to data sources and actually helps you- improve your writing as you... I, uh, Jev

    24. EMEzra Mechaber30:50

        Right. Okay, so this is how it works. So you've got, um, here you've got your title. So this is my title. And as soon as I stop, it's gonna start checking things. So maybe I've got, um, I was thinking about how we serve writers, and I thought TK. Great. And it's like you had a thought that you didn't finish, but then you'll see, like, it also ran a check and it said, "I d- I have no idea what you were thinking 'cause it's not in this library." But then I could say something like, uh, we spoke with X writers and they said... And it both knows that I need to finish this, and it went and pulled from the library, like, literal interview quotes from, like, writers we allegedly spoke to for this. Um, so this one-

    25. CVClaire Vo31:35

        A thousand out of 10

    26. EMEzra Mechaber31:36

        ... just, it blew my mind because I think-

    27. CVClaire Vo31:38

        I was like this. You, you rarely-

    28. EMEzra Mechaber31:40

        Yes

    29. CVClaire Vo31:40

        ... get a, one of these at How I AI. A, a full Claire hands up. Jev.

    30. EMEzra Mechaber31:45

        It's so cool.
13. 33:15 – 42:27

    Lightning round and final thoughts

    1. CVClaire Vo33:16

       Ezra, this has been so good. Okay, we're gonna get to lightning round. I just, to reiterate for folks-

    2. EMEzra Mechaber33:20

       Yeah

    3. CVClaire Vo33:20

       ... what do we see? PM OS with 31 skills, including what's up, what do I have going on today, find me people in the org, write specs, coach me, the whole thing. You basically are now using that through Codex entirely, also because Codex is magical for browser use. And then you built a real time Jev editor, author, writing tool, um, that you're afraid is going to cause you trouble in future PM interviews or impress you in future PM interviews. So many good workflows in this. Um, so the folks that have stuck with us this long, you have gotten some good stuff. Um, quick lightning round questions and then I will get you out of here. One, you told me a couple weeks ago you were using Instinct. Are we still on Instinct? What are we doing with our personal agent?

    4. EMEzra Mechaber34:11

       Okay, so I still get texts from Instinct and I don't respond.

    5. CVClaire Vo34:14

       Same.

    6. EMEzra Mechaber34:14

       It's kind of the way it works. So, um, for people who don't know, Instinct is this, um, this really sort of beautiful idea, like I'm really into it, which is this idea that what if you had a proactive agent, but instead of, like, needing a website to visit or an app to use, it just, like, texts you in iMessage. And I will say I still use it every once in a while. It was this kind of magical thing at the beginning where it was proactive in the right ways and this, like, concept of memory that was, like, useful in this kind of mind-blowing way. And I suspect there's a bunch of people who have been running OpenClau for months who are like, "What's the big deal?" But it just, it productized this really nice experience in a way that was really great. And for the first few weeks you could buy things on Amazon with it, um, which while not, like, a killer use case, was really nice 'cause I would just walk around the house and I would say, "Hey Siri, tell Instinct that I need to buy paper towels." Or, like, my current use case is I bike my kids around Brooklyn, and I will run into potholes all the time. And, uh, now you can, like, th- you can 311-

    7. CVClaire Vo35:20

       311

    8. EMEzra Mechaber35:21

       ... like a, a Fix My Pothole.

    9. CVClaire Vo35:23

       Mm-hmm.

    10. EMEzra Mechaber35:23

        And I never remember to do it on time because, like, you have to look at the address of where it is, and you have to report it, and the website is kinda trash. And so I have started, like, "Hey Siri, text Instinct there's a pothole at, like, 269 Atlantic," and it just remembers, like, what it's supposed to go do and submits it. So I do use it for that, and it's amazing for that, but I mostly switched over to Dots now that I'm, like, fully Codex built.

    11. CVClaire Vo35:50

        I, I, I ha- so I, um, am trying to make Dots ha- I'm trying-

    12. EMEzra Mechaber35:55

        You're like, you're like Groq and Muse, is that right?

    13. CVClaire Vo35:58

        I, so I'm like-

    14. EMEzra Mechaber35:58

        That's your combo?

    15. CVClaire Vo35:59

        ... max, Codex maxing. I love Codex. You could rip Codex out of my cold, dead fingers. But for work I'm all Groq bot.

    16. EMEzra Mechaber36:07

        Yeah.

    17. CVClaire Vo36:07

        And then for pers- you should check out Muse. It's a real, it's a real delight. Um-

    18. EMEzra Mechaber36:12

        I can't, I can't bring myself to do it.

    19. CVClaire Vo36:15

        I know, but they've been stealing our data for two decades, you know?

    20. EMEzra Mechaber36:18

        I know.

    21. CVClaire Vo36:18

        [laughs] That's-

    22. EMEzra Mechaber36:20

        I know

    23. CVClaire Vo36:20

        ... that's my, I, as I say to people, the devil I know. [laughs] This is really where I'm at-

    24. EMEzra Mechaber36:25

        Totally

    25. CVClaire Vo36:25

        ... with my personal agent

    26. EMEzra Mechaber36:26

        ... yeah, they already have the data, so fair.

    27. CVClaire Vo36:27

        They already have... I said, my joke is my Instagram ads will get better. I'll, I'll end up spending more money.

    28. EMEzra Mechaber36:32

        This is j- I think this is, like, a real value prop. Um...

    29. CVClaire Vo36:35

        [laughs] Okay, and then my last question for you. When your dot is-

    30. EMEzra Mechaber36:40

        Yeah

Episode duration: 42:29