How OpenAI Builds Products: The Framework You Need

Aakash Gupta 0:00 In the future, are PMs gonna have to learn how to manage not just humans, but agents?

Jake Brill 0:03 Absolutely, you're gonna be interacting with, managing, collaborating with other people. I think it's very much the case with agents, and so there's gonna be a new skill set that people need to develop.

Aakash Gupta 0:12 Jake Brill is the head of Integrity Product at the hottest AI company in the world right now. He works at OpenAI. Before that, he was a director of product at Instacart and a product manager at Meta. So you guys just released GPT-5. It was a whirlwind launch. What did it feel like inside?

Jake Brill 0:30 Truly energizing.

Aakash Gupta 0:31 What do PMs need to know about agents?

Jake Brill 0:33 If you look at your average product manager and you said, "What's your AI strategy?" or, "How are you building an AI-first product?" A lot of people wouldn't have necessarily had an answer for that. And over the past couple of years, it's like if you're not building a product that has AI fundamentally in its DNA, you're not really keeping up with the future of digital technology.

Aakash Gupta 0:51 One of the craziest things I heard about agents recently was that agent internally was manipulating performance data. [laughs] How do you deal with a world where agents learn to cheat?

Jake Brill 1:01 It's a hard problem, a very big research problem we're gonna keep working on.

Aakash Gupta 1:04 You mentioned Slack. I think that was one of the coolest insights that came out about OpenAI in the last few weeks, that the company almost entirely runs on Slack.

Jake Brill 1:12 Yeah, almost entirely. I would say conservatively, like, ninety percent of my written communication is in Slack.

Aakash Gupta 1:17 If you're a company who's maybe not a motto company, but you're trying to create an eval system as robust as OpenAI, what would be your lessons for them?

Jake Brill 1:26 Well, in a lot of cases, you don't have to build these evals from scratch. There are industry-standard evals that you can use.

Aakash Gupta 1:32 What does the PM role look like in five years? Really quickly, I think a crazy stat is that more than fifty percent of you listening are not subscribed. If you can subscribe on YouTube, follow on Apple or Spotify podcasts, my commitment to you is that we'll continue to make this content better and better. And now on to today's episode. Jake Brill is the head of Integrity Product at the hottest AI company in the world right now. He works at OpenAI. Before that, he was a director of product at Instacart and a product manager at Meta. We have a ton to talk about, especially as it relates to how does OpenAI handle negative scenarios within AI? How does OpenAI build product? Jake, welcome to the podcast.

Jake Brill 2:23 Thank you so much for having me. Um, it's a real thrill to be here.

Aakash Gupta 2:28 So you guys just released GPT-5. It was a whirlwind launch. You guys were actually everywhere. I think this was the most marketed launch I saw from you guys. What did it feel like inside?

Jake Brill 2:40 Truly energizing. You know, we've been working on GPT-5 for a very long time, and in the days, weeks, months leading up to the launch, the energy was just palpable. We knew how incredible of a model we had. We knew how much of a game changer it would be for people to experience a reasoning model, because most people haven't. And so bringing that power to people, um, it, it just felt su-- like we're just one step further in our quest towards AGI and fulfilling our mission. The energy was great. The day of the, the launch, you know, it was a really well-done, uh, launch event, and just seeing people's reaction in Slack, starting to see the, the graphs, like, watching people use this product, it, it just-- It felt incredibly fulfilling because this was-- This is a lot of really smart people working really hard for a long time, and seeing it come to the world, it was just like, "Okay, we're here." Very exciting.

Aakash Gupta 3:34 You mentioned Slack. I think that was one of the coolest insights that came out about OpenAI in the last few weeks, that the company almost entirely runs on Slack.

Jake Brill 3:42 Yeah, almost entirely. I would say conservatively, like, ninety percent of my written communication is in Slack.

Aakash Gupta 3:50 Wow. And what does it look like? Like, is there any special things about your guys' Slack setup? Do you guys have a lot of AI agents in there keeping things updated?

Jake Brill 3:59 Yeah. You know, it's funny because at Instacart, we were really Slack-heavy as well, and Instacart was fully remote for my last several years at the company, and so we relied a ton on Slack there. So when I joined OpenAI, I had something that I could fall back on. And so I was used to a company that ran heavily on Slack. In terms of what makes our usage of Slack, um, special and more unique for us, yes, absolutely, we have AI agents in the, in the channels. Uh, they can serve any number of purposes. You know, there's, there's one that we have that's like a general purpose answer, uh, agent, which is really helpful. So if you have a general Q&A in the channels where that agent exists, you can get questions without one of your coworkers having to see the post and respond to it manually. So, um, we very much believe at OpenAI in, in, in using AI in all parts of our, uh, user and enterprise experience. So yeah, it's, it's no different in Slack.

Aakash Gupta 4:47 Very interesting. Now, you head up Integrity Product.

Jake Brill 4:51 Mm-hmm.

Aakash Gupta 4:51 What is Integrity Product's role in a big launch like GPT-5?

Jake Brill 4:57 Sure. So there are a couple big areas that we're involved in this launch. The first is what you might classically think of when it comes to integrity, you know, preventing p- bad things from happening, people from doing bad things with our models and our products. And we certainly did a lot of work in the lead-up to and following the launch i, in that space for GPT-5. In addition to our work on preventing bad things from happening, um, one thing that of-often happens when we have new launches is we see a lot of people try ChatGPT for the first time or try our new, new models via the API for the first time. And another area that Integrity is responsible for is our iden- our identity system. Um, so when we, when we, uh, release new models, like this happened with our Image Gen model, um, as well as GPT-5, you see an increase in traffic, people wanting to sign up for the first time, people logging into their accounts. You wanna make sure those systems don't fall over and that you h- your systems are able to take meaningful increases in volume so that they have-- that they maintain their world-class uptime and latency because, you know, it would be a really bad user experience if people try to go use, use GPT-5 for the first time and they just get an error message. Similarly, our team is responsible for our financial systems and-Uh, when we launch new models, it's often the case that that will lead to people converting to paid subscriptions or more people using prepaid API credits to use the model. And so similarly, we wanna make sure that our payment systems are, are up, they're running, they're seamless, you have good a- good, uh, authorization rates, but also that you don't see an influx of people with stolen credit cards, uh, fraudulently, fraudulently using our, our models and our products. So a lot of work, um, behind the scenes to make sure that GPT-5 shines in, in all of our different products.

Aakash Gupta 6:43 Fun fact, Amazon put two holds on my account as I ordered the equipment for this in-person [chuckles] podcast today. So it's not just on the credit card side. Even the person who's making a purchase at OpenAI or in Amazon is monitoring those accounts to see, are these suspicious transactions, and flagging those appropriately.

Jake Brill 7:02 Exactly.

Aakash Gupta 7:03 Very interesting. So if we break down, let's say, let's stick to GPT-5 for a second. What are the big streams of work that a company like OpenAI is doing to make sure that it meets your integrity standards? Is there red teaming? Are there certain evals? Are there certain... What are the big-- How would I put it into buckets?

Jake Brill 7:22 Yeah, absolutely. We do, we do red teaming, not just for new launches, but for our products a- all across the board. And red teaming happens at all stages of product development.

Aakash Gupta 7:31 Right.

Jake Brill 7:31 When you're training the model, when you're training your production systems to, uh, to mitigate, uh, harms from happening. We have both manual red teaming as well as automated red teaming. Uh, we'll do that at different checkpoints while the model is being trained. We do it with the final version of the model before it goes live. And then we also do red teaming after models go live, just to make sure it's like, okay, well, we prevented the known jailbreaks. Are there new jailbreaks that are out in the wild that pe- that our automated systems or, or, uh, penetration testers are trying to, to, to find ba- um, ways to, to get around the protections in our model? Um, other sorts of work that we do, um, there's a lot of work to make sure that our automated systems have high precision and high recall. You know, we wanna make sure that, um, uh, if we're taking some sort of automated action, that we're really accurate with that, because we take it very seriously if we, um, block a generation or warn someone's account, or even at the extreme, ban someone. Like, those are very serious interventions, and we wanna make sure that we hold ourselves to a really high standard that when we take those actions, we have really high confidence that we're getting it right. Similarly, we wanna make sure that our recall is high so that we don't j- we're just not blind to false negatives and, and there's all sorts of bad stuff going out there that we're not aware of. Um, we also do a lot of making sure that we have robust, um, operational tooling and capacity, um, to make sure that, you know, we project forward, we think X number of things will require manual review. Um, and we wanna make sure our tools are sufficient so that they, um, that people have extremely e- uh, efficient workflows and that, um, we have enough human power and, and bandwidth so that we can get to things in a very timely fashion.

Aakash Gupta 9:14 And I think some people might be kind of rolling their eyes at this point like, "Is all this stuff important?" But I think this actually goes to the core of why even OpenAI was created at the beginning, and a lot of the values that you guys have from the beginning of creating this safe AI. Talk to me a little bit about that. How is that interwoven into OpenAI's philosophy, and how does that manifest?

Jake Brill 9:35 Yeah, I mean, i- if you read OpenAI's charter, safety is one of the things we talk about very publicly. It is part of the reason why I even wanted to join OpenAI in the first place, was when I had my first conversations with them, uh, it was very clear that safety was, was really important to the company at all stages of model and product development. And I figured if I was gonna leave a job that I really loved with people I really cared about, I wanted to do it not just to jump on, like, a hot AI train. I wanted to do it because I really believed that the company, uh, was responsible and took this, took that responsibility seriously. So safety and integrity, it's something we talk about, uh, throughout the entire development process. Um, and we, you know, the-- We do also believe in iterative deployment, so s- they-- it's not like we're launching models and there's absolutely zero possible way that something could happen in-- that, that could have a bad outcome. But we talk a lot about what risks are non-negotiable that we have to mitigate before launch, which are the other ones that we wanna have, uh, systems in place, and which are the other ones that's like, hey, actually, it's-- we, we could sit in a room stroking our chins thinking like, "Oh, how might someone do something bad?" But actually, at the end of the day, it's really helpful to, to follow OpenAI's approach of iterative deployment, because once you start rolling things out, you can actually see in the real world how people accidentally might misuse your products or professional bad actors might misuse it, and then you can very quickly respond and build sophisticated solutions.

Aakash Gupta 11:00 One of the areas that I imagine this is really important is the recent open source weighted model that you guys released. And I know you guys even delayed that launch a little bit because you said it wasn't ready. How do you really determine like, okay, this model is safe enough and ready to put out there?

Jake Brill 11:17 Yeah, well, a lot-- so much of that is evals. You know, we, we try to make sure we're, we're holding ourselves accountable to objective sources of truth on the relative safety of our models. Um, and, and there are a number of public evals that companies talk about, um, to, to ensure that their models are safe. Um, this could be safe in terms of like, is the model deceptive? Is it safe in terms of is it appropriately refusing some of those high-risk bio prompts, for example? Um, y- we, you know... Y- y- you can make vibes-based decisions on this sort of stuff, but ultimately, evals are, are really what's gonna guide the day in helping you objectively and with data determine if your, if your model is safe enough to, to release.

Aakash Gupta 11:57 If you're a company who's maybe not a motto company, but you're trying to create an eval system as robust as OpenAI for whatever... product you've created on top of the OpenAI API, what would be your lessons for them? Like, how, how can they build evals that they can trust?

Jake Brill 12:13 Well, in a lot of cases, you don't have to build these evals from scratch. There are industry standard evals that you can use, and we've published a bunch of them. Um, but there are many other, uh, excellent frontier labs creating, uh, safety-related evals. And so what I would ultimately say is if you're, if you're an earlier stage company, you don't have to reinvent the wheel here. I would really encourage you to just... You could even use AI to go do a search on, like, what are the best safety evals. Uh, simultaneously, there are a lot of, uh, o- open, open source, open standard, um, like, safety models that you can layer in on top of your model or your product. At, at OpenAI, we've built a moderation API, and there are plenty of other great options out there. And so I would say you're not starting from scratch, and I would enc- really encourage people to make use of, of the great open technology that exists out there.

Aakash Gupta 12:59 Got it. Cool. So we're walking back in time through all of your amazing launches. The one before that, you mentioned it earlier, agents.

Jake Brill 13:07 Yeah.

Aakash Gupta 13:09 Feels like I can't go on social media without seeing some post about [chuckles] AI agents. Why is everyone making such a big deal out of agents?

Jake Brill 13:17 Um, 'cause they're the bee's knees. [laughs]

Aakash Gupta 13:20 [laughs]

Jake Brill 13:20 Uh, the way I think about this is y- you rewind three years ago, ChatGPT launched. People started having their first sort of cons- m- broad scale consumer experience with AI. And for th- those first couple years, it's really been wha- what we call assistance. You know, you asked a model a question, you give it a prompt, and you get a response, and that's been incredibly powerful. I mean, the growth of ChatGPT and many other great AI products sort of speaks to the power of AI assistance. But where we foresee this technology going is not just question and answer, but rather, "Here's a task. Can you please complete it for me?" And that's really where agentic product experiences come in. It's, "Hey, there's something more complex than just, like, w- what's the weather today? Uh, can you go and take an action on my behalf?" And that could ha- that action could be synchronous or it could be asynchronous, and you come back because it's rather complex workflow. Um, and I think what's been so great about the progress that's been made in the field of AI is we've been talking about agentic product experiences for several years, but we're now really starting to bring to bear across ChatGPT agent and also agentic products that other companies are building, a real way to put tools in people's hands that can't just answer their questions, that they can also do things on th- those people's behalves.

Aakash Gupta 14:37 Yeah. I personally am using a lot of those no-code agent builders like Lindy, Relay, Make, Zapier. You can chain together, like, crazy-

Jake Brill 14:47 Oh

Aakash Gupta 14:47 ... workflows. So what do PMs need to know about agents?

Jake Brill 14:52 The thing I would say is, uh, I'm gonna go back again to 2022. Um, if you look at your average product manager and you said, uh, "What's your AI strategy?" Or, "How are you building an AI first product?" A lot of people wouldn't have necessarily had an answer for that. And, um, over the past couple years, it's like you're- if you're not building a product that has AI fundamentally in its DNA, you're not really keeping up with the future of digital technology. And I think we're sort of at that space again with agentic products. It's, you know, that classic Wayne Gretzky quote, "You pa- you skate to where the puck is going, not to where the puck is." The puck has been at assistance. The puck is going to agents. And so what I would say is, if you're not thinking about how to build products that are agentic in their fundamental nature, you're probably, A, not maximizing the power of this technology, and B, you're probably building a product that's going to be obsolete in a shorter time horizon because your competitors are gonna be thinking about building products that solve people's problems in a agent, agent-first fashion.

Aakash Gupta 15:57 Today's episode is brought to you by Jira Product Discovery. If you're like most product managers, you're probably in Jira tracking tickets and managing the backlog. But what about everything that happens before delivery? Jira Product Discovery helps you move your discovery, prioritization, and even roadmapping work out of spreadsheets and into a purpose-built tool designed for product teams. Capture insights, prioritize what matters, and create roadmaps you can easily tailor for any audience. And because it's built to work with Jira, everything stays connected from idea to delivery. Used by product teams at Canva, Deliveroo, and even The Economist, check out why and try it for free today at atlassian.com/product-discovery. That's A-T-L-A-S-S-I-A-N.com/product-discovery. Jira Product Discovery, build the right thing. Today's episode is brought to you by the AI PM Certification on Maven. Run by Miqdad Jaffer, who is a product leader at OpenAI, this is not your typical course. It's eight weeks of live cohort-based learning with a leader at one of the top companies in tech. OpenAI just doesn't stop shipping, and this is your chance to learn how. Run along with product faculty and Mo Ali, the course has a 4.9 rating with 133 reviews. Former students come from companies like OpenAI, Shopify, Stripe, Google, and Meta. The best part, your company can probably cover the cost. So if you want to get $500 off, use my code AAKASH25 and head to maven.com/product-faculty. That's M-A-V-E-N.com/P-R-O-D-U-C-T-F-A-C-U-L-T-Y. And what does it mean to build in an agent-first fashion? Because people are talking about, you know, user experience is gonna turn into agentic experience. You need to figure out how your product will be open to AI agents. Like, what are the elements that a PM should be thinking through for making their product agent-friendly and usable?

Jake Brill 18:01 Yeah. I- one of the first things I would say is so m- many digital proje- products are synchronous in their nature. Like, I take an action and a response or something else happens immediately. And if you start thinkingHey, what if I could enable a product experience that is far more complex and it doesn't have to happen immediately? I think that's one of the biggest changes that's happening right now is, is a stop boxing yourself in and thinking like someone clicks button and action happens immediately. Rather, it's someone clicks button and something far more complex can happen behind the scenes, and you don't have to sit there waiting for a response. Uh, the product can come and let you know in the future when that very complex task has happened.

Aakash Gupta 18:44 Mm-hmm. So what are some real-life use cases that you're using agents for?

Jake Brill 18:49 Yeah. Um, well, uh, we're growing a lot at OpenAI, and so I spend a lot of my time hiring and interviewing candidates. Um, and I find, uh, I, I find it's been deeply meaningful and helpful to, uh, to use, uh, agentic products to help with the recruiting process. You know, I'll say like, "Hey, we need to bring in a designer. Here's sort of the properties. Ideally, they have X years of design experience, or they've, they've worked, uh, in, you know, this size company, um, and they're located in the Bay Area. Please go help and like source some candidates." Really, really strong for that sort of use case. In my personal life, I've been using agentic products a lot to do longer horizon, uh, research. Um, I got this like super gnarly, uh, respiratory virus in December, and I s- I've been seeing a bunch of, of doctors about it and like I'm not... varying levels of usefulness-

Aakash Gupta 19:48 Hmm

Jake Brill 19:48 ... from that, from, uh, those doctors visits. And I've actually found that using, um, agents to go do some of the research and suggest alternative methodologies for, for like helping my lungs get back to 100% has been really helpful for me personally.

Aakash Gupta 20:03 Okay. And on the product management side, if I'm just a product manager, what agents should I think about building or using?

Jake Brill 20:11 Yeah. So for building, um, I think a- agents are really helpful for running market analysis, you know, understanding, you know, if, if you're a PM in a given area, like what sort of products exist in that area? What is the general sentiment towards them? What are the, what are the needs of, of people who are, are looking at products in that area? I think ag- uh, agents can be really helpful in helping you pull together collateral. You know, I, I, uh, I am not the best at like visual-- uh, like I'm not a great PowerPoint PM.

Aakash Gupta 20:42 Hmm.

Jake Brill 20:43 Um, you know, I can tell the story, but like the, uh, I, I don't have a background in consulting. I like, I was never the best at, at, at making beautiful slides. There are now some really excellent agentic products that can help you visually tell your story, uh, in a way to supplement, you know, all, all of your written communication skills. Um, I also think, uh, you know, agentic products, you know, uh, can help you prototyping a lot. You know, that's one of the things we've been starting to do internally is rather than just writing up a proposal for how something works, you know, just build a prototype of how something could work and you put that in people's hands, and it is a, uh, a great way of illustrating how your product might work without having to, um, you know, go and make mocks, uh, start building an initial prototype on the, on the software side. It's, it's like a very inexpensive way of bringing your ideas to life.

Aakash Gupta 21:44 Yeah. I ask a lot of heads of product on the podcast, like what are the big ways your PM team is using AI tools? And AI prototyping is always-

Jake Brill 21:54 Yeah

Aakash Gupta 21:54 ... the number one thing they talk about. What do you see as the future for PRDs in a world of AI prototyping?

Jake Brill 22:01 Yeah. It's a great question. So I still think there's a, a world for PRDs because in some level, like AI technology is making PRD writing so much better. You know, if I th- these days I'm more in a support role, so I'm not writing as many PRDs. But if I rewind like a year and a half when I was in my earlier days at OpenAI and spending more time writing PRDs, and I kept trying to use AI for writing the PRDs, and it just like came up short. The form, product form factor wasn't right. The m- it didn't sound like my voice. The like amount of context that was necessary, like the, the PRDs, like I, I ended up just writing a bunch of them. And you fast-forward to where we are now, and I think AI has made PRD development much better. Um, you know, at, at least within OpenAI, I think Canvas is a much better form factor for collaboratively putting together a longer documentation. I think our models are better at creative writing. With connectors, you can connect to external data sources to help, uh, bring these documents to life. And then lastly, there's with memory, these models learn a little bit more about what your writing style is, what other things that you've worked on. They can bring that to bear. So I think there definitely is a future for PRDs. I think they're gonna be AI first. Um, and the value of PRDs is you can do a prototype, but you may be missing a bunch of the things in that prototype. So like how do I handle, you know, like how do I handle this failure case? Or how do I, um, you know, what is, what is our marketing strategy that we're gonna bring with the product when we bring it to market? So I think there's, I think they're gonna go hand in hand, the prototypes and the PRDs. I do think PRDs will be less wordy [chuckles] 'cause you won't have to spend as much t- time describing, "Oh, you click on this button and this thing happens." You can just show people. But, um, I, I think, I think even a world where people are prototyping more with, with our, our models and our products, I still think there's a place for PRDs.

Aakash Gupta 23:57 One of the coolest things that you guys open sourced, I believe it's called your model spec-

Jake Brill 24:02 Mm-hmm

Aakash Gupta 24:02 ... which had, like, a ton of examples, I think probably that your team was responsible for, of, like, when user has request this, don't provide that information. When they say this, do that. Is that more the direction your PRD needs to go in a world of AI products, where you are giving the specific examples of what not to cover and how to respond?

Jake Brill 24:23 I think that's a good example. I think part of what you are talking about is also covered by evals, um, which right now aren't... We, we sort of view that as complementary to the PRD or spec. But I, I do believe increasingly in the future, yes, it's gonna be describing not just the way the product work, but the way the model that's powering the product should work.

Aakash Gupta 24:43 Mm-hmm. Yeah. Okay. Actually, can you clear that up for me? What should go in the eval and what should go in the PRD?

Jake Brill 24:49 I, I think a lot of the PRD is, like, classically... It looks a lot like what we have today. Like, here's the problem that we're solving. Here is the way a product would work. Here's our definitions of success. Here's our target user. And evals are really fundamentally like, here are the use cases this product should be good at, and here are ways that we can, we can test the model to evaluate is it successful at these specific use cases.

Aakash Gupta 25:16 Okay. So PMs, they typically own the PRD, even though they're jointly creating it-

Jake Brill 25:21 Mm-hmm

Aakash Gupta 25:21 ... with the research, engineering, and design teams. When it comes to evals, you know, who owns that and how do PMs plug into those?

Jake Brill 25:29 Well, you know, increasingly, we're asking PMs to take a really active role in developing evals. And I think as the p- role of PM changes over time, that's gonna be of increasing importance. You know, anyone can write an eval. It doesn't have to be a researcher. It doesn't have to be a software engineer. It doesn't have to be a PM. But we're finding that because the PMs oftentimes are the ones with the clearest vision of, of how the product should work in their head, they're very well-positioned to write evals 'cause they c- they have a very strong opinion on here's what the product should be good at and here's what the product should not attempt to do.

Aakash Gupta 26:02 Mm-hmm. I always love to get sidetracked on the tangents about PMs. I wanna return back to agents for a second.

Jake Brill 26:08 Yeah, sure.

Aakash Gupta 26:08 Um, [lip smacks] how do AI agents talk to each other?

Jake Brill 26:13 Yeah. This is a good question, and I don't think it's a solved question. I, I, I know... I really enjoyed your, your podcast about MCP and the, the power of something like MCP is it is a common protocol that multiple companies, AI developers can use that standardize how agents talk to tools. I think if you project out into the future, we're gonna need something like that for agents to talk to agents. There's not going to be just one company building agentic products. There's gonna be the f- frontier model labs building their models that can power agents. Those, those model labs are gonna build agentic products, and then there are gonna be tens, hundreds of thousands, millions of developers building agentic products on top of those models. And the, like, failure state would be if there's not a standard language for all of them to talk together. Agent A talks to agent B, but they're speaking different languages, and so it just falls apart. So this... I, I think the answer here is going to be, be that we need something in the vein of, of MCP, which is an open standard that companies can use so that whoever's making the agents, they can talk to each other. But it, it's still... It's something we're talking about internally, but I s- I, I think there's more work that we need to do to have, like, a consistent answer that can work broadly.

Aakash Gupta 27:24 Okay. And MCP is, it's developing. I guess people are trying to call it the USB-C. What are the limitations of MCP that people should know about?

Jake Brill 27:38 [sighs] You know, ultimately, the, the limitations are gonna be that it's just really early stage and there's a bunch of, uh, key functionality that needs to get figured out.

Aakash Gupta 27:46 Mm-hmm.

Jake Brill 27:46 So if you'd asked me a couple months ago, MCP hadn't quite figured out auth, and that was something that, that standardizing how authentication would work, what, um, was a, was a big open opportunity. I, I think that's gonna be the case for a while, um, that there are key things it's good at, tool calling. But I think it's just, like, it's early. It hasn't even been around for a year. So I think fundamentally MCP is just, like, the way it's gonna get better is more time, more people developing on it, and frankly, more companies contributing to it so it becomes more fully functional.

Aakash Gupta 28:18 Mm. So you don't need to evaluate the state of it as it is. Sometimes people are worried about the security in these things. It's still an open standard. It can still be improved and developed.

Jake Brill 28:26 Correct. Yeah.

Aakash Gupta 28:27 Okay. So one of the craziest things I heard about agents recently was that a agent internally was manipulating performance data. [chuckles] How do you deal with a world where agents learn to cheat like humans do?

Jake Brill 28:42 Yeah. Um, it's a hard problem and, uh, w- a very big research problem we're gonna keep working on. To me, a lot of that h- fundamentally gets down to the question of alignment. Like, are the, are the agents... Is AI acting in a way that is aligned with the values that you're trying to instill in it? So at, at the core of it, it's a lot of training the model, um, making sure that it's robust, making sure that it isn't deceptive. And one of the things we're really excited about for GPT-5 is, is, um, you know, in our evals, um, measuring deception, uh, this model is, is s- This is where you want the score to be the lowest-

Aakash Gupta 29:18 Yeah. [laughs]

Jake Brill 29:19 ... not the highest. Uh, so it scores the lowest. But, you know, it's greater than zero, and you want that number to be zero. So, um, a lot of it has to do with alignment, and then a lot of it has to do with you having systems to detect when, when, uh, the agents go rogue. It... They could be going rogue because, you know, a person asked the agent to do something bad. Um, and the way you solve for that is, you know, uh, multi layers of defense, you know. Again, like I said, you train the model not to do bad things, then you have, uh, model-level classifiers looking at the inputs going into the model and the, and the outputs coming out of the model, and then you layer on top of that a lot of account and behavioral signals about the per- uh, about the individual. So is that individual trying to, uh, do something sketchy with, with the model? Um, and then there's also is the model doing something because it got tricked into it? And that could be cheating, like you said, but it could also be, uh, data exfiltration. And so there's a lot of work to make sure that even if the model is perfectly aligned, even if the person who's acting in a very, um, uh-benign fashion, the agent may still interact with some external resource that tricks it into doing something bad. That resource could be a g- a code repository, it could be a, a website. And so there's a lot of work that we do to mitigate prompt injection to make sure that the agents are acting in a reliable fashion. Um, similarly, model training, model-level classifiers, a-actor-level classifiers, production monitoring, and then just, like, constant red teaming. So the answer is, we haven't-- Like, we don't have a silver bullet here 'cause there's never going to be a silver bullet. The answer is you need to have multiple layers of defense against agents, uh, acting in deceptive fashions, agents cheating, right? Or people using agents for nefarious purposes.

Aakash Gupta 31:01 AI Evals are one of the most important skills for PMs, and I know you know they matter. The question is, are you doing them right? Most teams are winging it with basic metrics and hoping for the best. Meanwhile, the teams that actually ship reliable AI, they've cracked the code on systematic evaluation. Today's episode is brought to you by the AI Evals for Engineers & PMs course by Hamel Hussein and Shreya Shankar. This live Maven course will teach you the battle-tested frameworks from Hamel and Shreya, who are the engineers behind GitHub Copilot's evaluation system and 25-plus production AI implementations. Four weeks, live instruction. Next cohort starts July 21st. Start shipping AI that actually works. Enroll at maven.com with my code ag-product-growth for over $800 off. That's ag-product-growth. Today's episode is brought to you by Maven. The problem with most courses online, like Udemy, is there's no live component, and the instructors aren't experts in their fields, they're professors. At Maven, you get direct live access to experts and operators from the world's best tech companies. You can't get that access anywhere else, in any university, and you usually can't find them on YouTube either. I've featured so many of Maven's experts in the newsletter and podcast for that reason. To help you out, I've put together a collection of courses I recommend at maven.com/x/aakash.

Aakash Gupta 32:32 This includes courses like AI Prototyping for PMs, Product Sense for PMs, and getting an AI PM Certification. Visit it now at maven.com/x/aakash. So agents is one topic that everybody, when I asked, "What should I ask Jake?" They were like, "Ask him all about agents." We covered that. The next second most voted topic was all about what does the product team culture look like inside of OpenAI? So if we start with the context, right?

Jake Brill 33:07 Yeah.

Aakash Gupta 33:07 How does Integrity Product fit in more broadly within the product teams at OpenAI?

Jake Brill 33:11 Sure. So I'm gonna be super reductive. OpenAI is a research and a product company. Uh, we have a number of PMs on the research side of the company, people, uh, who are focused on things like model behavior, like what is the pr-personality of the model? What are the vibes of the model? People who are focused on safety, th-things I was saying before. How do you, um, do research on frontier safety risks, train the model to be safe and, and not take harmful actions? And then there's the product side of the company we, um, and that's really focused on how do you take these frontier safe models and bring them to humanity? Um, and you've probably, uh, interacted with many of the products we built. You know, there's, uh, PMs working on the consumer version of ChatGPT, the enterprise version of ChatGPT. Got a team of PMs focused on, uh, third-party developers. And the way I talk about Integrity is we are a platform team that builds shared technology that helps those product teams bring our AI models to humanity. We are, we are fundamentally-- We, we build systems, products, and tools that are focused on minimizing risk and maximizing trust and control.

Aakash Gupta 34:19 So platform teams, I've PM'd a couple of them, managed a couple of PMs on platform teams. I feel like more than any other team, for them, like, the traditional corporate planning process was really helpful-

Jake Brill 34:31 Yeah

Aakash Gupta 34:31 ... because they could get buy-in from teams who would build on their platform that they were gonna actually use it so that they could actually figure out if they were building it. Does OpenAI have a planning process like that? How do you guys get the buy-in you need to understand what platforms to build?

Jake Brill 34:45 Yeah. Yeah. So we do have a planning process. We deliberately try to keep it pretty light, um, because it's a very dynamic environment and a very fast-moving industry, and, you know, I-- Kevin Weil, my, my manager, our chief product officer, often, often repeats the, the Eisenhower quote, "Plans are useless, but planning is everything." It's a-

Aakash Gupta 35:04 Mm-hmm

Jake Brill 35:04 ... it's a really helpful exercise for people to talk about what they think is most important, where they're gonna be spending their time. And even if you diverge from, from, um, from your plans, it's good to go in being mindful about what, what you think is most important over the n-those next three months, even though something's inevitably gonna change. So our planning process only lasts a couple weeks. We do, um, we plan, uh, f- over every three-month quarter, and often the guidance I give to the Integrity team is assume you're only gonna accomplish something like 60 to 70% of your plan. If you do anything more than that, it probably means you weren't being flexible enough to the needs of the business. If you do anything less than that, probably didn't do a great job forecasting what was gonna be most important over the quarter. But we very much write our plans in pencil, not in pen.

Aakash Gupta 35:48 Hmm. So you guys still do the three-month planning process, and how does that couple week sprint look like?

Jake Brill 35:54 Um, so it's very bottoms up. Um, there's some amount of the teams, uh, you know, for Integrity, we've got, uh, i-and the other, like, product verticals within the team, uh, within OpenAI, you've got your own, like, individual swim lanes within there. And so we, we ask the teams to, um, sort of come up with bottoms up plan for what they think is gonna be most important over the next three months. Simultaneously, we run like a, an internal dependency intake process where people say, "Hey, I want to do X. Can you help supportUh, X with this-

Aakash Gupta 36:28 Mm

Jake Brill 36:28 ... version of technology. With it being a platform team, we had a, a large amount of inbound requests, prob- probably multiple times more than the average team, other average team within, um, uh, within OpenAI, both because we're building shared technology for the other product teams, but also 'cause we have really close partners on our user operations, product policy, intel investigations teams, and we build tooling and data systems for them as well. Um, so it's-- the process only takes a couple of weeks. We put together, um, pretty lightweight documents, um, that follow a template of, like, you know, reflections on the prior quarter, here's what we're tr- you know, here's the big themes running after this quarter, here's how we're gonna measure success. Um, and then we'll either review those documents synchronously or asynchronously. We, we, we aim for async, again, in the, uh, because those meetings can be expensive, so we try to see how much we can get through in common form, and if there's still some meaty topics that, um, that we, uh, we can't resolve asynchronously, then we'll j- we'll jump in a room with the relevant folks and just talk through the, the main outstanding items.

Aakash Gupta 37:31 And how important are success metrics as part of that equation? Because some companies I've worked at, that's like the entire currency. I worked at a fintech company. Everything ultimately it's boiled down to GMV, right? [chuckles] And what is-- I, I feel like there's so many different goals OpenAI has as a company. You probably aren't just like an OKRs-based company. How do you treat success metrics?

Jake Brill 37:52 So we definitely have success metrics. Um, what I'll say is the success metrics for, um, Integrity have a different shape from many other teams. You know, you can measure the success for ChatGPT in classic things like how many people are using the product, things of that nature. Um, for Integrity we, we sort of view our success metrics as are we building systems that enable the success of other products? So there are some standard metrics that, that many teams use, but for us, we think a lot about latency, uptime, maturity of our systems.

Aakash Gupta 38:23 Hmm.

Jake Brill 38:23 And really what those systems are there to do is to make sure the other products can shine. So for example, uh, I mentioned earlier our team is responsible for our identity system, so we think a lot about, like, how many nines of reliability do we have? Because, like, if people can't log into their accounts or people can't sign up for their accounts, well, OpenAI just isn't gonna accomplish its overall goals.

Aakash Gupta 38:44 Hmm. Okay. So sounds like each of the little platform areas might have some product metrics, but there isn't, like, some overall business output metric that an integrity team would be going after.

Jake Brill 38:57 Um, the, the, the top metrics for the company, we, we view ourselves in service of them-

Aakash Gupta 39:01 Mm-hmm

Jake Brill 39:01 ... but, like, we, we don't, like, take goals on, on, you know, number of users of, of this product or that product-

Aakash Gupta 39:08 Mm-hmm

Jake Brill 39:08 ... 'cause we're a bit, uh, downstream from that. And, and so the way we think about success is those teams are going on those metrics, and what we do is we, we ask ourselves, are we, uh, building systems that make our products, enable our products to hit their goals effectively?

Aakash Gupta 39:26 Hmm. So you've been a PM at Facebook for a really long time. You were director of product at Instacart. What are the things that are unique about OpenAI's product culture?

Jake Brill 39:37 Oh, man. Um, so with the caveat that I left Facebook like nine years ago-

Aakash Gupta 39:44 [chuckles]

Jake Brill 39:44 ... so I'm sure it's a very different company now, and I left Instacart about two years ago. One of the things that's classically different about OpenAI is, uh, AI. You know, like, we are constantly asking ourselves, are we making-- are we truly dogfooding our technology? Are we using it as powerfully as we can in all parts of our product development process?

Aakash Gupta 40:05 Hmm.

Jake Brill 40:06 Um, and I, I imagine those companies are asking themselves the same questions now. Uh, they just weren't at the time when I was working there. Secondarily, you know, classic product development is, okay, who are my users? What are the problems they have? And what are the products we can build to, to solve those problems? And to be clear, we absolutely do that at OpenAI. But then there's this other pretty fascinating approach to product development is we've got the best researchers in the world building the most powerful AI capabilities in the world, and sometimes it's like, holy moly, we just had this big research breakthrough. How do we bring this capability to humanity? Which is quite different than, like, going and interviewing someone, uh, 'cause they might not even know that that capability exists and it could solve their problems. So it's, it's a very different environment through that lens, and one of the special parts of being at both a research and a product company.

Aakash Gupta 40:56 I think just having a research arm probably also makes it different. What is it like-- how do you correctly work with the research team? How do you bring them into your overall product trio?

Jake Brill 41:06 Yeah. So I- it-- fundamentally it's, it's all about having shared vision of the, the model capabilities and the product capabilities we wanna bring to humanity. And so a lot of this is, um, understanding the research roadmap, um, trying to make sure that as we are post-training the models, that they're capable of unlocking the capabilities, um, that match the products we're trying to bring to the world. So, you know, I mentioned Canvas earlier. Like, that does- doesn't just, like, come out of the box. You have-- there's a lot of collaboration between the product team building Canvas and the research team that's building it so that it can really shine for collaborative editing of a document, for example. Um, so in my experience, the products that are most successful, there's like a really, really close collaboration between the engineering product and research teams so that there's cl- extreme clarity on w- h- you know, what are the problems we're trying to solve and, like, what, what is the product experience we wanna enable for the world?

Aakash Gupta 42:05 Hmm. Yeah. Because I guess the best products will probably be ones that are using the most recent research breakthroughs, but also kind of sculpted in a way like Canvas for some set of user problems that is new or opening up some new ground for OpenAI.

Jake Brill 42:20 That's exactly right, yeah.

Aakash Gupta 42:22 Very interesting. What are the key rituals that you guys have as a product team?

Jake Brill 42:29 You know, I, i- many of the rituals are, are similar to rituals you would find elsewhere. We talked about one earlier, which is that we are very Slack heavy.

Aakash Gupta 42:36 Mm-hmm.

Jake Brill 42:37 Um, and we do a lot of information sharing and collaboration, uh, over Slack. We also, you know, uh, we, we really strive to have everyone in the office three days a week, so there's a lot of product development, which is just like, "Let's grab a whiteboard, um, and let's work on this together." I really enjoy that. You know, I, I, I loved my time at Instacart, and my last couple of years, uh, we were fully remote, and I had a great experience being fully remote, but there is something special about being in person with, with your collaborators and being able to have that real-time conversation. Um, we, you know, there's classic, like we do product reviews, we have recurring meetings. The product reviews are interesting because, um, you know, oftentimes when you see companies grow really fast, one thing that changes is there becomes a lot more process and a lot more polish around product reviews, really like expansive, uh, product review decks. And one of the things I'm-- I really appreciate about the way that we've been growing is we try to keep our product reviews, like they are, uh, th- they're not nearly as fancy as that. Like, people come in, it doesn't matter what level you are, you have-- you can talk directly with, with leadership. Um, you don't have to have a fancy slide deck. You bring in the product, you bring in collateral around it, and you have like a, like a very high integrity discussion on it. Um, and then I think, as I mentioned before, one of the rituals is just like constantly asking yourself, are you using, are you using our technology as much as possible?

Aakash Gupta 44:01 Mm.

Jake Brill 44:02 And that not only means in terms of developing the products, but we do a lot of dogfooding internally. Like, as soon as something is ready to be tested internally, we roll it out to employees. We heavily encourage people to test things out because that is one of the best ways you can answer the question of, is this any good?

Aakash Gupta 44:16 Mm-hmm. And if I think about that product review process being high integrity, not requiring a slide deck, there must be a certain culture of trust then within the organization. Trust that, hey, even if they haven't laid out all the data that they looked at behind this product decision that they did that data work, how do you-- what is-- how do you create that culture of trust?

Jake Brill 44:40 Yeah, and m- maybe I should clarify. We, we absolutely do include the, the data i- in the product reviews. It's just that, you know, we want a classic document as opposed to-

Aakash Gupta 44:48 Mm

Jake Brill 44:48 ... a very pretty slide deck. The way you create trust, however, there's a couple things. One is, you know, um, one way to get trust is you just hire the best people in the world.

Aakash Gupta 44:59 Mm-hmm.

Jake Brill 44:59 Because you know that people are gonna think through things very s- uh, very thoroughly and very intelligently. Two is you sort of communicate transparency-- transparently, both within the, the, uh, review and then afterwards we sort of share, "Here's the discussion we had and here's sort of next steps coming out of it." Um, and then lastly, you know, I think a lot of it comes from leadership. You know, create a space where you hold people accountable for having honest and hard discussions, but also in a very supporting and humble way so that people feel like, "Hey, you know, I may have been wrong about something," but there's no shame around that. Like, we're trying to solve s- some pretty gnarly problems for the first time ever. It's like, we're not gonna get it all right the first time. And so making sure that people feel comfortable, um, having conversations and understanding they may get directed to move in a different direction, but that that's not something they should be embarrassed about.

Aakash Gupta 45:51 Mm. What's OpenAI's take on experimentation? Are you guys experimenting with everything? How do you handle that?

Jake Brill 45:58 Um, we experiment a ton. Um, you know, when you have a product that's used by a tenth of humanity, uh, every week, um, th- th- like a lot of responsibility comes along with that. Um, because small changes, you know, may have like large downstream impact in how people experience the product or how they, um, uh, how, how it fits into their day-to-day l- day life. So, uh, I would say over my two years at the company, we've become, uh, we've really leaned into experimentation. We've become very rigorous around how we analyze data. Um, and, you know, within Integrity, our experimentation looks somewhat similar to cl- other classic product teams, and in some ways it's a bit more unique. You know, we do classic AB tests. Uh, we do do long-term holdouts. But, uh, th- one thing that's a bit more unique about this side of the world is we will also, because... F- let me give you an example. Like, let's say we wanna roll out logic to, uh, mitigate, uh, account takeovers or what we call ATOs. Um, rather than just like turn on a rule live for everyone or turning it on at 50%, one of the things we may do is run it in shadow mode. So we'll like turn it on and just log what the, what the hits to that rule might be, and then we either, through some other automated or manual process, sort of validate, "Hey, did this match our expectations?" before we then roll it out to production. So across the product org, we do a lot of experimentation.

Aakash Gupta 47:20 So we briefly talked about this, but I wanna get a little bit deeper on this. What does working in Integrity product look like versus a regular product manager?

Jake Brill 47:29 Yeah. I, I mean, being a platform PM is just, you know, it's like a Venn diagram. You know, there's a, a decent amount that looks the same, but then there's some stuff that's rather unique about being a, a platform PM at OpenAI. The first thing is that we are building systems that work regardless of who the end user or the end product is. So to go back on that example I mentioned earlier, the deploying logic to mitigate account takeovers, we don't just wanna build that like one ATO model that works on ChatGPT and then one for people when they're logging into the platform dashboard, or one for people that's working, uh, uh, who are logging into Sora. Like, that would be extraordinarily inefficient and it, it, it like that's not, that's not the w- right way to build leverage systems. So we build product systems and tools, and the goal is they work everywhere.

Aakash Gupta 48:22 Mm.

Jake Brill 48:23 And that's a bit different from many other PM roles at the company, which are focused on like, "Oh, I'm building this thing for ChatGPT or this thing for developers." The second thing I call out that's different about being an Integrity PM at, at OpenAI is-We kind of work on every launch, not every launch, but like pretty close to that. And so what that means is we, we get to s- like be involved with a lot of cutting-edge stuff, um, in a way that maybe not every PM team does because they're like really focused on, on, um, their product or, or their specific product surface. And that's challenging because, um, there's a lot of inbound requests to the team. But it's also super exciting because we get to, we get to see all the incredibly cutting-edge stuff that we're building across OpenAI, and we get to play, you know, play a role, sometimes very small and sometimes very large, in the vast majority of product launches that we have. And I think the third thing I would say that's different about being an integrity PM at OpenAI is, you know, classically working in the like trust and safety platform space, it's, uh, I, I... It's often compared to being like a soccer goalie or an offensive lineman.

Aakash Gupta 49:33 [laughs]

Jake Brill 49:33 It's like if you're doing a really great job, people don't notice, and it's only if like things go sideways, uh, that, that the, the light shines on you. It c- you know, as I mentioned before, it could be like payments just start failing, or people can't log into their accounts, or we mess up and we ban people we shouldn't.

Aakash Gupta 49:50 Mm-hmm.

Jake Brill 49:50 Um, and so what that means is a lot of the work that we g- do isn't gonna be on the front page of the Times or TechCrunch, or there's not gonna be a ton of threads about it on Twitter unless things go bad.

Aakash Gupta 50:02 [laughs]

Jake Brill 50:02 And so, uh, it requires a sh- very low ego to work on this team because what, you know that what you're doing is you're building really important technology, but that technology is setting up our, our, the like core products, um, to have those articles written about them on, on the Times. And so you have to, you have to accept that, uh, you won't get as much shine, but that doesn't mean your work isn't just as important.

Aakash Gupta 50:24 Mm. So when I asked PMs in my newsletter to rank the dream companies they wanted to work for, OpenAI was way ahead of the rest of the pack. Like, it was like, I think it was like 600 1st place votes, and the next was 200, okay?

Jake Brill 50:41 Oh my goodness.

Aakash Gupta 50:41 So you guys had a huge-

Jake Brill 50:43 That's awesome

Aakash Gupta 50:43 ... delta. You guys, everybody wants to work at OpenAI. So if somebody were wanting to work on your team, what would be your advice to them? W- like, outside of, you can't just sculpt your own background-

Jake Brill 50:54 Yeah

Aakash Gupta 50:54 ... but what are the things you should be doing, you know, side projects, learning, passion, to break in?

Jake Brill 50:59 Yeah. Um, there's a couple things there that I would, I would call out. One, you know, when I interviewed with OpenAI, uh, around two years ago now, consumer AI wasn't what it is now. And so the fact that I, I hadn't had a ton of experience working with AI-based consumer products outside of ChatGPT, obviously, that wasn't, you know, that, that could be explained away. These days, like you, you really need to have a facility with AI products and models, and there's no excuse for not. Uh, you know, it's so easy to use these products. It's so easy to vibe code. It's so easy to, um, to play around with these APIs that I think first things first I would say, like start playing around with these models and building things.

Aakash Gupta 51:41 Mm-hmm.

Jake Brill 51:42 Second, I would say is, um, you know, the, one of the best ways of, uh, joining OpenAI is if, if there's someone that you know personally who's worked there. And, um, you know, the bottom line is when we post jobs publicly, they, a lot of people apply to them.

Aakash Gupta 51:57 [laughs]

Jake Brill 51:58 And so it's, uh, you know, you have to really s- sift through to find out who's gonna be the right, uh, the right fit. And there's two things that really help us know if someone's the right fit. One is if someone at the company can, has worked with you and, and can, um, speak to your, to your skill and your t- and y- and how great a fit you would be. And the second is, you know, you have a presence pr- particularly LinkedIn, but it could be your resume, it could be elsewhere, that sort of speaks to, A, your comfort with AI, and B, sort of the specific experiences you have that are particularly relevant for any job that's been posted. The like, oh, you know, in, in integrity, uh, if we've posted a role, you know, fighting fraud, well, if you've fought fraud in, in your past, then like making that very explicitly clear is really helpful for us to figure out if you're the right fit.

Aakash Gupta 52:46 Yeah. I mean, if you're hiring the best in the world, I imagine you guys are very focused on people who have functional and ideally some domain experience within O- AI.

Jake Brill 52:55 The, uh, that's correct. You know, w- what I've found is, uh, at least for the f- folks that I, I've, uh, hired for the integrity team, the best candidates, um, have a pretty well-rounded background. They, they do have familiarity with AI. They have worked in the specialist space. But many of the best folks also have worked in other spaces.

Aakash Gupta 53:13 Mm-hmm.

Jake Brill 53:14 And so they've got like a pretty broad horizon. They're not just, "Oh, I..." You know, to use that same example, "I've, I've only ever worked on fraud in my career. I've worked on fraud, and I worked on ads, and I worked on, um, growth," things of that nature. That, well, the benefit of that is it means that you, you can think about problems from a lot of different perspectives, which is really important because we're dealing with a lot of ambiguous problems that have never been solved before. So having a pretty broad perspective is really helpful as you so- uh, encounter these problems for the first time.

Aakash Gupta 53:40 Mm-hmm. And I often coach people, like if you wanna make it to your dream company, think about going into the backgrounds [laughs] that the company is hiring from. And if we look at the product team backgrounds at OpenAI, there's a ton of Meta and Facebook presence there. So could an interesting strategy for somebody be to like kind of work their way through a big tech job? Would that help them eventually get a job at OpenAI?

Jake Brill 54:04 It could, but it, it, that, it's not a guarantee, and it's, it's not a necessity. Um, you know, one of the benefits o- of, of a FAANG-type company is you're getting to work at incredible scale. You work with super talented people, and you know that like they have a really high hiring bar. But, um, we also find that people who've worked, uh, people who are startup founders or have worked in hypergrowth companies that aren't at the scale of FAANG companies, that's really helpful too because like the pace of development, the, um, frankly like the, the fact that w- we sort of take pride in the fact that we're not process-heavy. Like, that's really important too for being successful at OpenAI because, yes, you can develop a lot of skills at these excellent companies. Certainly, I wouldn't be s- in the job I have now if not for my time at Facebook.But that alone is not going to, to mean that you're necessarily going to, to be, be able to be successful at OpenAI. It's, it's a, it's a wide range of skills that you will get both at large and smaller scale companies.

Aakash Gupta 55:04 Yeah. There's probably thousands of PMs at Meta and tens at OpenAI, so just by the math, you know, it's not a guarantee. But it's interesting then that there are certain backgrounds you've identified though, whether it's hypergrowth company or big tech, that will help you for OpenAI's scale and culture.

Jake Brill 55:22 Yeah. Yeah. You know, there's no one size fits all. Um, it, it, it is definitely the case that we have a number of, of ex, uh, Facebook and ex-FAANG, um, PMs at, at OpenAI, but we also have a ton of people who have not worked at those companies who are absolute rock star. I mean, like, uh, you know, you, you look at some of our product leads. Actually, if I think about our product leads off the top of my head, I don't-- I think I might be the only one who's, who's worked at Facebook.

Aakash Gupta 55:49 Okay.

Jake Brill 55:49 Um, so i-i-it, there's no one size fits all to be a successful PM at, at, at the company.

Aakash Gupta 55:55 For sure. So what was your story for breaking into OpenAI?

Jake Brill 56:01 [laughs] Oh, man. Uh, I certainly wasn't expecting it. You know, I, uh, Instagram was an incredible experience. I'm like so grateful I got to work on fascinating problems. Um, I got to work with brilliant people, and I have to say, the experience of being there during COVID is like super unique, and I am forever grateful I got to experience that firsthand, as intense [laughs] and wild as it was.

Aakash Gupta 56:22 Yeah.

Jake Brill 56:23 And so in 2023, you know, I had my like whole next couple of years planned out, and it did not involve leaving Instagram. I had, um, uh, I had, uh, qualified for my recharge. My wife was about to qualify for hers at her gig, and we were gonna like time them up and take it together. And then I, um, went on pat leave. My daughter was born in the spring of 2023, and I was a couple months into pat leave when I got a message from a former colleague of mine from Facebook. We hadn't talked in years, and, uh, he was in a, a new leadership position at, at OpenAI. And I-- when I saw the message, I was like, "Oh my goodness, what a, so great to hear from you, and also, like, I'm flattered you'd, you, you'd even think of me." And I went into the call being like, "It'll be great to catch up with him, and I'll sort of like courtesy say, and I'll say, 'Thanks, but no thanks,' and like, 'Great to see you.'" And I got off the call after thirty minutes, I was like, "Oh my goodness, this is, this is kinda mind-blowing," the sort of stuff that we were talking about. And I reached out to a buddy of mine who was working in the sort of trust and safety space at the time at the company. We spoke for an hour. And I very vividly remember that night lying in bed, and my mind was racing, and I could not sleep. I had so many thoughts racing through my head. It was like, "Wow, the scale of the problems, the sophistication of the problems that this company is trying to solve, um, is mind-blowing. The fact that this company is talking about safety so f- uh, s- so regularly in a way that they want it to be part of their brand and their products is, is really inspiring." Um, and, and like the fact that they even want to talk to me [laughs] in the first place was like deeply flattering because it's OpenAI. They can talk to anyone, you know? And I remember b- maybe it was because my daughter had just been born. I, I, I, I was being very circumspect to like s- in thinking about being like, "This kind of feels like what I've been working towards my whole career," and like, "What sort of world am I bringing my daughter into? And like, w- what role am I gonna play in that world?" And, and I did a lot of thinking. I was like, "I think this is the most meaningful thing I can be doing with my time, if I'm lucky enough to even have e- more conversations with the company." And so I spent a couple months in the summer of '23. You know, the, the shape of the job changed a couple times. I think there were like three or four different hiring managers over the course of the process.

Aakash Gupta 58:40 Mm.

Jake Brill 58:40 Um, and when they made the offer, I was like, "Uh, yeah." [laughs]

Aakash Gupta 58:46 [laughs]

Jake Brill 58:46 Like, "Who, who, who am I to turn this down?" So it's, it was not something that was on my radar in, in the slightest, and it just so happened that, you know, someone thought of my name from having worked together years ago. But you know, they, it was a, it was a very demanding interview process, and I think on some level, the fact that I was on pat leave really helped because I was either like changing diapers [laughs] or I was preparing for an interview, and I did that for like two months, and I, I think that was really helpful for it, it ultimately turn, turning out to be a, a hire decision on the company's behalf.

Aakash Gupta 59:17 Oh, wow. Two months of preparation. And it sounds like you took this intensely, like seriously.

Jake Brill 59:21 I took it very seriously because, you know, um, a- as I said, it sort of felt like I had been w- working my whole career to get to someplace like this. Do, do-- You know, I've-- I was never hired to be the first trust and safety PM at Facebook or Instagram. It just so happened that I ended up being the first trust and safety PMs at those companies. Like opportunities presented themselves, the org changed in such a way that, that, um, that ended up being the role I filled. And so this was the first time I was talking with a company and they were like, "We'd actually explicitly like you to join for this job."

Aakash Gupta 59:53 Mm.

Jake Brill 59:53 And so, uh, I figured in joining in that context, I just really wanted to make sure I understood, so what are the types of problems the company was thinking about? And you know, I was by no means an expert on this sort of like what are the safety risks in AI models. And so I just wanted to make sure I put my best foot forward for those conversations.

Aakash Gupta 1:00:09 Mm. I think that's really important advice, though, for people. When you get those really awesome interview opportunities, [laughs] really s- do everything you can. You may not be an exact expert in their industry, but you can be humble and learn also. So that's really, really inspiring. So you spent two months on it. If I were to reverse engineer though, I think part of it is then that you had positioned yourself really well in your career to be that first trust and safety PM at Instagram and Facebook because they wanna hire the best in the world. So you had that opportunity to create those career experiences. If you look back, what were the essential ingredients, maybe we can go story by story-

Jake Brill 1:00:46 Sure

Aakash Gupta 1:00:46 ... that positioned you for those roles?

Jake Brill 1:00:49 You know, if, uh, I moved to San Francisco almost 20 years ago. Um, it'll be 20 years next week. AndIf you had told me 20 years ago that this is what I'd be doing, I, I... That, that wouldn't have made any sense to me because I didn't study this stuff. I, I, I studied, like I, I, I s- studied at a small liberal arts college in the Northeast. I majored in psychology and Latin American studies. [laughs]

Aakash Gupta 1:01:10 Oh, wow.

Jake Brill 1:01:11 And when I graduated, like what I wanted to do was go into the mental health field. And when I moved out here 20 years ago, it was to establish my residency so I could apply to get an advanced degree in psychology at a UC. And so, um, I was like full steam ahead on psychology and mental health when I came out here. And the story of me getting to Facebook was basically I'd spent a couple of years working in the mental health space and had a quarter-life crisis and I was like, "I don't think this is the right thing for me." I'm like not super happy. I don't feel fulfilled in the way I thought I would. What am I gonna do with myself? And it just so happened, um, I had a... Some of my really good friends from my first couple of years living in San Francisco were in the customer support team at Facebook, and they loved it, and I was a huge fan of the product. I had moved from the East Coast and left a lot of my best friends behind, and Facebook was this lifeline to keeping in touch with them. And so when I went through the interview process, it was not 'cause I knew I wanted-- I didn't even know what a product manager was, but I thought, you know, worse comes to worse, I do this for a little while and I realize, "Oh, actually I do wanna go back into mental health." And this was a good way to, to, you know, just like the grass isn't always greener. Um, and I ended up joining in customer support and it was like a, you know, not a super glamorous job. My first year I was doing a lot of really like, you know, it was like changing people's passwords, approving their name changes, stuff like that.

Aakash Gupta 1:02:33 Mm.

Jake Brill 1:02:34 Um, and towards the end of my first year, I end up moving into a role, uh, leading our, our privacy team in, in support and that team-- When we had our first phishing attacks on the site, our team was sort of at the front of the operational response. And, um, when we created the site integrity team at Facebook, we should show you a little bit of how long ago. It was called Site Integrity 'cause all Facebook had was a website at that point.

Aakash Gupta 1:02:58 Oh, wow.

Jake Brill 1:02:59 And, uh, I applied for a role to be a project manager on the team, but like I didn't know what a project manager was, and Facebook certainly didn't. I was the first project manager at the company, and if I... You know, ultimately it was like I, you know, I worked hard my first year and I, I like put myself in a position to be in a leadership role, but I never would have guessed it was like, oh, because like there's gonna be a phishing attack. I didn't even know what that was. And so when I moved into project management, I, I like was sort of figuring out on the fly what I was doing. And for my first year, I don't think what I did was actual project management. I think it was like, it was like coordinating our response to spam and malware and, and, and phishing attacks, and that had a lot of different shapes and it was like, just do what's necessary, and that could be like training up some of our support counterparts in how to do certain workflows or, um, uh, like making sure like, you know, this engineer's working on this fire and this engineer's working on this fire. [laughs] Sometimes it was just like running command line scripts to clean up a bunch of bad stuff.

Aakash Gupta 1:03:56 Mm.

Jake Brill 1:03:57 And after about a year, it became clear that we had to stop being so reactive. We had to start building user-facing products to help explain like, "Hey, your account got hacked. Here's what you do." And then I was like, "Oh, I think we need to start building products." And, but like that wasn't my job and I didn't know how to do it. And I remember I was chatting with, uh, one of my, um, counterparts, um, and he gave me some of the best advice I ever got. And I haven't talked to him for years, but Dan Levy, if you're listening, if this makes its way to you, thank you for the advice.

Aakash Gupta 1:04:29 [laughs]

Jake Brill 1:04:29 I've given it to many other people and it was, "If there is a job that you're interested in and there is an opportunity to do that job, even informally, just start doing it and good things will happen." And so I kind of in a de facto fashion just started being the product manager for the site integrity team and, you know, the products I, I, I built were of varying quality. Some of them we turned off pretty quickly, and some of them are still there in some form. Like if you ever, uh, you know, if, if you ever log in from somewhere suspicious and you get asked to verify your identity, you can blame me. I sort of built the first version of that up for Facebook.

Aakash Gupta 1:05:04 Oh, cool.

Jake Brill 1:05:05 And I remember I'd been doing like de facto product management for a while, um, when, uh, like a, a new lead for the integrity team joined and I was like, "Hey, I've been doing this job. Like can I like formally become a PM?" And I remember having some conversations and, and the answer was like, "Jake, we think you're smart, but like you've never had any formal training or mentorship with this. Like, let's give you some of that before you actually transition to PM." And yeah, my ego was bruised a little bit, but like, boy howdy were those people correct and, you know, I learned a lot. Um, I had two really important mentors, uh, these two gentlemen named Dan Rubinstein and Arturo Bejar, and they like really invested in me, helped me learn a lot about product craft, strategic thinking, how to bring teams along with your vision and, you know, by, you know, I don't know, maybe like six, seven months later, I went through the interview process and I think if I'd gone through it when I originally wanted to, I would've just fallen flat on my face. But because I got more hands-on mentorship, that set me up to be successful when I interviewed. And then when I interviewed and ultimately passed it, I think I was the first person to ever tran- transfer internally, interview and move over to PM at Facebook, and at that time there were maybe like 30 PMs at the company. And so when I think about what like put me in a position to be, to move into PM there, it was a lot of just like rolling up my sleeves and doing what the business needed. And then also being open to coaching, um, and finding mentors who were really willing to invest in me, and then learning from where I did well and where I didn't do well. Um, yeah, so that's, that's my Facebook story. It was, it was great, but like I never would've guessed, uh, when I started at Facebook that it would've ended up being a PM in the integrity space.

Aakash Gupta 1:06:46 Facebook's first internal product management hire. That's amazing. That was probably a lot v- much harder to do than the average move. So when we think about like taking on additional responsibilities, I think the first thing that probably people is popping into their head is like, "But I'm already working 40, 50 hours a week. How do I take on additional responsibilities?" What did that look like for you?

Jake Brill 1:07:07 Um, it meant working more hours in some cases.

Aakash Gupta 1:07:09 Yeah.

Jake Brill 1:07:09 But also like, you know, a huge part of being a PM and being a leader is like-Figuring out what is the highest leverage thing for me to spend my time on, what is something that's really important but you can, uh, delegate out to other people, and then what are some of the things that are important but not urgent, and you sort of just, like, put them on the back burner. And, you know, there's some amount of trial and error. But, like, you know, at that point it was really clear that even though it wasn't my job, we needed to start building user-facing products. And so, like, that was one of the more leveraged places for me to spend my time on. And so, you know, e- as I mentioned before, it was like part of the way I'd been spending my time was interfacing and training up some of u- my operational counterparts on sort of, uh, how to respond to the, to the phishing attacks, and then it became the, well, can, uh, we, can either the, someone on the engineering side tr- train them up or can we have a, you know, a leader on the operational side sort of take over this responsibility?

Aakash Gupta 1:08:05 Okay. So that's the Facebook story. What about Instacart?

Jake Brill 1:08:11 Um, so when I left Facebook, I was there for about 10 years, and it was just, like, an incredible experience. But by the end of my time, you know, I was ready for something new and, and I left. And I, I, I had this plan that I was gonna take a couple months, plan a wedding. I had just gotten engaged, you know, recharge, and then start up working again. And Instacart wasn't on my radar then. But I ended up being out of work for two and a half years because a couple months into my time after Facebook, I suffered a very, very serious brain injury, and it took me about two years to recover from it. Um, and thankfully, I, you know, I, I found the right rehab program after a while, and I went through that rehab program. And when I got to the other side of it, I wasn't positive I was healthy enough to work again. You know, I hit all these milestones in terms of I can do... You know, I can read for this long before getting a headache, or I can drive for this long without getting a headache, or I can, m- you know, look at a screen for this long without getting a headache. But I still didn't know that I was ready to work again. But I figured, like, the... I think at this point, the best way to f- to find out if, um, if I'm ready to work is to get a job. And Instacart really resonated with me because I had had a firsthand experience of what it was like when I couldn't drive, 'cause that was part of my concussion recovery, is I couldn't drive. And so all of a sudden I'm like, "Oh, I've taken for granted that, you know, up until m- me having these health issues, I could just drive to the grocery store and drive home, and then have everything I wanted." And then for two years I would literally walk to the closest Safeway and walk back with stuff. And I had this whole new, uh, empathy for what it was like for people who had mobility issues, either physical issues or they didn't have access to a car or public transportation. And, you know, I remember my conversations with Instacart and I went in being like, "Eh, how, how hard can this n- grocery delivery thing be?" And I was like, oh my God, it's so hard. It's like so complicated. And that was fascinating to me. Um, and so when I joined, it was... You know, I was talking with a couple companies, and it was really important to me that I joined a company as an individual contributor because, um, I was like, "I don't know if I'm gonna be healthy enough to do this job, and I don't feel comfortable. Like, I need to be responsible for my own career. I don't feel, feel comfortable being responsible for supporting someone else's career success. I have to really nail my own personal career success first." And so when I joined Instacart, I joined as an IC working on sort of like shopper engagement products. Um, and like very happily, it turned out that I was in a good place health-wise and, you know, experienced some success there. And after a period of time, I got offered an opportunity to, to move into a leadership position to, to build out a, a, a team, uh, that was focused on customer support and internal tooling, and I was super excited about that 'cause it was a huge opportunity and because I started my career in support, so I had a particular passion for that. But, uh, the job changed a ton over the years. The mandate, the, the, of the team grew a bunch, um, which was super exciting, um, especially during the days of, of COVID where, like, some of the things we were responsible for w- whether it was customer support or shopper safety or fraud, like those became really important when the co- when the company went from like a, you know, a niche delivery product to an essential service during a black swan pandemic. So that, that was like, that was just like such an incredible experience that I, I, I, I know for sure, you know, I, I wouldn't have gotten the OpenAI role and whatever version of success I've had at OpenAI, I learned so much during my time at Instacart that's been really instrumental.

Aakash Gupta 1:11:43 Okay. So at Instacart, again, you weren't hired for trust and safety. How did they-- How did you position yourself to be that first trust and safety PM?

Jake Brill 1:11:52 So this is like, I still vividly remember the conversation where, where my eng counterpart and I sort of came to this conclusion. As, as I mentioned, uh, I was moving into this new role where I was gonna be s- building out a team focused on customer support and, and support tooling. And as we were wrapping up some of the work that the, the, uh, our team had been doing in the shopper engagement space, my, my eng counterpart and I took a step back and we were like, "Huh, we were building this to address some shopper feedback." Uh, but I think a couple of the products we built were, were actually like focused on shopper safety and, and, uh, giving people tools to report fraud. And we viewed it through an engagement lens, but like that's actually trust and safety work. And we looked around and we were like, "No one on the technical or product side here is really like building out scaled systems and user-facing products for this stuff." We were particularly operationally heavy, and I'd gotten the company to that point in 2019. But we felt like that's probably not gonna work long term. You need to also have more, more sophisticated technical solutions. And so we made the case to leadership that like, "Hey, I know the purpose of the team was to just focus on support and tooling, but what if we expanded the mandate to also be responsible for, uh, trust and safety?" And, you know, I, I think for a number of reasons, like people sort of got why it was important as we made the case. You know, our operational counterparts were, were really eager to work. You know, we had a positive experience working with them on those first couple projects, and they were eager to have like dedicated technical counterparts. And, um-You know, I, I think the fact that we'd had a couple of wins that, that we-

Aakash Gupta 1:13:30 Mm-hmm

Jake Brill 1:13:30 ... could point to, we'd, we had built some sort of confidence with, with leadership. And so when we made this case, it was actually a pretty straightforward conversation. People were supportive of it. And then it just so happened that that was incredibly fortuitous because fast-forward three months, COVID-19 hits, and as I mentioned, all of a sudden, like, our customer support times balloon to three and a half hours, and we had to get that down to minutes in very short order. We were sending shoppers into public spaces, um, and we had to figure out ways to sh- you know, in product, make sure that they were healthy, that they were getting them PPE, that we were, uh, helping them understand best practices about how to interact with their customers. You know, we certainly saw an increase in people stealing groceries during that time. You know-

Aakash Gupta 1:14:15 Oh

Jake Brill 1:14:15 ... as often happens, you know, when a company goes through hypergrowth, the, uh, the, like, good usage goes, grows fast [laughs]

Aakash Gupta 1:14:22 [laughs]

Jake Brill 1:14:22 ... and also the bad grow- usage grows fast. And so there was a lot that we had to do to, to clamp down on, on, on that sort of, uh, us- um, fraudulent usage. And so I... Y- it was one of those things where, like, I couldn't have guessed it would [laughs] become that important that quickly, but I do think we were onto something that we're like, "Well, it's gonna become important at some time, so let's get in front of it."

Aakash Gupta 1:14:42 Mm-hmm. Mm-hmm. And then the key, bringing this all back, right, was you had a former colleague reach out who was now working at OpenAI. And so part of it is that you are also just good at building a good reputation for yourself in your workplace. Are there any things you've learned over the years that help you do that?

Jake Brill 1:15:02 Um, I've... One, I th- I think one of the best ways you build a good reputation is you do the work.

Aakash Gupta 1:15:09 Yeah.

Jake Brill 1:15:10 Like, you build great products, you build impact. Um, I think that's, that's, like, some of the core currency of being a PM. But more so than that, I think people really get it when you care about them as individuals. Like, this isn't just, like... You're not just, like, a vehicle to, to, you know, build X. Like, you are a partner to work with on solving people's problems.

Aakash Gupta 1:15:32 Mm-hmm.

Jake Brill 1:15:32 And I think when you invest in relationships with people, um, they appreciate it, and they... And, and I know I appreciate it when my colleagues really invest in getting to know me and care about me as a person, one, 'cause that just means the vibes are good, but also that means that when we go through tough times, when we disagree or when... You know, like, I screw up all the time. And when I make a mistake, people don't think, "Oh, Jake, he's a bad person," or, "Jake, he's incompetent." It's like, "Oh, Jake's human."

Aakash Gupta 1:15:56 Mm-hmm.

Jake Brill 1:15:56 "He's, like, not going to be perfect 100% of the time, but we have some trust, and, like, I know that he's got my back and that, you know, he has proven that he's competent at his job." And I, I think that ma- ma- really lays a strong foundation for, you know, people wanting to work with you again.

Aakash Gupta 1:16:12 Mm-hmm. Do good work plus build real connections.

Jake Brill 1:16:16 Yeah.

Aakash Gupta 1:16:17 Okay. So as I mentioned, I polled my followers. I asked them what I should ask you, and we have to end on these topics because they're just so excited to hear your answer. What does the PM role look like in five years?

Jake Brill 1:16:33 Um, I'll start with what I think looks different, and then I'll sort of wrap with what I think looks the same. So we've talked about a couple things that I think are going to be different. One, AI prototyping. Like, this is a new world where we as, as PMs are empowered to not just use words to express the thoughts that we have, but also use code to express the thoughts that we have. And I think increasingly the role of a PM is going to be doing prototyping. Um, you know, we're still gonna be very important to have world-class designers and engineers actually designing and building those products, but, like, coming up with a tangible version of, like, "Here's what I think we could build," I think is gonna be a huge part of the job moving forward. Secondarily, I mentioned this before, evals, evals, evals. I think that is something that PMs are uniquely well-positioned to do because w- you know, we c- we are most well-positioned to say, "Here's what the product needs to be good at, and here's how we know if the product is good at that."

Aakash Gupta 1:17:31 Mm-hmm.

Jake Brill 1:17:32 In terms of what I think is going to be the same, it's sorta, uh, uh, along the lines of the last question you asked me, which is, like, the interpersonal part. You know, um, a, a question I get a lot is, like, "What do you look for most in PMs?" or, "What do you think is most important?" And, like, my answer has been the same for the f- you know, decade plus that I've been in the role as a PM, which is I think the most important skill that a PM can have, and that, like, is going to be the case in five years, is empathy. F- fundamentally, you are building products for people who are nothing like you. You know, if you're... You know, they may live in a different part of the world. They may be, uh, you know, a different age, different gender. Um, if you're just trying to build something for yourself and you don't know the people you're building for, you are just gonna miss the mark completely. But I, I also call out empathy because I think it's such a critical part as a leader for a team. You're working with engineers, with designers, with people from comms, marketing, data scientists, uh, other PMs. And, like, you know, the w- what motivates them, the way that they communicate, um, it's gonna be different for everyone. And s- having that sort of context and having clarity of leadership and the deep ability to understand who a person is and how to help them do their best work, I, I think that is going to be even more important in the future as other parts of the job change and maybe go away.

Aakash Gupta 1:19:05 Mm-hmm. In the future, are PMs gonna have to learn how to manage not just humans, but agents?

Jake Brill 1:19:11 Absolutely. But I don't think that's just PMs. Like, I think that's gonna be a lot of the way jobs change in the future, is that absolutely you're gonna be interacting with, managing, collaborating with, with other people. But I think that's very much the case with agents, and so there's gonna be a new skill set that people need to develop in terms of, like, how to give the best prompts to get the outcomes you want, how to... Like, what sort of, um, uh, actions you're comfortable delegating to an agent versus you say, "No, I want to be in the loop for, for this given action or decision." Um, so, so I think agents absolutely for PMs, but I, I think that's kind of where a lot of, uh, of jobs are gonna evolve over time as AI and agentic experiences become more common.

Aakash Gupta 1:19:55 I could do 90 more minutes of questions, but this has been a master class. Jake, thank you so much.

Jake Brill 1:20:00 Thank you so much for having me. This was great.

Aakash Gupta 1:20:02 If people wanna get in touch or learn more, where should they find you?

Jake Brill 1:20:06 Uh, best place to find me is on LinkedIn. My name is Jake Brill. I'm happy to chat with people. Uh, and again, thank you so much. I really enjoyed our conversation.

Aakash Gupta 1:20:16 All right. See you guys later. So if you wanna learn more about how to shift to this way of working, check out our full conversation on Apple or Spotify Podcasts. And if you want the actual documents that we showed, the tools and frameworks and public links, be sure to check out my newsletter post with all of the details. Finally, thank you so much for watching. It would really mean a lot if you could make sure you are subscribed on YouTube, following on Apple or Spotify Podcasts, and leave us a review on those platforms. That really helps grow the podcast and support our work so that we can do bigger and better productions. I'll see you in the next one.