Never write an update again: Notion's AI-powered engineering meetings | Ryan Nystrom

1. 0:00 – 2:48

   Introduction to Ryan Nystrom

   1. RNRyan Nystrom0:00

      One line that I've been putting in my prompts lately is, "I literally don't know what I'm doing here. You gotta explain it like I'm a five-year-old." I didn't start with writing code. I didn't start with anything. I just started with an empty markdown document. I actually just opened up Whisper and just started yapping about how this feature should work. I gave the yap session to Codex, and it was like, "Write a spec." I then opened up Codex again, pointed it at this spec file, and I said, "Build it," and basically one-shotted this.

   2. CVClaire Vo0:27

      I've been in software engineering for 20-plus years. We were writing these documents, and we were sitting in meetings with other engineers debating the merits of one implementation versus another. And now, no more waiting for the meeting, no more waiting for review.

   3. RNRyan Nystrom0:41

      I'm not a CI expert, but I kinda know what I want. And so other folks were kind of like, "Can you just bring some of your, like, puppy dog energy to, like, CI, and just see what we can do?"

   4. CVClaire Vo0:53

      Your AI, your agent, is never gonna complain when you ask it to do this five minutes before the meeting starts.

   5. RNRyan Nystrom0:58

      It is more relaxing, and it's more fun, and I feel like I'm getting more done. It's weird to have this, like, win-win-win.

   6. CVClaire Vo1:04

      They do the triangle, and they're like, "Pick two," and you're like, "No, I wanna pick all three."

   7. RNRyan Nystrom1:07

      Give me the whole triangle. [laughs]

   8. CVClaire Vo1:08

      Give me the whole triangle. [upbeat music] Welcome back to How I AI. I'm Claire Vo, product leader and AI obsessive, here on a mission to help you build better with these new tools. Today we have Ryan Nystrom from Notion, and he's gonna show us, as an engineering manager, how you can never prep for a standup again. We're also gonna see how you can get a background agent to write code for a fix your friend texted you, and how spec-driven development really works in a code base at scale. Let's get to it. This episode is brought to you by WorkOS. AI has already changed how we work. Tools are helping teams write better code, analyze customer data, and even handle support tickets automatically. But there's a catch: these tools only work well when they have deep access to company systems. Your copilot needs to see your entire code base. Your chatbot needs to search across internal docs. And for enterprise buyers, that raises serious security concerns. That's why these apps face intense IT scrutiny from day one. To pass, they need secure authentication, access controls, audit logs, the whole suite of enterprise features. Building all that from scratch, it's a massive lift. That's where WorkOS comes in. WorkOS gives you drop-in APIs for enterprise features so your app can become enterprise-ready and scale upmarket faster. Think of it like Stripe for enterprise features. OpenAI, Perplexity, and Cursor are already using WorkOS to move faster and meet enterprise demands. Join them and hundreds of other industry leaders at workos.com. Start

2. 2:48 – 4:30

   How AI has upended 12+ years of the same working routine

   1. CVClaire Vo2:48

      building today. Ryan, welcome to How I AI. I am really excited because you're gonna show us, I think, start, medium, advanced mode on some AI coding stuff. And so before we jump in, how has AI just changed how you live your life at work?

   2. RNRyan Nystrom3:11

      I mean, it's... It has completely upended the way I work. Like, I, I've been doing this... I did a lot of, like, mobile iOS work in my past, and I worked the same way every day for, like, 12-plus years. And then in the last year, I have changed IDEs, terminals, tools, like, whatever, like, 10-plus times. Um, so it's, like, really weird and scary to be, like, changing this stuff so much, but it's also... I feel that, like, I've been doing this for a while, and I'm feeling, like, so much, like, joy and freshness and, like, newness in this that, like, I wake up every day super excited to, like, tinker and build things. Uh, and I'm also, like, working faster and harder than I feel like I ever have, but, like, in a, in a good way. I know people are, like, kinda freaked out about, like, "Ah, everything's changing. The pace is up," but, like, it's, it's really energizing for me.

   3. CVClaire Vo4:08

      Well, you are not alone. I think every How I AI guest has come on and said, "I'm having more fun, I'm working faster, and everything is different." And what I love about what you're gonna show us is it's not just the set of tools which I think has changed or how we write code has changed. How you, like, run a team has changed, so... And I think for the better, you

3. 4:30 – 9:00

   Project Afterburner: Notion’s push to cut CI time to a quarter

   1. CVClaire Vo4:30

      know. So I would love to see how you're using Notion AI to actually run teams differently.

   2. RNRyan Nystrom4:36

      Yeah, for... So for context, I, I manage a team of, like, six, seven people. Um, I'm, like, an engineering manager or a tech- technical engineering, tech lead manager, or whatever we call it. I manage people, and I write code-

   3. CVClaire Vo4:51

      When you call it that

   4. RNRyan Nystrom4:52

      ... uh, is, like, my role, which I love. And I've run a bunch of different projects here at Notion. The, the one I'm gonna show today, uh, we've, we've nicknamed Afterburner. Like, kind of a quick backstory is I've been kind of vocal about our, like, DevX CI for a while. Um, I've just... I've worked at places where it's really slow. I've worked at places where it's really fast. And I came here, and I was like, "We're kind of, like, in between, but I feel like we're, like, slower than we need to be." Uh, and eventually this, like, caught up to me, and somebody was like, "Could you just come fix it and, like, come work on it?" And I'm not, like, a infra expert. I'm not a CI expert, but I, I kinda know what I want. And I think also most importantly, the, the group that I manage and the, the org that I work in, we're a little notorious for being, like, really fast and, like, very, very AI-pilled. And so other folks were kind of like, "Can you just bring some of your, like, puppy dog energy to, like, CI and just, like, see what we can do?" And that's what we've, that's what we've done. So we had this really aggressive goal to cut our CI into, like, a quarter of what it is. Um, we're on the path to, to doing that. But, so what, what I wanna show you is a little bit about how-We run projects in Notion and how I'm using, um, AI to kind of streamline the, those projects. So what we're looking at here is, like, our, like, project hub called Afterburner. And so in here I've got all this documentation. I've got databases. Um, we'll look at some meetings. I have, like, a automation that looks for any sort of, like, little wins if we knock off seconds from different jobs or whatever, and we keep it, we keep track of them in here. But what I wanna show you is basically how we run our meetings. Our small group, we run a standup every single day, and doing standups where everyone just, like, is kind of, like, dead-eyed and going around being like, "I did this, I shipped this change," or, you know, "No updates for me, thanks," is, like, painful. And, uh, in my opinion, like, a huge waste of time. I wanna, like, get to the, the meat. So we have this kind of, like, automated meeting template that shows up every single day that we run our meetings, which is basically every day, and it starts blank. And then what I have set up behind the scenes, uh, is a, um, custom agent. So we chipped this Notion AI custom agent stuff a little bit ago, and this runs right after the meeting template gets generated, and it looks through all of our, like, Slack conversation in the last 24 hours, any tasks in Notion that we closed, any, um, pull requests that we've, like, merged, just, like, all sorts of context. Oh, and it looks at, like, yesterday's meeting transcript as well. And then it compiles it basically like a pre-read. And, um, this is a, this is one from a week or so ago, and it, yeah, it shows... It pulls metrics. It can show us, like, what our latest CI time is. Uh, it shows some of the things we've decided, shows, like, progress on different, like, projects or, like, different things that we're trying to, like, make faster, bugs, feedback, open questions, ta- like, anything that's, like, of concern. And, like, I can basically work up until, like, the minute of our meeting without having done a bunch of, like, prep. And then we all get on a video call, and we look at the screen, and we're like, "Okay, here's what we need to talk about." And we'll, like, hit each bullet. We have, like, meeting notes that we run down here, um, within Notion, and, like, all the context is basically, like, captured. All of the, like, agenda is, like, set up for me. So we spend the entire time talking about, like, problems, decisions, wins, findings. Like, what are we gonna work on next? And it's less the, like, "Oh, I did this

4. 9:00 – 11:10

   Why high-frequency, high-quality meetings beat lower-frequency standups

   1. RNRyan Nystrom9:00

      thing."

   2. CVClaire Vo9:00

      Yeah. What I wanna call out for folks that are maybe listening and not watching is this is a very detailed meeting n- like, kinda like pre-read/status update, which if you're a, you know, TLM or an EM, running good meetings unfortunately is part of, part of the job.

   3. RNRyan Nystrom9:18

      Yep.

   4. CVClaire Vo9:19

      And there's a really big difference between a good standup and a bad standup, and I think the, the ones you described, these, like, wrote, like, "I wrote this PR. Today I'm gonna start on this," and then basically, like, a, a notes document that's very high level because some-

   5. RNRyan Nystrom9:35

      Yeah

   6. CVClaire Vo9:35

      ... human is putting it together, it stinks. And-

   7. RNRyan Nystrom9:39

      [laughs]

   8. CVClaire Vo9:39

      ... you start to, like, do... I, the, the other thing that I found, and maybe I'm, I'm curious if this has impacted how you work or just made it easier, which is, like, you start to have those meetings less because the updates-

   9. RNRyan Nystrom9:50

      Mm

   10. CVClaire Vo9:50

       ... aren't super rich. People don't feel like it's a good use of their time, and I think you lose something by reducing the frequency. But if you can have high bandwidth, high quality meetings w- with high frequency without the overhead, I think you can get better, more detailed work done collaboratively, especially if you're running, like, a remote team where not everybody's in the same, same room. So I'm curious, do you feel like just being able to get to this level of detail in your standup has improved how you can actually build this product and do these, do this work basically?

   11. RNRyan Nystrom10:25

       I've been in way too many meetings where I can tell, like, everybody's eyes are glazed over. Nobody's paying attention. And I, like, I have run enough projects at this point where, like, to me that's, like, pretty dangerous, 'cause, like, everybody in that room has ideas, they have insight, but, like, because we've made it, like, so non-productive and, like, not engaging, that, like, we're not... We're just like, we need... The point of the meeting is to, like, exchange ideas and information, and if we're, like, not doing that, like, literally what's the point? The whole, like, this could've been an email. Like, that's what the meme is from. Um, and so I, I've found that this is basically a good conversation starter. Um, and

5. 11:10 – 12:15

   How automated context surfaces every engineer’s work equally

   1. RNRyan Nystrom11:10

      a, a key thing to me that, that tells me that this is working is when I'm, like, I'm running these meetings and kind of going through, like, bullet by bullet, and then I see, like, oh, somebody, like, fixed our mock server environment in our, our Jest tests, and we're seeing a test improvement by, like, up to 13%. I'm like, oh, I missed that, you know? Like, that's, that's super cool. Like, tell, like, let's talk about it. And then from that, maybe, like, oh, there's additional headroom that we can make on this. And so all of a sudden, like, let's drill into it. And so just, like, I also think it sort of democratizes that, like, sharing so that you've got some people that, um, like I, I could talk for an entire 30 meeting without shutting up, and then other engineers are, like, you know, kept, keeping to themselves, but, like, brilliant and super talented, and I think it's great. This kind of, like, raises everything up to the top, um, and makes it very visible.

   2. CVClaire Vo12:06

      I was, I was thinking last night, truly this is a little bit of a sidebar, but I was thinking last night how people don't know this. It's very, very confusing, as Claire Vo, the podcast

6. 12:15 – 14:26

   Why cutting meeting prep is a burnout protection mechanism

   1. CVClaire Vo12:15

      host. I am such an introvert. I'm a crazy-

   2. RNRyan Nystrom12:18

      [laughs]

   3. CVClaire Vo12:19

      ... introvert. I will avoid humans with every fiber of my being.And I find like AI as a proxy for me to get over my own anxiety communicating with the outside world. [laughs] It's like so insane, but I really feel it. Whenever I like message my OpenClaw, and I'm like, "Hey Polly, can you email so and so, blah, blah, blah on my behalf?" My anxiety is gone, but when I sit in front of my Gmail and start typing an email, my introvert start showing and I'm like, "I can't, I can't do it." So I think this is a real point, which is like you gotta pull information out of everybody.

   4. RNRyan Nystrom12:53

      Yeah.

   5. CVClaire Vo12:54

      And there's like spikes and valleys of who is gonna come and show off their work, how detailed they're gonna be. It does not mean the work is not good. It does not mean they are not talented. They just have different skills in different environments. And so I love this call out of like we're gonna pull equal amounts of information out of everybody because the playing field is, is equal. I also wanna go back to something you said, which makes me think about burnout, which you said, "I can like work up until this meeting starts."

   6. RNRyan Nystrom13:23

      Yeah.

   7. CVClaire Vo13:23

      And I, I know so many people who feel like they're in meeting after meeting after meeting, one, because when they're not in meetings, they're preparing for meetings. So you're getting rid of that.

   8. RNRyan Nystrom13:34

      Yeah.

   9. CVClaire Vo13:35

      And then two, I think managers being able to, for example, code up until the standup is such a burnout like protection mechanism, which is you would much rather your manager's hands on the code, hands on doing work, filling a creative impulse than prepping for meetings. Sure, they're great at running the meetings, but let's do that in like this just in time mechanism where they're supported-

   10. RNRyan Nystrom14:04

       Yeah

   11. CVClaire Vo14:04

       ... um, through these automations. And so I just imagine, I used to feel so stressed as a leader and as a manager being like, if I'm not in a meeting, I'm prepping for a meeting, and then I'm in a meeting, and if I can carve back more of my time to just do real things, that feels so much, so much, so much better. So I'm guessing you're having more fun just through the balance of the kind of work you're

7. 14:26 – 16:13

   The case for engineering managers writing code

   1. CVClaire Vo14:26

      trying to do.

   2. RNRyan Nystrom14:27

      I'm, yeah, I'm having so much more fun working this way. I, I have done the like run a big engineering group-

   3. CVClaire Vo14:34

      Yeah

   4. RNRyan Nystrom14:34

      ... where, yes, you're spending half of your time just compiling information, synthesizing it, writing reports, like doing all this stuff, and I, I hated it. Like, it's so draining. And now I feel like I'm in a sweet spot where I can support and work with a team of like very talented individuals, but also not have to, yeah, be like doing like paperwork the entire time.

   5. CVClaire Vo15:01

      Yeah, yeah.

   6. RNRyan Nystrom15:01

      Like I, I don't want to do like the tedium. I get a lot of joy out of working with people. It's like why I like managing. I like talking with them, having fun. I love like sar- solving hard problems together. And then I like building stuff. And, um, yeah, I think that we're maybe at an inflection point where like, I don't, I don't... Maybe this is controversial or not, but like if you're like a line manager, like write, write code, you know?

   7. CVClaire Vo15:31

      Yeah.

   8. RNRyan Nystrom15:31

      Get in there and like-

   9. CVClaire Vo15:32

      100%

   10. RNRyan Nystrom15:33

       ... stay, stay close. Maybe don't do the, the, the P0 hero projects, but like-

   11. CVClaire Vo15:37

       Yeah

   12. RNRyan Nystrom15:38

       ... help your team fix bugs, like make optimizations, like whatever, you know? I... It's just, it's so easy now.

   13. CVClaire Vo15:44

       I mean, I'm gonna pull that thread all the way up, which is like-

   14. RNRyan Nystrom15:48

       [laughs]

   15. CVClaire Vo15:48

       ... directors of engineering, VPs of engineering, like CTOs-

   16. RNRyan Nystrom15:52

       Yeah

   17. CVClaire Vo15:52

       ... CPOs, it... Write, write some code. Now is the time.

   18. RNRyan Nystrom15:56

       Yeah.

   19. CVClaire Vo15:56

       And I say this all the time on this podcast, this is the era of the hard skill. This is not how do I get better at my, my soft skills and managing stakeholders.

   20. RNRyan Nystrom16:05

       Mm-hmm.

   21. CVClaire Vo16:05

       This is literally like how do you write code? How do you write automations? How do you learn these new tools? How do you understand what models do, do what for your own skills?

8. 16:13 – 20:30

   Inside “Boxy”: Notion’s internal VM-based background agent system

   1. CVClaire Vo16:14

      That I think is super important. Okay, we could go on forever on this topic.

   2. RNRyan Nystrom16:17

      [laughs]

   3. CVClaire Vo16:17

      I do wanna show folks how you built this.

   4. RNRyan Nystrom16:20

      Yes.

   5. CVClaire Vo16:21

      Because we haven't seen, actually haven't seen... We've had a couple Notion folks on the podcast, but we haven't seen Notion AI in action, and I just wanna see kinda your thought process on how you build something like this out.

   6. RNRyan Nystrom16:32

      So I'm gonna flip over to, uh, this is our custom agent. Don't ask me why we got this like potato theme for like the entire project. I think it was kind of something about like CI is just this like cobbled together like mess, and so we're gonna like make the potato a, like a rocket ship. I don't know. I don't even know if that makes any sense, but like now we're having fun with it, and we have reactions and like agents and y- it, it has spun off into its own thing, which is fun. But this is our hot potato agent. So you can see I have this set up to run at 9:00 AM every single day. Um, it's also set up for chat. It's set up if the agent's mentioned, but we never really use any of that. And probably the important part is the actual instructions. So in this instruction page, giving context on like what the purpose of this agent is. I am telling it to run, yeah, look back at 24 hours, so basically telling it your job is to run every single day, and I only want you to look back for the last like 24 hours of activity. I'm explicitly telling it to use subagents, which is kind of a sleeper feature in Notion AI. Like, this exists, but we don't, we don't really push it to use it very often yet 'cause it's, one, it's very expensive, and two, it can be kind of finicky sometimes. But, um, I helped build it, so I'm... I know how this works. Uh, and then I ask it to kind of like fan out and do like a MapReduce where I'm saying, "Go use the Honeycomb MCP to figure out what the latest metric is. Look in our project channel and like find updates, feedback, questions." I tell it where the task, uh, database is and how to look for tasks within this project, and then how to find yesterday's meeting. And then I give it a template in the instructions where I'm like, "This is your format. I care about CI speed, decisions, progress, changes, bugs, questions, risks."... a little bit of guidance on writing. And then when it's done, uh, I have it post to Slack, and I, like, emphasize this, like, bri- I want it to be brief and fun. And sometimes it's really corny, and then sometimes it's, like, really good. Uh, and it'll be very quirky, and just post this, like, link in our Slack channel, and it's like, "Hey, here's your, your pre-read," some little quibble about, like, whatever, you know. Like, "Hey, you guys are not making enough progress." Uh, and then that's it. And then it's, we have our meeting note. It's updated. What I like the most about this one, I can show you some of our, like, internal settings. So this is, like, I give it access to all of these things, and I'm like, you can only view all of this stuff, 'cause I don't want it going and, like, modifying our task database, our project database. Like, everybody at Notion uses those. But this meetings database in particular, I'm like, you know, you can edit content, 'cause this is the one you're gonna, like, write and update the page. It can read from our Slack channels, respond to our project one, and then th- this was new to me, actually, when I set this agent up, is our MCP. So we've had MCP, and we have this other thing called Workers, which is, like, kind of like writing code. I haven't used v- them very much within Custom Agents, but in this one in particular, I'm like, I know exactly where this metric is. It's in Honeycomb. And so I, like, just configured the MCP in Notion. Uh, by the way, I like, I like used the agent to, like, set itself up. 'Cause I was like, here's the query. I literally gave it a screenshot of the Honeycomb query, and I was like, "I don't know how this works. Can you just, like, update your instructions?" And it like-

   7. CVClaire Vo20:08

      I love that you screenshot it. You didn't even copy and paste it. You're like-

   8. RNRyan Nystrom20:12

      No

   9. CVClaire Vo20:12

      ... "Please OCR this screenshot." [laughs]

   10. RNRyan Nystrom20:14

       Uh, exactly. Too lazy. I'm like, "Here it is. Just take it. Figure it out." And it, it kind of, it got it mostly, like, most of the way there. I had to fiddle it, with it a little bit, but yeah.

   11. CVClaire Vo20:26

       What, what I appreciate about this, and again, for anybody trying to just brainstorm

9. 20:30 – 24:51

   Old World vs. New World code review

   1. CVClaire Vo20:31

      workflows where AI can actually have a huge impact on your productivity at work or life, I just feel like write down what you would do if you had time. If you had time every morning at 9:00, you would sit down, and you with your eyeballs would go through Slack. You would go through Honeycomb. You would ask people what's going on. You would look at GitHub, and then you would compile it, and then you would try to be very fun in Slack. Like, it's, it's just a description of what you would do, and-

   2. RNRyan Nystrom20:59

      Yep

   3. CVClaire Vo21:00

      ... it's, it doesn't have to be that complicated, and you can iterate so quickly on it. What I appreciate about this, you know, versus old era of more deterministic, like, workflow style builders, is the updates are so easy to make. Just change the natural language, redo-

   4. RNRyan Nystrom21:17

      Yeah

   5. CVClaire Vo21:17

      ... the order, change the trigger, give it more, um, access to data, and then it's, it's ready, ready to go.

   6. RNRyan Nystrom21:24

      Yeah, and you know what I think the other thing I, I've gotten hung up on when trying to think about, like, these automations, and I, I've seen others do, is that, like, you get this, like, you start giga-braining it, and you're like, "Well, how am I gonna save, like, five hours of work a day?" And I, what you just said made me realize too that, like, the, the tedium that this removes for me is not, like, world-changing, but it's like 20 minutes a day. And that's, like, 20 minutes I can spend doing other stuff, and that, like, it's not even just about, like, saving that 20 minutes, but it's, like, protecting my brain from, like, having to context shift about all this stuff and, like, ingest it. And instead, yeah, it's just, I know that the information will be there when I'm ready to, like, read it, and I'm ready to, like, shift gears to this project, um, rather than, yeah. The, I, I hate doing the, like, read this update, copy all this information, like, put it into a, like an update board. Like, it's soul-sucking.

   7. CVClaire Vo22:26

      I mean, you and I have been doing this for a while.

   8. RNRyan Nystrom22:28

      [laughs]

   9. CVClaire Vo22:29

      That's like, that was, I feel like, 70% of my job. At some point, 70% of my job was just like, "What's going on?"

   10. RNRyan Nystrom22:36

       Mm-hmm. Mm-hmm.

   11. CVClaire Vo22:37

       "And how do I massage it-

   12. RNRyan Nystrom22:39

       [laughs]

   13. CVClaire Vo22:39

       ... into a format appropriate for the audience at hand?" And it's always the same information. [laughs]

   14. RNRyan Nystrom22:45

       Yep.

   15. CVClaire Vo22:45

       It's just like, what's the executive version of it, and what's the team version of it, and what's the full team version of it? And like, oh, like, my shoulders drop out of my ears when I realize we don't have, like, we just don't have to do it anymore. And the other thing that I think people maybe under-appreciate about AI and this, like, just-in-time delivery is your AI, your agent is never gonna complain when you ask it to do this five minutes before the meeting starts.

   16. RNRyan Nystrom23:11

       I know. [laughs]

   17. CVClaire Vo23:11

       Right?

   18. RNRyan Nystrom23:11

       It's so, it's so great. [laughs]

   19. CVClaire Vo23:13

       It's so great. It's just like, when you have it, drop it, get it done and out of your brain, I think is ju- A- again, I go back to, like, burnout and enjoying your work and reducing toil, and it just feels like a more relaxed way to work.

   20. RNRyan Nystrom23:30

       Yeah.

   21. CVClaire Vo23:31

       [laughs] And we're, we're happy.

   22. RNRyan Nystrom23:33

       Yeah, which it's so funny, 'cause it is more relaxing, and it's more fun, and I feel like I'm getting more done.

   23. CVClaire Vo23:38

       Yeah.

   24. RNRyan Nystrom23:38

       It, it's weird to have, like, this, like, win, win, win.

   25. CVClaire Vo23:41

       Yeah. You know, they, they do the triangle, and they're like, "Pick two," and you're like, "No, I wanna pick all three."

   26. RNRyan Nystrom23:45

       Yeah, no, I'm like, "Give me the whole triangle." [laughs]

   27. CVClaire Vo23:47

       Give me the whole triangle. That, stamp it for the YouTube thumbnail.

   28. RNRyan Nystrom23:51

       [laughs]

   29. CVClaire Vo23:51

       Give me the whole triangle. All right. This episode is brought to you by Orkes, the company behind open source Conductor, which powers complex workflows and process orchestration for modern enterprise apps in agentic workflows. Legacy business process automation tools are breaking down. Siloed low-code platforms, outdated process management systems, and disconnected API management tools weren't built for today's AI-powered world. Orkes changes that. With Orkes Conductor, you get a modern orchestration layer that scales with high reliability and brings humans, AI, and systems together in real time. It's not just about-It's about orchestrating everything, APIs, microservices, data pipelines, human-in-the-loop actions, and even autonomous agents. So build, test, and debug complex workflows with ease, all while maintaining enterprise-grade security, compliance, and observability. Orkes, orchestrate the future of work. Learn more and start building at orkes.io.

10. 24:51 – 29:20

    Prompting Codex from Notion comments

    1. CVClaire Vo24:51

       Let's talk about, so we're talking about how meetings happen. Love this. You do write code, though.

    2. RNRyan Nystrom24:57

       I do write code.

    3. CVClaire Vo24:58

       Sometimes, sometimes with your fingers and sometimes with your my favorite harness of the moment. Um, so let's go to how you get coding.

    4. RNRyan Nystrom25:08

       Yeah. I, I wanna show the little bit of a new workflow that we have going on in Notion. Um, I honestly don't think this is necessarily a big feature that we're gonna ship, or some- we might ship some version of this feature, so this is all basically internal only at this point. But prior to this, the way, one, I love Codex. Uh, I've been a, a Codex stan for, like, I don't know, six, seven months now. And we started, uh, we started building this, like, Codex integration into Notion. Prior to this, it was like, w- I mean, obviously you're using the CLI to, like, write your prompt, and then they created the Codex app, which is, which is nice, but I'm still, like, writing my prompt in this thing. So I started actually writing prompts in Notion pages where I can be a little bit more, like, freeform and structured and, you know, I'm in, like, the CLI, don't have to worry about, like, hitting enter and then like, "Oh, shit," like, "I sent my prompt." Uh, I can actually, like, write a document in, uh, Notion, but then of course I'm like, cop- or like highlighting all of the text, copying it, going into my terminal, hitting paste, and, like, letting it go. It's, like, fine. Um, there's like MCPs and other stuff that I could be using, but I'm, I'm a very simple person. [laughs] There's like too much, too much stuff. Um, so we built this thing that, uh, we're kind of like calling it... I think we're calling it both Software Factory, but I like its internal project name is, uh, Boxy, um, 'cause it's like all these little VMs that we install Codex and Claude Code on. It's our little boxes. Where now we can actually invoke them from, like, tasks within Notion. And so, uh, literally happened this morning, a friend of mine, um, who's a Notion fan, texts me and he's like, "Hey, I like the tab block that you built," um, which is this thing. He's like, "But I really wish I could, like, you know, I can, like, copy..." I'll show you now. If I click on the dot, dot, dot thing next to a block, there's, like, copy link to block. He's like, "Man, I really wish I could just, like, copy link to a tab and then, like, send it to somebody." And I was like, "Oh yeah, that, that sounds really easy." So I opened up this task and I just took some notes and I dropped in this screenshot showing him, uh, where it could live. You know, this is, like, on the, the tab block. If I right click on it and we get this little flyover menu. And I just described the task. I was like, "Yeah, let's put a copy link button here." I also noticed that hovering over the delete button didn't change to red, so I was like, "Yeah, we should fix that." And I was like, "I know one edge case is, like, you're gonna have to, like, if you land on a URL where we're, like, linking to this block a- and this tab, like it's gotta, like, switch to the tab on, like, fresh refreshes." So I, like, kinda outlined all the, the cases. But you can see here, this is one, two, three paragraphs, four, like four sentences and a screenshot. It was like, not a, not a lot. And then, uh, this new thing that we built, I can actually mention Codex from within our comments, and this triggers our, like, Boxy, uh, this is all our, like, internal dev tooling stuff. And then it got to work. And I was looking at the timestamps earlier, and I think 10:40, 10:51, started the implementation, and then another 10 minutes later it replies with a pull request link and a preview URL, 'cause we, like, we do the, um, like preview environment stuff. And, uh, it ha- like, built the entire thing, and if I switch actually over to... Here's the pull request. It built, like, the entire thing, and it was like, "Here's how I tested it." And this, this was actually the coolest part to me is, like, it actually uploaded screenshots of it doing its own, like, UI verification. And there was, like, a CI failure in it, so I was like, "Hey," I replied down here, like, "Oh, yeah, this part of this code change," I was like, "I don't know what is going on here. This doesn't make sense." And there's some type check things. And it was like, "Cool. Here's why we did this change,

11. 29:20 – 31:01

    The emotions around code review

    1. RNRyan Nystrom29:20

       and I fixed your types."

    2. CVClaire Vo29:22

       Can we-

    3. RNRyan Nystrom29:22

       And then there was a merge conflict

    4. CVClaire Vo29:24

       ... let, let's also talk about just old world, new world.

    5. RNRyan Nystrom29:27

       [laughs]

    6. CVClaire Vo29:29

       The emotions around code review. [laughs]

    7. RNRyan Nystrom29:32

       Uh-huh.

    8. CVClaire Vo29:33

       Like, I love that you're like, "I don't get this." Not, "I'm gonna try to get it," or, "I've done my best to investigate and I'm pretty sure this is wrong." Like, literally just, "I don't get it. I don't get it."

    9. RNRyan Nystrom29:43

       Yeah.

    10. CVClaire Vo29:44

        Like, imagine, like, sending, [laughs] sending that to somebody. [laughs]

    11. RNRyan Nystrom29:48

        Oh, yeah.

    12. CVClaire Vo29:48

        One of your human teammates. Be like, "No, man, look it up. It makes it..." So, uh, that's one of my, one of my code review mechanisms as well is I'm just like, "I don't think this is right."

    13. RNRyan Nystrom29:59

        I, yeah, one, one, one, uh, line that I've been putting in my prompts lately is, I'm like, "I literally don't know what I'm doing here. You need to explain this to me." Especially doing all this CI stuff, I'm like, "I'm in over my head." Uh, like, "You gotta, you gotta, like, explain it like I'm a five-year-old." And sometimes it's, like, [laughs] literally, like, caveman style, like, "Here are the things." And I'm like, "I needed this."

    14. CVClaire Vo30:23

        Well, what I appreciate about this, and we'll talk about it in, in a little bit later, I think, is we're not getting any of these Claude Code warm fuzzies from Codex. It, I love it, but-

    15. RNRyan Nystrom30:33

        No, it is-

    16. CVClaire Vo30:33

        ... when it talks to me I'm like, "I feel real dumb when you're talking to me, Codex. Can you-"

    17. RNRyan Nystrom30:38

        It's like, "Okay, okay dum-dum, here's what's going on."

    18. CVClaire Vo30:40

        Maybe we're on that piece. [laughs] So I think that sort of like, "I don't understand what I'm doing, please explain it to me," is, I love it, but I think it is a, a Codex specific experience [laughs] and that you get... Claude Code shows up and is like, "Hey buddy, guess what I made for you?" Okay. So stepping

12. 31:01 – 32:00

    Quick recap

    1. CVClaire Vo31:01

       back before we go too deep on the personalities of all these coding models, you have built just @mention, and I think people miss this, but it, it's important for particularly our engineering team and engineering leader listeners to pay attention to, which is everybody they know that's smart, it has some background agent hooked up to a virtual machine that they kick off so that you're not spinning up your local environment and doing all this stuff on your machine. Um-

    2. RNRyan Nystrom31:28

       Yeah

    3. CVClaire Vo31:28

       ... and it just, I think the velo- We're, we'll, we'll talk about this, but the velocity plus your DevX plus your CI, I think is a huge piece of AI adoption in engineering. So if you are not, if you don't have a like VM strategy and background agent strategy in your large engineering org, time, time to get one. Time to do it.

    4. RNRyan Nystrom31:48

       Totally agree. Totally agree.

    5. CVClaire Vo31:49

       And we'll just ask Codex to build it. Be like, [laughs] "I don't know how to-"

    6. RNRyan Nystrom31:51

       Yeah.

    7. CVClaire Vo31:51

       "I don't know what I'm talking about, but I think, I think this is... Claire said this is a good idea. Please build. Make no, make no mistake." Um, okay.

13. 32:00 – 35:10

    Spec-first development: writing and checking agent specs into the repo

    1. CVClaire Vo32:00

       Last, last one. We have a last use case you were gonna, you were gonna walk us through.

    2. RNRyan Nystrom32:06

       Yeah. So we recently rebuilt our entire agent harness. The quick TLDR is just like everybody else, we reached like this point of tool and instruction fatigue where you have this bloated system prompt. So we're like, "Okay, we need to dumb this down," and we borrowed the, the kind of like skills and progressive disclosure concept from coding agents and brought that to Notion AI. And when we were doing this big rewrite, we were asking ourselves like, you know, it, it had only been like six months since we shipped the last rewrite, and this time we were like, "What would, what would we do differently?" Like seeing the state of the art with coding agents. And someone had this really great idea to like, let's not start with code. Like let's just start with specs. And what we've ended up building, um, is we have this like in our, checked into our code base, you see this, we have this... We're looking at, this is a Notion repo, but we have this agent specs subfolder, and within this subfolder we have all of these markdown documents. This is one that, that I worked on. Uh, we have this thing in our, uh, AI called ask mode, where we basically, um, ban all the like mutating tools, so it can only just like read and answer questions. And so when I was building this, I didn't start with writing code. I didn't start with anything. I just started with an empty markdown document. And, uh, I mean, I actually just opened up like Whisper and just started yapping about how this feature should work. And then at the end of it, I gave that, [laughs] I gave the Yap session to Codex, and was like, "Here's our other like spec library. Learn the format, take my information, write a spec." And then it spiked the first version. I did a couple, uh, revisions on it, and I ended up with this markdown document. Now, the markdown document is like, it's nice, but what we did with it next, in my opinion, is, is I, I kind of think that this is like the future of software engineering, where I then opened up Codex again, pointed it at this spec file, and I said, "Build it." And it basically one-shotted this, because the entire spec file is so comprehensive with code pointers, with um, down at the bottom we have verification, and it was like, "Here is how you're gonna verify all of this stuff works." And we've even built our own, um, CLI tools so that you can run Notion AI from the CLI, and it could, once it's done seeing that all the tests pass, it can actually just spin up Notion AI itself, send it queries, send it questions, enable ask mode, disable ask mode, and then see the transcripts and like see what actually happens. And I think the first shot of this took a couple hours, but I came back to whatever, couple thousand lines, did some code review, played with it myself, and I was like, "It's right. It's like done." And, you know, since then we've made... I can... The other beauty of this is like this is in version control,

14. 35:10 – 37:53

    The spec as changelog: version control for how a feature actually works

    1. RNRyan Nystrom35:11

       so I can go to the past changes of this spec file and I can see how the, the spec has evolved. And I, it, I could go look through all of the code changes, which also have their own like history, but this is now the sort of like source of truth for how this part of Notion AI works. And it's just in plain English that can then be verified and implemented by agents.

    2. CVClaire Vo35:34

       Well, I think the other thing that people don't appreciate is, you know, taking this outside of the engineering flow is this plain English can be ingested by other parts of the business that need this information. So let's say you need to then release this feature via some sort of marketing. This is actually like a pretty good asset that explains how it works, that can be translated into another, another thing, in a way that like code itself-

    3. RNRyan Nystrom36:01

       Yeah

    4. CVClaire Vo36:01

       ... is still a little intractable. And so this idea of spec-driven development, but what I like about what you said I don't want people to miss, is the way you make these updates is you update the spec and go, go, go look, make the, make the update, change the date.

    5. RNRyan Nystrom36:15

       Exactly.

    6. CVClaire Vo36:16

       Um, and so like the spec is the source of truth. The spec as the, the, s- the change log, I think is a really interesting model. And for people that aren't watching, it's very detailed. It's very technical. So it's not like there is not code in the spec.

    7. RNRyan Nystrom36:33

       That's right. That's right.

    8. CVClaire Vo36:33

       There's just not all the code [laughs] in, in the spec.

    9. RNRyan Nystrom36:37

       Exactly.

    10. CVClaire Vo36:37

        And I think that's a, a really kind of good hybrid model for experienced engineers to start to bridge into what would it look like to have an agent do more of your coding work while you still do architecture work, while you still do design work, while you still make sure that the thing is gonna scale.

    11. RNRyan Nystrom36:56

        Yeah, [clears throat] exactly. I, I view our job as, like, engineers evolving into, like, systems thinkers and architects. And not, and not even just necessarily writing, like, the spec and thinking about the behaviors, but most importantly is, like, the verification loop. Like, is it a... Like, how should it verify correctness of this feature, um, or this change? And honestly, it's like if it can't or if, like, the verification's a little hazy, it's like that's the first thing you actually should be going and doing is, like, do you have a tool to let the agent, uh, run itself? That's like one of the first things we did with this project was like we, we should actually build a CLI so that I can tell Codex, like, send this prompt and, like, see what happens. And now that we have that, then we can take these specs and actually just, like, go deeper and deeper and deeper. And so it's like we're still doing engineering, it, but I'm not doing the, like, plumbing work of, like, wiring up this ask mode feature.

15. 37:53 – 39:00

    How engineers’ roles are evolving

    1. CVClaire Vo37:53

       Well, and what I think is really funny is, I, me, I've been in software engineering for 20-plus years. Like, we were writing these documents anyways. We were writing technical design-

    2. RNRyan Nystrom38:04

       Totally

    3. CVClaire Vo38:04

       ... documents and spec des- di- documents anyway, and we were sitting in meetings with other engineers debating the merits of one implementation versus another, and then we still had to go write the code. [laughs] And so-

    4. RNRyan Nystrom38:15

       Yep

    5. CVClaire Vo38:16

       ... it is, like, really... It, it hasn't added work to, to go into this model. It's maybe, like, shifted the emphasis of where the human attention goes, but these were documents that I, at least I was, in every org that I've ever been in, writing before. And the other thing that I think has changed so much is those me- those docs then waited for review, and they waited for a meeting. And now, no more waiting for the meeting, no more waiting for review. Ship it, have a verification loop, debate it on the merits of it being live and working versus the theoretical merits [laughs] of it sitting in a document waiting for everybody's calendar to open up for an, you know, a live argument.

    6. RNRyan Nystrom38:59

       Yep. Yep.

16. 39:00 – 45:21

    Lightning round

    1. RNRyan Nystrom39:00

       Couldn't agree more.

    2. CVClaire Vo39:01

       Let's, let's do it. Uh-

    3. RNRyan Nystrom39:02

       [laughs]

    4. CVClaire Vo39:02

       ... this is the my... The... I, I just you and I could talk all day about this.

    5. RNRyan Nystrom39:06

       [laughs]

    6. CVClaire Vo39:06

       Just to recap for everybody, 'cause I know I gotta get you out of here, three use cases. One, never prep for a meeting again. Hook it up not only to your Slack, but to your meeting notes, your GitHub, your telemetry, and build the best standup meeting so no one has to stand there glassy-eyed, being boring, giving updates. Second use case, background agents, @mention from wherever you work, Notion being a great place to do that, kicking off virtual machines, getting PRs done, just saying yes when your friend texts you, "Can you ship this feature?" And then the last one, putting all your specs in your repo, using them as a source of truth for a more autonomous coding agent like Codex. Let it cook for a couple hours, review the code, and then when you update, update the specs. Don't update the code. Did I get it right?

    7. RNRyan Nystrom39:53

       Nailed it.

    8. CVClaire Vo39:54

       Ugh. Okay, let's do a couple lightning round questions and I'll get out, get you out of here. You and I love Codex. Why, why you love Codex? I'll tell you why I love Codex, but you go first.

    9. RNRyan Nystrom40:02

       Um, okay. I... Well, so I first, I first fell in love... Oh my God, did I just say that?

    10. CVClaire Vo40:09

        You did.

    11. RNRyan Nystrom40:09

        I first, [laughs] I first fell in love with Codex, uh, because, uh, when I was, like, evaluating both, like, Claude Code and Codex, um, I found anytime Claude Code, like, filled up its context wind- window, it would just kinda, like, lose the plot really quickly. And Codex, I don't, I don't know exactly what it's doing, if it's the model, if it's the compaction, if it's, like, both, but it can, it can grind for, like, hours. Um, and, uh, with, with the way that I work, both the systems and things that I work on, and just, like, I, I like to be able to fire off a bunch of them, like, at the same time, and then, like, go to a meeting or go do something else or, like, uh, spend my time kind of, like, round robin managing, like, all of these agents. Like, I, I, I don't necessarily wanna... I'm not the person that is, like, sitting with the browser open and, like, um, some agent next to it and, like, iterate, look at the browser, iterate, look at the browser. I'm like, the more, the closer I can get to one-shotting solutions, the better, uh, 'cause that frees me up to do other stuff. Um, and so I found that, like, Codex was, was pretty good about that. I also just feel it's, like, pretty simple. Like, it's... There's not a lot of bells and whistles. It's not, like, too fancy necessarily. Um, I'm happy with the addition of, like, MCP and skills, um, and some, like, other stuff coming out. Uh, I also really love GPT-54. Um, I think it's a great model. So I'm like, all of those things together, it's just... It really, like, matches my working style, uh, and type of work a lot.

    12. CVClaire Vo41:47

        Yeah. I'll, I'll tell you why I love Codex, and I sent this to somebody. I said, "Work trees everywhere."

    13. RNRyan Nystrom41:53

        Yeah. [laughs]

    14. CVClaire Vo41:53

        "Ports 3000 through 3009 spoken for." Like, we're just, we're just going [laughs] across, across the board. Um, I do think it's good at long-running tasks. I like its concept of projects, 'cause I run a lot of different projects.

    15. RNRyan Nystrom42:06

        Me too.

    16. CVClaire Vo42:06

        I think that's just, like, a very helpful mental model. And then it's great at code review, honestly. Like, it-

    17. RNRyan Nystrom42:13

        Oh, yeah. It's pretty good

    18. CVClaire Vo42:14

        ... it's just, like, a really good code reviewer. It's a really good security reviewer. Um, tireless, uncomplaining, um, with the most tedious of things. And so I, I find it's one, my, one, one of my daily drivers. I just... I really-

    19. RNRyan Nystrom42:28

        Yeah

    20. CVClaire Vo42:28

        ... really like it. It's good. Okay, second question, because you and I also agree on this. Give people the reason... This isn't even a question, this is a, a demand for you to share my point of view, which is why bang on developer experience and CI speed right now?

    21. RNRyan Nystrom42:45

        A couple, a couple things. To, to me, CI was, like, super important prior to agents, um, because in my opinion... I mean, it's not even an opinion, it's, like, fact that, like, the faster your CI completes, the faster you get signal.Um, the faster your, like the more comfortable your engineers will feel with like making changes and doing things because they know, like I can make a change and get it pushed into dev or into production really quick because I know my CI's fast. If it's slower, then you're building up these like monster changes, and you're gonna be even slower about like judiciously like reviewing every little, like tiny thing. When I, I'm like a learn through like doing sort of person, and if I can close the iteration loop, like even tighter, then I'm gonna be putting out a change, people are gonna be using it, I will take that feedback, I will make another change, and I don't have to worry about, um, you know, it taking another day before like the deploy train is ready to go. I just wanna like, I wanna crank really, really fast. And that was all pre-agent. And now that we're like in the agent land, it's, it's that, but like on steroids. Because agents don't get tired, they can work on a VM, they can work like while I'm sleeping, and, you know, if I'm take, if I've got a CI loop that takes an hour to run, the, your agent's just gonna sit there and spin for like an hour waiting for results to like do something. If it takes three minutes to run, like holy crap, like how much more stuff are you, you as a human, and then especially as your like little swarm of agents going to be able to get done? Like so, so much more. Um, and so I... Super important.

    22. CVClaire Vo44:27

        I, I agree, and we just had Steve from Stripe on, and they're doing like 1,300 agent PRs a week.

    23. RNRyan Nystrom44:33

        Oh my God.

    24. CVClaire Vo44:33

        Like you cannot do... [laughs] Yeah. You like cannot do that if-

    25. RNRyan Nystrom44:38

        Yeah

    26. CVClaire Vo44:38

        ... your CI is slow. It's just-

    27. RNRyan Nystrom44:39

        Yeah

    28. CVClaire Vo44:39

        ... you might as well be throwing all those PRs in the trash. And so I do think there is just a true mathematical limit-

    29. RNRyan Nystrom44:47

        Yep

    30. CVClaire Vo44:47

        ... on your capacity to ship code to production that is a reflection of how fast your CI pipeline is. And so if you are, again, engineering leaders, if you are not spending time on making that fast, and you are tolerating something slow there, you're not gonna get the, the benefits of AI that you could. And honestly, like good for AI, good for humans, no engineer wants to sit waiting for their stuff to hit, hit production. It's miserable on the engineer side. Lots of downsides. So allocate time to your pipeline, please.
17. 45:21 – 47:51

    Where to find Ryan

    1. RNRyan Nystrom45:21

       [laughs]

    2. CVClaire Vo45:21

       All right, last question. When AI is not listening, it's writing bad specs, it's not being funny in Slack, what is your prompting strategy? Do you yell? [laughs]

    3. RNRyan Nystrom45:35

       Yeah. I can, I can-

    4. CVClaire Vo45:36

       Yeah, I can tell

    5. RNRyan Nystrom45:37

       ... I can be a little bit of a diva sometimes if it like really goes off the rails. Though the, the other, the other prompting strategy that has like saved my ass working on the CI stuff lately is because like even the best models I feel like can sometimes be a little sycophantic. I'll be like, [laughs] I will just like be like, "You're wrong. Like you need to defend your argument." Because I like, I want it to defend it in the way that I, like I want, but I'm like, I just need, I just need to see the evidence that if I push counter to what it has done, that it can like back up with like good, pointed reasons, rather than just be like, "Are," you know, "Are you sure this like change looks okay?" It's like, "Oh boy, it's like totally fine." I'm like, no, no, no. I need like the cited, hard argument against it. 'Cause like, and, and a lot of times, like with the CI stuff, I'm like, I don't know what I'm doing. I know generally what I'm doing, but like the specifics are a lot more nuanced, and I like, I need to get this right. Um, and that's been super helpful. And then when it goes off the rails, I'm like, [laughs] I can be such a diva to it.

    6. CVClaire Vo46:45

       [laughs] Escape, all caps, no.

    7. RNRyan Nystrom46:48

       [laughs]

    8. CVClaire Vo46:49

       That's what I do. Interrupt and no. Steer the conversation. Little arrow button has never gotten so much work. Ryan, this was so much fun. Where can we find you, and how can we be helpful?

    9. RNRyan Nystrom47:02

       Uh, you can find me on X. Um, I'm out there shilling for Notion right now a lot. Um, but that's, that's basically where I spend my time. Yeah, I, my DMs are open, so if you have Notion problems, I like to try and fix them. I, you, I'll send it to Boxy, and we'll get it done.

    10. CVClaire Vo47:20

        Perfect. Love it. Well, thanks for joining the podcast.

    11. RNRyan Nystrom47:23

        Thanks for having me, Claire.

    12. CVClaire Vo47:23

        [upbeat music] Thanks so much for watching. If you enjoyed the show, please like and subscribe here on YouTube, or even better, leave us a comment with your thoughts. You can also find this podcast on Apple Podcasts, Spotify, or your favorite podcast app. Please consider leaving us a rating and review, which will help others find the show. You can see all our episodes and learn more about the show at howiai pod.com. See you next time.

Episode duration: 47:53