Running an AI-native engineering org

1. SPSpeaker0:00

   [ on-hold music]

2. SPSpeaker0:17

   Hey, everyone. [laughs] [clapping] Aw, thank you. I didn't expect applause. Well, thank you so much for joining me today. I'm excited to speak with all of you about, you know, some lessons I've learned as I've been leading, uh, Claude Code and Cowork. So actually, first, I should do an intro. So yeah, my name is Fiona Fung, and I lead Engineering and Product for Claude Code and Cowork. And before Anthropic, I had also built and led teams at Meta and Microsoft. So with that, let's dive into some lessons I'm excited to share with all of you. Hopefully, maybe you'll find a couple of tidbits that may be helpful to you. So there are five topics I really wanna cover today. The first is the bottlenecks have moved. And so when the bottlenecks have moved and it shifted, what are all the team norms that we felt, "Oh yeah, we have to rewrite some of these because it's changed." How we rolled it out, I'll talk about, like, how we rolled out some of these updated team norms. And in terms of the proof, what were some signals that I feel, "Okay, this is giving me confidence that we're trending in the directi- right direction," and maybe it'll help you as well. And lastly, I'll leave you all with maybe a topic for you to all take back to your team to discuss 'cause I'd love to hear about, um, yeah, whether any of this resonated. So with that, let's touch on this first topic, which is the shift. So the bottlenecks have moved. And you'll notice there's that little subtitle there of, "What served you prior may no longer..." This is one of my favorite sayings. Like, actually, I think this is the muscle that has really helped me, whether at Anthropic, but also before at the other companies, always being of that growth mindset. Like, you know, from all the talks this morning, you see our rate of change is just increasing. And so I've found it's always helpful to look at what are either team norms that you've set up for your team or team processes, and always ask yourself, "Is it still serving its purpose?" So for years, many years, engineering bandwidth was an expensive thing. And when you think about how did we all do software engineering, everything was around kind of like protecting this resource, right? Like, we want to do a lot of planning 'cause we wanna make sure when we're building that we have high confidence that we're building the right thing. Or we want to do reviews. Like, there was a lot of action around when engineering bandwidth was a limiting factor. And maybe I'll pause there. Like, when you think about our industry, this is not the first time that the bottleneck has shifted. Come-- I'll, I'll take you all back in a time machine with me. We'll go all the way back to the early 2000s. And if you all remember, so back then, I had just joined, uh, Microsoft. I was working on Visual Studio. There was no such thing as cloud. It sounds crazy now, right? Like, now cloud is so ubiquitous. But back then, there was no cloud, and I remembered how we built Visual Studio was this one server room, and everybody had to be on call for a week. And then you'd just go into this room, and I remember the whole build process. It was a queue. We can only merge six PRs at a time, and any time, you know, one of the tests would fail, you have to debug which of those PRs caused that failure. So back then, that was actually a big bottleneck. And then now with cloud and with continuous build, that bottleneck has also shifted. So this is just another shift for us to think about of when coding is no longer the bottleneck, what changes? And so on the Claude Code team, coding is rarely the slow part anymore. And so, like, that's why all the upstream and all the downstream kind of like processes, we felt like we've had to change a little bit. So as an example, what were some of the old bottlenecks, right? Like writing code, writing tests, refactoring. In fact, I'll share a story with you all. When I first joined Claude Code, I wanted to onboard and fix some bugs. And I thought, "You know what I'm gonna try? Test-driven development." I remember that was a really big thing a while back. You know, the idea is write the test first, make sure it fails, then do the change, then now you already have a test. But it was kind of like eating broccoli, you know. Like, even though I actually really do love broccoli, [chuckles] but I use it as an example. Uh, but I remembered, you-- it feels like eating broccoli to write the test first 'cause it's not always the most enjoyable part of the process. And I remember always itching to build a product and get to, you know, playing around with it. And so now with Claude, I remember the first bug. I'm like, "You know what? I wanna try test-driven development again." Let's write the test, but first it will fail. Let's verify it fails, 'cause that's how I could verify the test is actually testing something. And then we, we-- you know, like, I remembered, like, making the bug fix, and then we already had a test. And I remembered that first experience, test-driven development was actually so much more fun and so much more pleasurable. It kind of took the tax out of test-driven development. So that was a big aha for me. Refactoring is another fun topic. Like, I don't know if you all have felt this before, but, you know, I've, I've been on, uh, teams before where we have to either do, like, some big refactoring or some software architecture cleanup. And it's always a debate of when would we find time to do this work, this work that we know is so important, but it always has to come off at the cost of trading, you know, time to build product. So again, thanks to the models, and with Claude, refactoring is also no longer a bottleneck. So now when those shifts happen, what am I starting to see that as the new bottlenecks? Uh, verification, that's a big one. A lot of it's because the bandwidth has just increased so much. We have to pay even more attention to, is it correct? And I'll talk about this in a little bit. But also, when roles are blurring and more people are, you know, checking in changes, we also wanna make sure everybody feels confident that their changes are correct. Who reviews? That's a really popular topic. And also, how is it maintained? And this one is interesting because, again, the throughput is now so much more. Also have to think about, you know, the cost of maintenance as well.So these were some of the old processes that we felt quietly stopped working. And I love that phrase, quietly stops working, 'cause actually if I step back, any time, you know, hopefully when somebody puts a process in place, it's to solve a problem. But very often we forget to audit to go, "Wait, are those processes still required, or is this still serving its purpose?" So for example, what were some of the things that we had to change? Planning norms. When coding is no longer, you know, the, the, the bottleneck and, and not as... Like, and you have, like, much more coding bandwidth, how did we rethink through planning? Uh, code ownership is of-- also another interesting question. Almost all the Claude Code commits now are also, you know, co-authored by Claude. I don't think I've, in the last few months, ever saw a commit that wasn't. A code review is a good one. Like, that's a lot of questions I get a lot as well. Also, team makeup. Roles are blurring, and what are the skillsets that we're thinking, "Hey, you know what? These are actually skillsets we're kind of, like, doubling down into." And also knowledge sharing. Uh, what happens when maybe documentation is no longer your source of truth? So given some of those shifts, these are some of the team norms that we had to rewrite. We talked about code review. Uh, and actually Cat this morning in the talk really talked about Claude Code Review as well. That was a really, uh, like that's how... A, an amazing tool for us to make sure we're also doing the right thing. Onboarding has also changed. I'll share with you all my onboarding story. Planning, I've talked about this already, of when coding is no longer a bottleneck, how does that shift our planning? Hiring, and also org shape. Org shape is a fun one. Like, you know, we really want to keep everybody more agile and flatter orgs, but every manager on Claude Code actually started as an IC first. So with that, how have our planning and technical debates changed? So in technical debate, code wins. You know, it's like building is cheap, arguing is expensive. I'll actually share with you all, like I remembered, you know, as I was onboarding, I also wanted to do some code refactoring. I was kind of like, "Hey, what's some refactoring that we think is good to do but we haven't yet done?" And I remembered me and Boris had a couple of debates of approach. And in the old days, I remember-- And I almost did this. I almost said, "Hey, Boris, let's just go to this meeting room and go to whiteboard and whiteboard it out." Almost did it, and I'm like, " [gasps] Wait. If, if, you know, with thanks to Claude, I can actually generate..." I remember generating three different versions of the PRs, and actually that's what me and Boris used to have a g- really good technical debate, 'cause I wanted to not look at how the code was implemented, but also impact to the colleagues. Uh, prototyping is another good one. So now, like any time we're really having an idea, go ahead and prototype, and let's actually go ahead and dogfood it, and then we'll try. I remember many years ago also having this, uh, prototyping conversation on another team I was at. Um, you know, 'cause there was a debate. There were kind of two camps. One camp of prototyping is great. It allows you to move quickly to build, you know, kind of like a wire skeleton and get a feel for the product. But on the other camp, I think there was a concern. Well, you know, co- prototyping, you're kind of doing it with speed. You're cutting corners, and there's concern of what happens when you show a prototype that kind of works, and then, you know, maybe you get hung up on it. You don't wanna throw the code away, and then you end up trying to ship that thing. It's not built to scale. Again, with Claude, now actually prototyping is a great way to get started 'cause we could iterate and learn. And then thanks to, you know, the model's help, we can also then actually scale out a prototype to production a lot faster. So what's one thing we reduced on the Claude Code team? It's, uh, you know, I might have been on teams before and maybe y'all as well, like design docs, like really in-depth planning and design docs. Most of our discussions actually happen in PRs or prototypes. And again, that's just because the engineering bandwidth, you know, is, is now increased and coding is no longer a bottleneck. Uh, what do we really wanna double down on? Verification. And this is something that I wanna make sure, you know, me and our team keeps, keeps doing better as well. Because now with all the bandwidth, it's even more important to how do you verify quality? And I call this, like, you know, like shift left. So for example, what's better than you running-- It really sucks for me when I hear from our customers and developers and you all run into a code. I'm always like, "Ah, I wish we caught it first." But you know what's actually better than me running into the bug first? Us having automation in place to actually catch it closer to the source. So that's, like, that's something that we're really constantly looking at as well, of how to make sure we keep shifting left and automating more. Who made this change? This used to be a question that a lot of folks would ask, of, "Hey, who, who, who was the last person that touched this?" Or, "Who's the code owner?" And now I encourage you all, like if you find yourself asking that question, to get to the root of the question. Like, what are you trying to answer? Is it are you looking for someone that caused a regression? Are you looking for someone to answer a question? Are you looking to gain context? And is there a way that Claude could actually help you with that? Uh, Cat actually talked about routines as well, like earlier today. That's one of the routines I have set up. Like automatically, you know, in the morning I like-- Over my morning cup of coffee, I like to review all the feedback in our various feedback channels. So I have a routine that will go through to kind of amalgamate all the feedback for me and help me to identify themes. How do you keep up with code reviews? Before we actually shipped Claude Code Code Review, this was a question that I would get all the time, and now I can share that answer with you all. Like, definitely Claude Code Code Review is one, one tool that has really helped us to make sure we keep up with the coding bandwidth. And so where Claude is really good, right? Like the style and lint, obvious bugs. Um, any time that you actually have, like, like if, if there is a spec, I encourage you to actually check the spec into your code base, 'cause Claude is very good about verifying against spec drift. But it's still really important to still have a human in the loop. So where do we find even in code reviews is really important for humans still? Legal reviews. Anything that is actually about risk tolerance. So especially when you look at trust boundaries, it's all about that trust but verify and where humans bring a lot of the needed expertise.Product sense and taste is another fun one. So last, uh, December, I wanted to update Claude in our CLI to, you know, give Claude a little holiday theme, and I was ambitious. I wanted to turn Claude into a snowman. I, you know, like I, I coded up an example, and I asked our designer, "Hey, can you go ahead and review this for me?" And she gave me such excellent feedback of, "No, this looks nothing like a snowman. You, you, you create- you turn Claude into Mr. Peanut." I don't know if Mr. Peanut's an American brand. In, in, in the US, we have this jar of peanuts and, you know, their, their mascot is this little peanut character. And I was like, "Holy crap. That's totally right. I, I thought he was a snowman, but you're right, it's a peanut." And so that's where, like, humans also, like with that product sense, it's really important to keep that expertise in the loop as well. Now, what should my team makeup be? 'Cause roles are blurring and Claude is augmenting. And so when it comes to engineering, these are the two profiles that I now really, like, focus on. One is creative builders with product sense, um, and then another is deep system expertise. And so this is definitely like, you know, the hard part. Want to make sure that we can still have that trust but verify. Maybe I'll talk a little bit about product sense. When I gave this talk in San Francisco, afterwards a lot of, uh, engineers came up to ask me a question of, "How do I build this product sense muscle?" So I figure I will share with you all of what I have found helpful, and it's helpful for me, and maybe it will resonate with you all. Like, for me, it's this muscle that you build with... It first starts with dogfooding, and especially for managers, actually. Like, making sure you're spending time using the product that you're building and your team is building. In fact, most of the time, any time I join a new team, like, that's one of the first things I do. I'm, like, dogfooding the product to make sure I understand it. 'Cause this way you really start feeling your product in your bones, and you remem- you always ask yourself, "Wait, when I was building this product, what problem were we trying to solve? Like, what experience was I trying to enable for our users?" So with that in mind, like, dogfooding is a really good way, especially for managers who, before Claude, you may not have time to be in the code base anymore. Um, but yeah, like actually, any time I joined a team and I would dogfood, a lot of team members would always come up to me and say, "It's so awesome to see you using our product 'cause you really care." 'Cause that's a way for you to experience the work as well. So for me, product sense comes from a lot of dogfooding, and then also, like, iterating, shipping, and then going out to speak to customers. One of my personal passion projects is working with small businesses. I love small businesses. I feel small businesses are the lifeblood that forms our community. And, uh, we actually just announced Claude for Small Business, which is a project I'm so excited at. Um, but I have a lot of friends that run restaurants. They m- and they work incredibly hard. And so I met with a lot of them to onboard them onto Cowork 'cause I wanted to see how, you know, like outside of enterprise with small businesses, how do our users actually onboard? And I got such valuable feedback. It's actually really humbling to see where in the onboarding flow we... There were so many things that we could've been doing better. So I really encourage you all to, to dogfood your product so then you really, I call it like feeling it in your bones, 'cause it really g-keeps you, um, giving you a sense of what your product is doing. If not, after a while, you might find yourself making product decisions that's all just based on either metrics or dashboards or PowerPoints. So definitely dogfood, dogfood, dogfood, or as we call in Anthropic, ant food. So filling cross-functional gaps with Claude. This is interesting 'cause I actually see this for all roles. So for example, designers on Claude Code team previously, you know, any time we wanted to make polish or UX fixes, designers would go ahead and create that redline and then hand it off to engineering. And it's always a wait, you know, f- 'cause engineering is always about, "I have to build this product, I have to fix this bug. When will I time to actually do this polish?" Designers on Claude Code actually, you know, use Claude to make a lot of those polish fixes, so then it really closes that iterative loop a lot faster. And again, this goes into why it's so important to do the verification. On the flip side, for me, like for example, one time I was fixing a bug, and I needed a little bit of content help. I, y- I don't know about you all, but as an engineer, I tend to be very verbose, and when I'm like, you know, if you ask me to write a very short sentence to describe what something's supposed to do, I tend to go too much into the weeds. But Claude was a really great content design partner for me. And so that's what I found really interesting on Claude Code team. For all roles, Claude is actually augmenting, uh, all the areas where you may not be as strong. So this was the manager slide that I was like, "Oh, this was, you know, maybe a good spicy topic." But when I first joined Claude Code, this was one of the first things I changed of like, "Hey, I, I was working with the recruiting team to go," uh, because I believe so much in dogfooding. And on Claude Code, we're using Claude to build Claude. We're using Claude Code to build Claude Code. And, uh, so every manager started as an IC first. I actually felt this worked really well 'cause it enabled managers before you have to worry about the responsibility of supporting people, actually having time to, you know, like roll up your sleeves and get into the code base and learn what it's like to be an engineer on the team. Um, maybe I'll also, you know, like touch on this is also a great time that if you're a manager to actually get some maker hours back and actually get back into the code base 'cause the onboarding is a lot less daunting than it used to be. Um, actually even before Anthropic, there were different roles I've had where I've switched between manager and IC, and I think every time I switched to an IC, it really helped me build up the engineering toolbox. But that onboarding was always a lit- like, you know, that slight hesitation 'cause it could be daunting if you haven't done it in a while. But, like speaking for myself, I remember when I onboarded to Claude, it was, uh, it was such a different experience. And actually, that's what [laughs] I'll go into next. So in terms of knowledge sharing, what becomes a new source of truth? So when I was onboarding to Claude, the code is our source of truth. And so, uh, again, like in the old days, any time I join a new team, I'd meet with all the engineers to... I still do meet with all the engineers on the team, but now we talk a lot more about what's on top of mind and-Because before, I would always so, like, do these tech deep dives. And now I actually did my first, you know, like, tech deep dive just with Claude, like, because Claude has-- is really good at answering all these questions I have. So even when I was doing that first bug fix, I remember asking Claude, "Hey, before I dive into this bug fix, can you actually teach me a little bit about the surface area and also what aro- what about all the areas around that bug as well?" And so for Claude Code and, and Cowork team, our code is a source of truth. I would encourage all of your teams, like if whatever is your source of truth, where- whether it's a spec, you might actually change that into a skill that you check into the code base. Like, whatever you can make sure becomes your source of truth, keep in the code base, 'cause this way you can actually keep it up to date. 'Cause that's the other thing. When coding, you know, bandwidth is so much, you know, like, so much more. It also makes it easier for any documentation that also doesn't-- isn't part of the update loop to get out of date. So how did we roll out all these changes on the Claude Code team? There were things that we-- was really important that we aligned as a team, and then there's also I wanna make sure we also give space for bottoms-up because each team is focused on different things and there might be, you know, like, different tool chains or anything that resonates more with each team. So what did we align with kind of like the teams as a must-do? These are some of our Claude Code team principles and what we call forcing function. And this slide is out of date. Like, I shouldn't put every engineer. It's every teammate uses Claude Code and also Cowork actually. So, like, us using our product day and day has been really important. Claudify everything we can. And so like anytime we have a question of, "You know what? Yeah, I'm doing this. What's better than me doing it? Can Claude do it for me?" 'Cause then it actually frees up more of my bandwidth for other problems to solve. And my last one is, uh, explicit permission [chuckles] to kill old processes. Again, it's the e- even our team principles and even processes as we put on, even after a few months when we notice, "Hey, is this really serving its intended purpose?" We really always give ourselves permission to always critique and make sure to always revisit. And but bottoms up, there's a lot of freedom for, you know, like, you know, team or pods to adapt. So for example, how Claude shows up in team triage or any teams like planning rituals or standups or which workflows get Claudified first, that is a, a lot of bottoms-up. So I found that this balance of make sure you align with the team of-- in terms of, like, what's important in terms of team culture and also update those as you go, but then leave some room for each pod to adapt. So if I zoom out, what were the three things that I prioritize for Claude Code and Cowork that I think has worked really well? Uh, number one, it was, you know, like keeping the org agile and as flat as possible. Having managers also, you know, like, support pods of work, but, like, also really get into the code base and d- be directly responsible for parts of the product as well. Uh, number two, that's our Claudify. So if Claude can do it, Claude should. So that's always a question we're asking. And, and maybe that's the other interesting too. Like, um, you know, this morning you heard the talks about the models. The models are improving at an amazing pace. So sometimes we might find even if Claude wasn't good at doing something two or three months ago, now with, like, a model update, it's actually gotten really good. And this is why, again, it always goes back to that whole, like, always revisit and always have the growth mindset 'cause the models are also improving really fast. And again, you know, people don't delete processes on their own. They tend to pile up. I remember I was on a team where we had so many different SLAs. There was an SLA for P0 bugs, an SLA for incident response. And after a while I'm like, "With all these SLAs, I need a stacked rank list [chuckles] to actually have..." 'Cause even the engineers are coming to me going, "There's only twenty-four hours in a day. Like, which SLA am I supposed to do?" And but that's why it's always so important to audit. I remember at that time I'm like, "You know what? We really should be auditing are all these SLAs still really necessary?" So now for the proof. What are some signals that we're seeing of, like, is it actually working? So these are some numbers that I find as helpful that I wanted to share with you all. And so these were some of the shifts we've seen as we've been kind of, uh, improving our workflows. Onboarding ramp-up time goes down. Like, I, I think that's actually a really good signal of, uh, like we used to have this thing of, like, h- what's the span of time it takes for the engineer to land their first PR? Uh, the onboarding ramp-up time and also, like, cost to other team members we've noticed keeps going down. Like, I, I, I even, like, a lot of times sometimes when a person joins a team, they'll ask me a question, I'm like, "That's a great question. We should ask Claude." Like, that's how we kind of, uh, like Claude has really helped us to onboard and answer a lot of questions. Which is also why for the managers, I-- it's why it's fun for me to actually get back into the code base. 'Cause in the old days I would actually feel bad 'cause I feel like, "Oh, I need to learn this area. I'm gonna ask an engineer and take time away from their day 'cause they're all so busy." But now thanks to Claude, I don't feel like I'm wasting anybody's time. The PR cycle time should also go down, but this one's interesting. I'm gonna say it's important for you to not just look at the end-to-end, but actually break it out into, like, what are all the different funnels in terms of a PR. 'Cause if your PR cycle time isn't going down, it might not mean you're not adopting AI. It could be other parts of your queue is actually getting jammed up because of all the throughput. So for example, are your build and CI systems keeping up? So for PR cycle time, I think it's actually really healthy to break it down into all the different chunks and seeing which of those you might be able to improve or automate. And lastly, Claude-assisted commits go up. As I mentioned, I don't think I've ever, like in the last few months I've not seen a commit that wasn't, uh, you know, like assisted by Claude. But if I step out, like I would say what's m- even more important than just throughput is whatever, you know, you and your team are trying to solve, whatever the product or problem, find some way to measure that 'cause outside of just throughput, hopefully we're also making meaningful change to make the product better or improve quality.Okay, so now it's time for me to maybe share like a little takeaway home exercise that might be fun for you to take back and go have discussions with your team. But first I want to be really transparent. I still have questions as well, so I'll share some of those that we're still working out through the Claude Code team. So for example, do you still need separate iOS and Android orgs? Like, that was kind of like more of the traditional org approach of you would usually have like one for iOS, one for Android. But now with Claude and enabling our engineers to go flex across different mobile platforms, we're thinking, "Hey, do we really still need the, the, you know, two separate mobile teams?" Uh, how far do you push fully automated reviews? We always want to make sure we're striking that right balance between kind of speed and fast enough and make sure we're not losing something important. And roles are blurring, and how do we make sure everybody is productive? And that's why we're, we're right now looking through verification as a big one so that, hey, as we're going through, make sure everybody that makes a change has confidence in their change. But also, what are all the different parts of the workflow too? Like, I'm working with my designers. We actually just, you know, announced Claude Design. Also thinking through how can we make sure that we're also improving the workflows of all the functions on the team. So with that, this is what I would love to leave you all with. Pick your noisiest workflow, or it could be just some workflow or team meeting that you don't particularly enjoy, or it's a really high tax on the team. Like, I remembered I was on another team where we would have this really expensive weekly meeting, and I notice everybody's on their laptop and, you know, except for when they have to pop their head up to give status, and then they go back to their laptop. I'm like, "This is... When you count, you know, like how many people's in this room, this is a very expensive room." So always like kind of like pick a workflow and always ask yourself, "Is this still serving its purpose?" Or if it's like really expensive, is this something that Claude might be able to help you with? And kind of like do this one step at a time. And with that, thank you so much for listening to my talk. [laughs] [clapping] It's been a pleasure. [laughs] [upbeat music]

Episode duration: 26:37