The future of AI in software development | Inbal Shani (CPO of GitHub)

1. 0:00 – 4:17

   Inbal’s background

   1. ISInbal Shani0:00

      ... the user of the AI tools to develop software needs to form a different thinking. You need to start figuring out how are you using these AI tools to help you be successful. And it's no longer just the actual code writing, it's really evolving your thinking to the big picture, to the connected experience, to connected systems. Which today, we just find it more in the world of more senior developers and less and less in the junior developers. The junior developers when they start, usually we expect them to be able to write simple codes. But if now there is an AI assistant that is helping them writing code, they can spend more time from the get-go understanding the system, underce- understanding the environment that they're building or understanding that product that they're building, which today they don't have time because they are still learning how to code.

   2. LRLenny Rachitsky0:45

      (instrumental music) Today my guest is Inbal Shani. Inbal is Chief Product Officer at GitHub, formerly a general manager at AWS and at Microsoft, and also Senior Software Developer on Amazon Robotics. In our conversation, we explore the end state of Copilot and AI-based tooling for software engineers, what the future of software development looks like with AI, what's under-hyped and over-hyped in AI and software development, what metrics the teams look at to understand if Copilot is doing its job, what mistakes teams and companies make jumping into AI, the design philosophy of Copilot, plus what's helped Inbal most become the leader she is today, and also where GitHub is heading as a product and an organization. With that, I bring you Inbal Shani after a short word from our sponsors. You fell in love with building products for a reason. But sometimes the day-to-day reality is a little different than you imagined. Instead of dreaming up big ideas, talking to customers, and crafting a strategy, you're drowning in spreadsheets and roadmap updates and you're spending your days basically putting out fires. A better way is possible. Introducing Jira Product Discovery, the new prioritization and road mapping tool built for product teams by Atlassian. With Jira Product Discovery, you can gather all your product ideas and insights in one place and prioritize confidently, finally replacing those endless spreadsheets. Create and share custom product roadmaps with any stakeholder in seconds, and it's all built on Jira, where your engineering teams are already working, so true collaboration is finally possible. Great products are built by great teams, not just engineers. Sales, support, leadership, even Greg from finance. Anyone that you want can contribute ideas, feedback and insights in Jira Product Discovery for free. No catch. And it's only $10 a month for you. Say goodbye to your spreadsheets and their never-ending alignment efforts. The old way of doing product management is over. Rediscover what's possible with Jira Product Discovery. Try it for free at atlassian.com/lenny. That's atlassian.com/lenny. This episode is brought to you by Sanity. Your website is the heart of your growth engine. For that engine to drive big results, you need to be able to move super fast, ship new content, experiment, learn, and iterate. But most content management systems just aren't built for this. Your content teams wrestle with rigid interfaces as they build new pages. You spend endless time copying and pasting across pages and recreating content for other channels and applications. And their ideas for new experiments are squashed when developers can't build them within the constraints of outdated tech. Forward-thinking companies like Figma, Amplitude, Loom, Riot Games, Linear, and more use Sanity to build content growth engines that scale, drive innovation, and accelerate customer acquisition. With Sanity, your team can dream bigger and move faster. As the most powerful headless CMS on the market, you can tailor editorial workflows to match your business, reuse content seamlessly across any page or channel, and bring your ideas to market without developer friction. Sanity makes life better for your whole team. It's fast for developers to build with, intuitive for content managers, and it integrates seamlessly with the rest of your tech stack. Get started with Sanity's generous free plan, and as a Lenny's Podcast listener, you can get a boosted plan with double the monthly usage. Head over to sanity.io/lenny to get started for free. That's sanity.io/lenny.

2. 4:17 – 5:54

   Why generative AI is not going to replace developers in the near future

   1. LRLenny Rachitsky4:18

      Inbal, thank you so much for being here and welcome to the podcast.

   2. ISInbal Shani4:22

      Awesome. Thank you for having me. I'm excited to be here.

   3. LRLenny Rachitsky4:25

      I want to start with just, like, a broad question about where software development is going. It feels like you're at the center of the storm of what is changing in software development. Almost feels like GitHub kicked off the storm with the launch of Copilot. So let me ask, what do you think is over-hyped in terms of what is changing in software development? And then what do you think is under-hyped?

   4. ISInbal Shani4:46

      So I joined GitHub as a CPO basically a year ago. Two days ago, I celebrated my first year. And for me the initial approach is let, let's look at the platform, let's look at the software development life cycle a- and draw some conviction. And for me, the biggest conviction is that developer happiness and productivity strongly depends on their, their environment and their tooling. And as such, AI is really critical to that mission. When we're looking into what's happening right now with AI, it really became stable stakes on ex- expectations among developers, and it's really central and critical for them to be able to do their job. Uh, we know that something like 92% of developers are already using AI tools. But in that landscape, some of the things that are over-hyped is that generative AI will replace humans. I don't see that happening in the near future. The way I think about it, you always need that human in the loop because AI cannot replace innovation, right? That, that creative spark, that creative thinking that is the center of humanity, this will not be replaced by AI, at least not in the near future.

3. 5:54 – 7:48

   Why AI-driven testing is underhyped

   1. ISInbal Shani5:54

      If we think about what, what does that mean having an AI tool that is successful, you need data. In order to generate data, you need humans using tools, acting with tools, doing something something. So I think the over-hype that is happening right now is that...... generative AI will replace humanity. It's going to be the solution for everything, and I think it's a tool. If we're drilling down to the things that are under-hyped right now in the world of software development, I'm starting hearing a little bit more on AI-driven testing, but there's not lots of mention on it. And if we're thinking about the world of AI, where we're going to generate much code, much more productive, much more efficient, more lines of code, more application, then testing is becoming such a critical part of that journey of developing software. And I would like to hear more about that and what companies are thinking about, how do they use AI to generate, uh, a larger suite of testing to be able to validate their work?

   2. LRLenny Rachitsky6:53

      And by testing, you mean like unit testing, integration testing, things like that?

   3. ISInbal Shani6:56

      All of them. Load testing-

   4. LRLenny Rachitsky6:58

      Load testing.

   5. ISInbal Shani6:59

      ... infrastructure testing, um, and security testing, penetration testing. Think about, uh, when you need a human to write all these testing capabilities, you need a lot of humans. That needs to form kind of different thinking. Can we leverage AI to really generate that testing suite, that test everything that you need, from unit testing and functional testing, to load testing, to performance testing? There's a lot of testing that we do that we don't even consider them as testing, so we don't necessarily spend a lot of time in doing them. But if we're now saying software is in the center of everything, all companies will become a software company in some form or another, there is more code being generated. Hence, the importance of testing is becoming much more critical.

   6. LRLenny Rachitsky7:43

      Let's follow that thread. You talked about how you don't see software engineers completely

4. 7:48 – 10:13

   What the next 3 to 5 years will look like

   1. LRLenny Rachitsky7:48

      disappearing. What's your just general sense of how software development will be different in, like, three to five years? What do you think is gonna be different? And then also just, how do you see the end state of all of this, of Copilot and tools like this getting better and better and better? Where do you think this ends?

   2. ISInbal Shani8:02

      For me, and, and I keep on saying that again and again, Copilot is a copilot. It's not a pilot. You still need the human in the loop. But that means that now, the software developer or the user of the AI tools to develop software needs to form a different thinking. One, you need to start thinking more systems, more architecture. You need to start figuring out, how are you using these AI tools to help you be successful? And it's no longer just the actual code writing. It's really evolving your thinking to the big picture, to the connected experience, the connected systems. And what we will see in the fullness of time, the developers, one, will adopt much more of that mindset, which today, we find it in software development. Don't get me wrong. We just find it more in the world of more senior developers and less and less in the junior developers. The junior developers, when they start, usually we expect them to be able to write simple code. But if now there is an AI assistant that is helping them writing code, they can spend more time from the get-go understanding the system, understand- understanding the environment that they are building or understanding that product that they're building, which today, they don't have time, because they are still learning how to code. So I think that's one element. The second element that I expect to see changes is in the world of hardware development. Because we know that AI has a, uh, has a big footprint and requires a lot of resources. And in order to be able to meet the scale, to meet the demand, we need to be able to improve our hardware, our CPUs, our GPUs, our compute, and optimization of code. And that's another area that today, it's a niche of people that are spending time, and I think it will become much more, uh, aligned across the software development, uh, life cycle. Integration with AI is something that will definitely become something that is more, um, normal for every developer, is today something that we start seeing that revolution, that acceptance that AI tools, they need to figure out how to integrate with AI, they need to know how to use AI, it will, will become something that is more custom. And we, we will start seeing that shifts from very early on. When developers are start shaping their thinking, they will starting understanding all these components.

5. 10:13 – 12:07

   Stats around the use of GitHub Copilot

   1. ISInbal Shani10:13

   2. LRLenny Rachitsky10:13

      I saw a stat that at Shopify, over a million lines of code in their code base was written by Copilot, which I don't know if, uh, I can do the math, but I imagine that's more than one person can, would have written in their lifetime, and probably many engineers write in their lifetime. And I think that number reminds us just how fast things are moving and how much is actually changing.

   3. ISInbal Shani10:35

      Right.

   4. LRLenny Rachitsky10:35

      You also mentioned 92% of engineers are using AI tools already. Is there any other stats that you can share that might surprise people about the usage of Copilot, growth of Copilot, things along those lines?

   5. ISInbal Shani10:46

      Yeah. Well, we have over 37,000 organization and more than 1.5 million developers that are using Copilot.

   6. LRLenny Rachitsky10:54

      Wow.

   7. ISInbal Shani10:54

      And they're writing codes, based on our surveys, 55% faster. So if... Imagine that you can give a software developer even few minutes, half an hour back in every given day, how much more productive they're becoming, and also ... how much happier they're, they're becoming. So we know that in a survey that we have run through our users is 85% of the participant felt more confident in their code quality, and also code reviews were completed 15% faster without Copilot. And we know that usually no one likes to do code reviews. So if we can help accelerate and make developers take their time in doing code reviews. And also, if we think about removing frustration, then 88% felt less frustrated and more focused on their ability for coding. And that is just the beginning. We have different examples from our customers. I think Accenture is a recent one that we got, where they had 88% of suggested code was retained. So we have a lot of these stats that are coming together to showcase the size and the impact of using, uh, Copilot and AI tools for software development.

6. 12:07 – 13:38

   How Copilot enables engineers to work more efficiently

   1. ISInbal Shani12:07

   2. LRLenny Rachitsky12:07

      Say companies hear these stats of like, "Oh, we're gonna be 25% more efficient," some people may think, "We need 25% less engineers. We're gonna do all the same things."I imagine what you're actually finding is your engineers can do more. So it's not like you need to cut your people, it's that people can be faster and better.

   3. ISInbal Shani12:23

      No, l- let me be very clear. You cannot cut your people. You have to have a human in the loop. Copilot is a co-pilot, it's not a pilot. And I keep on saying that sentence again and again. I think the second thing that what companies need to realize, and that's another developer experience survey that we have run recently, that throughout the years we have added so much tasks on a single software developer. From writing code, writing testing, interacting with people, collaborating with 21 people in every given week, going to meetings, waiting on builds, uh, digging into legacy code. All of that is added to the fact that they need to write code, which is their basic work. So most developers spend less than 25, some say less than 20% of their time writing code. So if we're able to give them half an hour back, one, they can write more code. Second, they can have a break and, and take a breath, so we don't burn them out and they're more happy. Third, we give them more time for collaboration and creative thinking. So that sparks innovation. That is not something you wanna cut back. This is something you wanna be able to give as a gift to your developers so they're happy with you and you can retain them, they can do their job, they're productive, hence they're happy.

7. 13:38 – 16:42

   Common mistakes when adopting AI into your workflows

   1. ISInbal Shani13:38

   2. LRLenny Rachitsky13:38

      I've talked to a bunch of engineers that use Copilot. One thing that surprised me is the most amazing engineers love Copilot. You think this would be like for junior engineers learning to code better and faster, but like 10x engineers say that... Like someone told me it made them 50% better and faster too.

   3. ISInbal Shani13:56

      Right.

   4. LRLenny Rachitsky13:57

      Which is incredible.

   5. ISInbal Shani13:58

      Right.

   6. LRLenny Rachitsky13:58

      What mistakes do you see product teams and engineering teams make when they kind of rush to start adopting AI, integrate AI into their workflows?

   7. ISInbal Shani14:07

      I think there are a couple things. The first one is, companies expect a change to happen magically. Here's a tool, go use it. And it's not always flying the same way across all companies. So investing time in really taking the company through a change management process. The second thing that I'm seeing is that since AI is so hyped right now, there is often the questions of, "Oh, what should we do with AI?" So there is that sense that I have to do something with AI because if not, I'm not doing the right thing. Versus the, the question that for me is a big focus for, for my product team is, what is that problem that we're trying to solve? And how can we leverage AI better to help solve the problem versus what do we do with AI? So it's really working backwards from the customer problem, from what we're trying to solve, and then realize what are the best tools that we have in order to do that work better and think through that landscape. Versus, "Oh, AI is, is here, we have to use it because if not, we're not doing something right."

   8. LRLenny Rachitsky15:11

      Is there an example of that that comes to mind if someone that did that incorrectly or wasted a lot of time?

   9. ISInbal Shani15:17

      It's, it's not necessarily examples, but I've been talking to some of our customers and, and they're coming to us. "Uh, we heard about AI. What do you think we should do with AI and how should we adopt AI? And can we learn from your experience?" And, and I'm sharing our experience when we started thinking about AI it was with the idea to make developers much more productive. And what are some of the things that we've seen that developers have multiple tasks on them and they don't have enough time to spend on coding. So how can we give them time to spend more on coding and what are some of the coding element that we can automate? And from that comes the need to, okay, let's incorporate OpenAI, let's incorporate ChatGPT, let's con- connect all of them to m- to help create that tool that helps developer be more productive and is AI based. So I'm sharing how we went through that journey to think about how to use AI and that I, I see a lot of the times light bulbs with the customers like, "Oh, we didn't think about it like that." It's more about we wanted to plaster AI on a lot of things versus I have this workflow and it requires a lot of manual work or it requires a lot of configuration. Maybe I can think how to incorporate AI into that and really shorten the time or make it, uh, seamless or make it less friction for developers to adopt that tool and so on and so forth. So it's really sharing more how we are thinking about it and, and helping our customers to understand the same.

8. 16:42 – 18:46

   How GitHub operationalizes “dogfooding”

   1. ISInbal Shani16:42

   2. LRLenny Rachitsky16:42

      Along the same lines you're talking about how you think about it with your teams, feels like your product teams are probably living in the future of how other product teams will operate because you have access to stuff everyone else d- doesn't necessarily have access to. They probably adopt these tools a lot faster than the other teams. What are some ways that your product teams operate that other teams may not yet be operating in?

   3. ISInbal Shani17:04

      In GitHub in general, it's not just the product team. GitHub is GitHub. Uh, I, I'll say we eat our own dog food. And that means that we are the first to try every feature, every capability that we're developing. And it's not just the, our engineering team, it spans across GitHub. So for example, GitHub is running on GitHub. There are several blog posts and talks. Even in the recent universe, we had a specific talk in terms of how GitHub is using GitHub and adopting AI across software development lifecycle. And the biggest focus for us is if we're saying that GitHub is a developer platform and it's more than that, it's a software development platform, how are we making sure that all the persona that are taking part in the development of GitHub are also using GitHub? So for example, our finance team is using, uh, discussions and posts and PRs and reposts to, to communicate our AR numbers and so on and so forth. We have our legal team using, we have our, our HR team, if they like it or not, it's a different question, (laughs) but the idea is that we're using GitHub to run GitHub and the product team is, is one of the first early adopters along with our engineering team. So for example, when we've tested chats and we wanted to see the quality or the recommendation, the search, this is something that the product team was one of the first users to go and figure it out.... is it giving me the right thing that I need to do? That, is it summarizing the PR the way I wanna summarize? And, and so on and so forth. But we are really strong in advocating that we have to be the one testing our tools. If we cannot use them, our customers cannot use them, for sure. So nothing is shipped before it spends enough months cooking inside GitHub, so we have a say if this is working for us or not before we put it at the hand of our customers.

9. 18:46 – 20:24

   The philosophy behind Copilot

   1. ISInbal Shani18:47

   2. LRLenny Rachitsky18:47

      One of the magical elements of Copilot is how it just kind of fades into the background. You don't have to chat with it, you don't have to tell it anything, it just infers, "Here's what you're doing. Here's the answer for you if you wanna use it, that's cool. If you don't, that's fine." Is there anything you could share about just the design philosophy of Copilot?

   3. ISInbal Shani19:05

      Yeah. Uh, the, the design philosophy for Copilot is very much aligned with the working backwards concept that I just mentioned. It's really putting yourself in the shoes of your customers and figuring out what is it that they need? How is that experience going to work for them? If it's an extra tool, and if you need to ask for it, and if you need to ask for it or if you need to wait for it, then developers will not adopt it. So because we develop it for ourself first, and we wanted to experiment with that, the design philosophy was put yourself in the shoes of a developer, how would you like that experience to be? You know, in 2020, we had, uh, a group of GitHub engineers that were working with the design team, and with the OpenAI GPT to really figure out how we're going to build that, and how it can work to help developers to do their job more efficiently, more productively, um, improve their collaboration and happiness and so on and so forth. But the grounding principle was a developer needs to want to use this tool. And the more you add friction, the more you add churn, the more you add complexity, developers will not wanna use their tool. We just talked about all the things that the developer needs to do in every given day. Now, add to that mix, "I need to learn how to adopt AI," we knew that it will not fly. So it has to be a seamlessly integration, it needs to be something that is very intuitive for a developer to use to be successful.

10. 20:24 – 24:54

    Copilot’s success metrics

    1. ISInbal Shani20:24

    2. LRLenny Rachitsky20:24

       What are the success metrics for Copilot? It's such an interesting problem of measuring efficiency gains and productivity gains for engineers. What metrics do you focus on to tell you this is doing what we want it to be doing?

    3. ISInbal Shani20:35

       We are in a world that there are no right metrics. Uh, there is no one, uh, one metric to rule them all. It's a combination of the things that you're looking to measure out of adopting AI. You can think about measuring code quality using AI. Can you improve the code quality? Can you improve the security of your code by introducing, for example, our GitHub Advanced Security using AI? So there's a lot of different metrics that if you combine them together are providing you with that developer productivity. And then there's developer productivity, devel- developer collaboration, time gain, that if you're bringing them together are yielding kind of the developer happiness. The biggest area of focus for us right now is the definition of productivity-

    4. LRLenny Rachitsky21:20

       Mm-hmm.

    5. ISInbal Shani21:21

       ... because we have so many users that are coming to GitHub and they are looking for that productivity gain. But one of the things we're very conscious of as we continue evolving Copilot and AI across the software development life cycle across all of the tools, productivity is not the right metrics against each one of these components. When we're implementing AI into GitHub Advanced Security, writing more secure code is the right element. It's like, how many secrets were we able to prevent from leaking? How many issues in the code were we able to detect and find and fix before you ship that? So I think that the world of the metrics for AI is really a big one. We are working a lot with our customers as they're asking the same questions, and they're running their own experiments within their companies to figure out what is it that they wanna measure. The most easiest one is time, but time is... and it's funny what I'm going to say, but w- uh, time is not quantifiable as a success metrics because you can write really bad code really fast. So it's really about how are we taking time and translating it to efficiency, to productivity, to something that are harder to measure? And then how does that lead to developer happiness, which is another harder metrics to measure. So the combination of all these input metrics leading to eventually developer happiness is what we are focusing on.

    6. LRLenny Rachitsky22:40

       Yeah. It's so interesting 'cause engineers could end up spending more time coding because they're enjoying it more, they're getting more done. You also, like more lines of code isn't a good metric 'cause you won't know if those are good lines of code. That's like a classic-

    7. ISInbal Shani22:52

       Right. Right.

    8. LRLenny Rachitsky22:52

       ... way to not measure engineers well. So to me it feels like developer happiness is the ultimate metric, and there's... Uh, we had, uh, Nicole on the podcast who came up with this framework, DORA. That's an interesting way of just measuring developer experience, developer happiness.

    9. ISInbal Shani23:05

       Right. Uh, I think one of the things that we're talking to customers and we're trying to figure out how we articulate is instead of time, can we talk about time to value? So from the moment you put a developer on a task, how long did it take you until you realized the full potential or the full value of that? If it's, uh, generating revenue, if it's in adoption, if it's time to market, like you can define your time to value in different ways. But eventually, you're investing in AI tools to help your developer to be more productive, to be more efficient, to be more happy. And then is it impacting your time to value, which is something that, uh, that's the business side that they're looking to measure.

    10. LRLenny Rachitsky23:44

        This episode is brought to you by HelpBar by Chameleon, the free in-app universal search solution built for SaaS. Your help content lives outside your app and is scattered in many places, forcing users to waste time hunting for answers. HelpBar solves this. It delivers answers directly inside your app and eliminates context switching. Users can search or ask questions to get AI generated answers and lists of the most relevant documentation from all of your help sources, including your knowledge base, docs, blog, and video libraries. You can also use HelpBar to navigate your app and launch actions, such as scheduling a meeting or viewing an interactive demo.The best products today use Cmd+K for in-app search and navigation. HelpBar makes that readily available within your app, without engineering or new code. Give users a faster and more delightful self-service experience that reduces friction and increases in-app engagement. Upgrade your user experience with this modern component, and supercharge your product-led motion. Sign up for HelpBar today. It's free and easy to set up in minutes. Check it out at helpbar.ai/lenny. That's helpbar.ai/lenny.

11. 24:54 – 26:37

    How Copilot encourages collaboration

    1. LRLenny Rachitsky24:54

       Coming back to where this all goes, I imagine you've seen all these demos of people like sketching a website, taking a picture, sending it to ChatGPT and it builds the website for you?

    2. ISInbal Shani25:04

       Yeah.

    3. LRLenny Rachitsky25:05

       How do you see all of that sort of thing playing into this? Do you see like CEOs being like, "Here's the iPad app I want. I'm gonna sketch it for you, and then just run it through Copilot, version 10, and it writes it for you?" I know that you're, you're saying engineers won't be cut, and only increase and become more efficient, but I guess where do you see that version of it going, of just like here's a sketch, build it for me?

    4. ISInbal Shani25:27

       I'm thinking about that as a better collaboration tool, versus a production tool. Because right now, a lot of the time where we see challenges in articulating ideas is the communication, it's the clarity of thoughts. So if we can leverage AI to improve collaboration, and you basically put a drawing in front of Copilot and it helps tell the story of what is it that you're asking your developers to do, it shorten a lot of cycles in the back and forth. So what did you mean? Did you mean the button here or did you wanted it pink or did you wanted that application or did you wanted that operating system? So if Copilot can help refine communication, and it's becoming likely the best collaboration tool that exists (laughs) in the market, uh, especially in the global world where we are, where we see communication is an ongoing challenges between different personas that are taking part in coming up with an idea. And, and basically we see AI a- and, and in our view as Copilot becoming that next natural language conversation. It's that translator that helps bring everyone on the same page, because it's creating that universal conversation language, the same that math used to be. (laughs)

    5. LRLenny Rachitsky26:36

       Mm-hmm.

12. 26:37 – 29:35

    What we lose when AI writes code for us

    1. LRLenny Rachitsky26:37

       Reminds me, there was another conversation I was having with an engineer, and he was saying how he like loves writing those simple functions...

    2. ISInbal Shani26:44

       Mm-hmm.

    3. LRLenny Rachitsky26:44

       ... that Copilot is now doing for him, and he's like sad that it's doing it for him. But he doesn't turn that off, but he kind of is like, "Aye, that's what I really enjoy, these little things that I could just crack, like sort function of some sort." I guess, is there anything you think we lose with so much of our code being written for us?

    4. ISInbal Shani27:01

       I think each developer have their, their own part that they enjoy doing. And you don't have to use Copilot to write your simple code. You can use Copilot through a chat experience and, and use them as an AI assistant and brainstorm. We're giving you a set of tools. And we have a scenario in mind how you should use them. But what we're seeing is developers are using them very differently. Everyone is choosing their own flavor on how to use the tools. There is no right and wrong way to use it. It's a tool. It's a language. Whe- when, you know, when I was, uh, learning how to code, I used to code in C, and then, uh, and then Java became a thing. I'm like, "Oh, but this is taking away my job." Because I used to write all these functions, and all these libraries, and now Java has it all. And then Python came, which is another obstruction layer. It's like, okay, so now I don't even need to think about that, because Python can do all of that. So it's a matter of how you use the tools and what you use them to do what. There are areas that I will still go and write in C, even today. (laughs) I don't trust someone inverting metrics for me (laughs) in an efficient way if it needs to run on a very small CPU. So I'll, I'll do that myself. On the other hand there is going to be elements that I'm going to delegate to a higher obstruction tool. And that's exactly wh- the way I think about AI. You pick and choose how you want to use it. There is no global way that is the absolute truth that everyone has to follow. You need to think how you use your tools to do your job in a way that you're still keeping the things that make you happy. On the other hand, use AI to do the things that you less enjoy. Maybe you like writing functions but you don't like writing testing, so generate unit testing and so on, and let AI do its job for you, and you still focus on the things you enjoy.

    5. LRLenny Rachitsky28:48

       Are you actually still finding time to code? Are you still writing some C on the side?

    6. ISInbal Shani28:52

       A little bit.

    7. LRLenny Rachitsky28:53

       That's great.

    8. ISInbal Shani28:53

       Now, now I'm mainly playing with my older son. Um, so he, he has more time... (laughs) to code. But sometimes I enjoy spending time with him and kind of brainstorming and, and talking about things and architecture. I think recently, a few months back, uh, he did a project on sensors and sensors integration, and he came across Kalman filter, which was the thing that I used to do many years back. Uh, and he's like, "Okay, what is it and how is it done?" And then we, we went through that and I helped him kind of think about the structure and the sensors fusion and all of that. And then sometimes I'm like, "Okay, move aside. Here you go. This is helping you." (laughs)

    9. LRLenny Rachitsky29:28

       (laughs) Turn on Copilot.

    10. ISInbal Shani29:30

        (laughs)

    11. LRLenny Rachitsky29:30

        I'm just kidding.

    12. ISInbal Shani29:31

        Mommy's Copilot. (laughs)

    13. LRLenny Rachitsky29:33

        That's so funny.

    14. ISInbal Shani29:34

        Yeah.

13. 29:35 – 30:47

    A retrospective on the generative AI space

    1. ISInbal Shani29:35

    2. LRLenny Rachitsky29:35

       Um, that reminds me, I was watching a talk of yours, and you were training tuning models and LMS way back in the day before it was very cool.

    3. ISInbal Shani29:43

       Right.

    4. LRLenny Rachitsky29:43

       And so you've kind of been in the space for a long time. Looking back, is there something that you see now of where things are going that you didn't see at the time?

    5. ISInbal Shani29:51

       I, I grew up in a world of niche AI, where you have to be an expert, and you need to be trained for years understanding how to tune a model, what is the right output, how, and you needed to understand how to build a simulation to test it, how to ingest data, focus on optimization. I've seen the evolution of AI throughout the year with the, um, you know, all the talking devices and the conversational AI, and then it started as a single turn, it came up as a multi-turn, so I started seeing that trend of more democratizing AI.... but I didn't expect that boom of generative AI that everyone that even, doesn't even know what AI is (laughs) thinks that they need to use AI. I think that was the thing that caught me by surprise, because I was so trained that you have to be an expert to use AI. So for me, that was aha moment, like, "Hmm, this is interesting. So what's next?"

14. 30:47 – 32:35

    Inbal’s thoughts on the future of AI

    1. ISInbal Shani30:47

       (laughs)

    2. LRLenny Rachitsky30:47

       Do you have an opinion on whether large language models are the end-all and be all of where all this goes and just continue to add and compute in data? Or do you think there's another, um, I don't know, something on the horizon that's gonna co- again completely transform, uh, the transformer model of AI?

    3. ISInbal Shani31:06

       I think we have pivoted very hard towards a generative AI, and generalized AI can solve everything. I believe that the future is in going back to scale back a bit to the niche AI. So we will live in a hybrid world where you still have specific AI models that are solving very specific problem, where the generative AI will have its limitation, you know. It's a general-purpose tool, so it can be as good as the datasets, but it also trying to find like some, some statistical recommendation of is it good, is it not good? So it's limited by the training set that it has, and it's trying to solve to be everything for everyone. There are going to be areas where you still need to train models. For example, I, I spend my time in the aerospace industry, and the, in the automotive industry. I don't see self-driving car (laughs) models, uh, that require so much delicate, specific models, tuning, uh, high safety regulation, all of that, being developed on a ChatGPT, you know? I, I might be wrong, (laughs) but I think that eventually we will find ourself in the world of hybrid models and multi-models, where there will be several LLM models coming together because each one of them will have their own benefit, and then there's going to be a leash- a, a niche AI or a more configurable, customized model for specific use cases. So that's my prediction of the future, but, you know,in 10 years from now, (laughs) we'll all be smarter.

    4. LRLenny Rachitsky32:32

       Yeah. The future, hard to predict.

    5. ISInbal Shani32:34

       Yes.

15. 32:35 – 34:37

    How to make space for innovative product ideas

    1. ISInbal Shani32:35

    2. LRLenny Rachitsky32:35

       I wanted to ask, so GitHub kind of came up with this whole Copilot idea. I think it was... We had Ryan on the podcast talking about how there's, like, a researcher-

    3. ISInbal Shani32:45

       Mm-hmm.

    4. LRLenny Rachitsky32:46

       ... just experimenting with, "What can we do with a bunch of data?" And then that essentially turned into, "Wow, this is actually working. Let's quickly scale this and launch it."

    5. ISInbal Shani32:53

       Right.

    6. LRLenny Rachitsky32:53

       Is... I guess, is there anything you learned from that huge success within GitHub and Microsoft to find other big opportunities? Is there something you do about how do you structure product teams or create space for people to experiment with things like this to find the next Copilot?

    7. ISInbal Shani33:09

       Uh, it's a lot about that. It's a lot about creating that bandwidth for the team to innovate and experiment, and carving that time out, but also being okay that not every experiment is going to be successful. So the ship to learn or the fail forward, these are the, the philosophies that I believe in. If you d- don't try, if you don't experiment, you will never innovate. And if you don't fail, you will never learn from your mistakes, you'll never learn from your experience, so you will not progress forward. So really creating that ability to experiment, the same that we have done with Copilot or we've done in GitHub Advanced Security, or even the way GitHub was created, it's all that innovative thinking that the teams have that are thinking about, "What is that next generation use case? What is the future developer will need to be happy, to be more productive, to do their job?" In a world that software development is changing so fast, you have to create that bandwidth for the next generation, the next innovation. And if you didn't have a chance to see the, the two-day keynote in, in Universe, I highly recommend because we, we basically shared our next vision for, for Copilot with Copilot Workspace. We also had Satya Nadella stage and he was very excited like, "Oh my God, GitHub, if you built that, this is amazing." So you see that we keep on putting this vision forward again and again, at least six months to a year before we even have something in the market because we are... We're already thinking, we're already innovating, we're already on the next

16. 34:37 – 36:44

    How GitHub stays on the cutting edge of innovation

    1. ISInbal Shani34:37

       thing.

    2. LRLenny Rachitsky34:38

       Is there something that you do to, uh, allow for that other than just these principles of ship to learn and it's okay to fail? Is there some way you structure the teams or resource teams? Is it like you have a percentage of time to think about other ideas? Or is it just generally, "We're okay failing. Find time to work on other things"?

    3. ISInbal Shani34:56

       It... It's... It's couple things. One, it's spend time with customers and learn from your customers. Because a lot of the innovative ideas are coming basically from conversation with customers, because they're sharing with you your- their frustration. They're sharing with you what they would like to have. They're sharing with you what will be a, a, an amazing invention for them. So that's the first thing, is creating that mechanisms. And we spend a lot of time talking to our customers and also talking to our community. We're fortunate to be the home for all developers and home for all open source. Large open source foundation and small open source foundation are running on top of GitHub. So that's another feedback cycle that we have, uh, to really gather those ideas and those asks to help us shape the future. And then the second thing is really about the teams pitching ideas, like, "I have this idea. I've been doing some research. Here is a paper that describes what is it that I wanna do." And then we figure out, "Okay, when can we fund it? How can we fund it? Can we allocate a specific bandwidth? Can we do a POC? Do we do that in that team? Do we do it as a V-team?" So we keep it flexible, and we also have a research team which are GitHub V Next that, basically, that's their day job (laughs) is to think about the next big innovation. But in GitHub, innovation is coming from everywhere. It's coming from our field team that are talking to customers and are implementing different variations of GitHub for our customers, and they'll come back with some feedback or an idea. It's coming from, uh, the product team. It's coming from the design team, from the engineering te- It comes from everywhere. So if you try to structure innovation...... you're losing that organic spark that is humanity. Yeah. Imagine that you s- someone said, "You have 15 minutes a, a day to be creative." I don't think it's doable. (laughs) Uh, so it's, it's encouraging that thinking more than structuring it.

17. 36:44 – 39:20

    The GitHub Next team

    1. ISInbal Shani36:45

    2. LRLenny Rachitsky36:45

       Can you talk more about that team that you mentioned? The... What is it called? The VNeuxT team?

    3. ISInbal Shani36:48

       The GitHub Next, yes.

    4. LRLenny Rachitsky36:49

       NeuxT team. S- can you talk a bit more about what that team is and how that works?

    5. ISInbal Shani36:53

       They're basically applied scientists, research scientists that are working together, and they're really thinking about three, five years horizon. They're not thinking about the right now. They're thinking about what is the future for software development is going to look like? They have... They're writing papers, they're running experiments, they're doing POCs. They are... Their job is to invent the future. Uh, but again, it's, it's really a synergy because they're talking to the product, to the engineering, so they're getting ideas from there. They're talking to customers or we're sharing feedback. So that's, that's our GitHub Next. It's basically our research team.

    6. LRLenny Rachitsky37:28

       And that's where Copilot emerged out of.

    7. ISInbal Shani37:30

       Right.

    8. LRLenny Rachitsky37:30

       Is that right?

    9. ISInbal Shani37:31

       Right.

    10. LRLenny Rachitsky37:32

        It's so interesting 'cause there's... I think I talked about this with Ryan. There's so many companies with a team like that.

    11. ISInbal Shani37:36

        Yeah.

    12. LRLenny Rachitsky37:36

        Like Facebook had a famous one, New Product Experience team. Google had Google... Forget what it was called. Uh, but none of them have really been successful is my understanding, and this one was. Is there anything you think you're doing differently? Is it... Is it more it's like science-based and research versus just, like, product managers trying a bunch of different apps? I don't know. Is there anything that you think is key to this team being successful?

    13. ISInbal Shani37:59

        I think it... There are two elements for that. One is having the right people with the right mindset on the team, allowing them and giving them the bandwidth and freedom to innovate. And the second thing is, therefore, focusing on making things real. So we're not keeping them far or in disconnection from the product and engineering. There's a lot of synergy. There is a lot of discussion. Because what, what happens in, in teams that are not successful, at least from what I've seen, one, that the team is becoming basically another university. They write papers, but nothing is coming out of that, so nothing find its walls to production. So if you don't introduce that how do we take it to production from day zero when you're thinking about a new idea, then it, it's rarely that it will go through that hum cycle and end itself in production. And the second thing that companies tend to use these teams as too much tactical. And here is that we need something in six months, go invent it right now, and then they are basically creating another engineering or a product team to think about the, the things that are, yeah, now in horizon one. So I think in GitHub, we found the right balance between having the right people, having the right mindset, but also creating that synergy between the future thinking to how we make it real so we have a strong relationship, especially between the product team and, and, uh, uh, GitHub, uh, Next to make sure that these relationship are ongoing and ideas are flowing both ways.

18. 39:20 – 42:17

    Advice for early product managers

    1. ISInbal Shani39:20

    2. LRLenny Rachitsky39:20

       Today, you're Chief Product Officer at GitHub, which is, I think, a dream job for a lot of people, early product managers, senior product managers want to figure out how to get to a place that you're at today. And I'm curious what you found most helped you build the skills that were necessary to be successful in this role. Was it mentors, exec coaches, just doing the job for a long time? Anything else along those lines?

    3. ISInbal Shani39:47

       Combination.

    4. LRLenny Rachitsky39:48

       (laughs)

    5. ISInbal Shani39:48

       Um, it's, it's never one thing, because the, the role of the chief product officer is so broad. You're not just the head of product. A- and that's, that's the mindset that is now evolving in the industry is that a chief product officer is more than just the head of product. They need to have a business thinking. They need to understand our go-to-market strategy. They need to understand the sales play. They need to understand how engineering or building products. It's not just about, "Let me write a set of requirements or think about the vision and the future," it's how all these components in the company are operating together to build that product. And some of that you learn from looking into the leaders that you work for and see how they're thinking. From my first boss in the aerospace industry, I learned how to think system. Not to think just on the problem that I'm solving right now, but how the problem that I am in charge in solving, that filter, that I'm responsible in tuning, is connected to their control system and the ignition system and, and all of that, and how is it eventually being displayed? So that requires me to think from the get-go on the bigger solution, not just my component. So I think that's one. The second thing is really thinking about the people, and change management, and how you take people with you, because a product manager role is a very influential role, and a lot of the things you do is influencing other people. Although everyone thinks that as a product manager, you get to do things. (laughs) You get to write papers, sure, but you need to influence the engineering team to build a product that you want them to build. You need to influence the revenue team to go with a go-to market and sell the product the way you envision it. So there is a lot of influencing happening. You need to understand how to do that, and building that skillset as a product manager from the get-go is very critical. So for me, it was the fact that I was fortunate enough to do different roles in the industry and learn from different people, but also looking up, looking across, and, and looking down into the people that I managed and what are the things I could learn from them? What are some of the skills that they have that are not necessarily in my toolbox and how can I adopt them? What does success look like and what are some of the things that make people unhappy? And you don't necessarily want to have these tools in your toolbox. So it's a lot of what yes and what's no in a combination.

    6. LRLenny Rachitsky42:04

       Yeah. It's exactly like you said, it's all the things (laughs) combined.

    7. ISInbal Shani42:07

       Yes.

    8. LRLenny Rachitsky42:07

       Uh, and I, I really like that last point of just identifying people that are good at something that you wanna get better at-

    9. ISInbal Shani42:12

       Right.

    10. LRLenny Rachitsky42:12

        ... and just learning from them, watching them, maybe pr- asking them questions about how they're operating.

19. 42:17 – 45:34

    Inbal’s “biggest learning” from her career

    1. LRLenny Rachitsky42:17

       I am trying a new segment of this podcast called Failure Corner, and my question to you is what's the biggest career or product failure of your career and what did you learn from that experience?

    2. ISInbal Shani42:30

       Uh, I'll call it the biggest learning because I don't, I don't...... I'm not a believer in failure, I'm a believer in learnings. And every time you fail, it's a big learning, and I prefer to look at that as opportunities. Um, I think my biggest one was likely early on when I started being a leader, uh, and a manager of a team, I'm a very go, go, go person, and I have a lot of energy and when I'm coming to something and I see all the issues that need to be fixed, it's like, "Okay, let's take the toolbox out, let's go fix everything." And that includes driving change. And I've seen that when I started doing that without building that understanding of why we need to do a change or why this is a problem that we need to go and fix it, then you don't take people with you through that journey. So, really analyzing that not everyone appreciate change the way you are. Not everyone has that mindset of the go, go, go, let's go, let's go do it. And really figuring out, taking a step back, and assessing what is happening so you can take the team with you on that journey of, of changing. That was for me the biggest learning a- and likely, yeah, a lot of the not supportive feedback (laughs) I got from the team at that time was r- "You're moving too fast. Slow down. Explain the why. Why are we doing that? How should we move forward?"

    3. LRLenny Rachitsky43:52

       Where did this actually happen? Which company? Which role?

    4. ISInbal Shani43:55

       In TomTom when-

    5. LRLenny Rachitsky43:56

       Mm-hmm.

    6. ISInbal Shani43:56

       ... I was leading the, the location-based services team. And I think for me it was, one, it was the first time I was working in international company, so really figure out that, uh, my character not always fly with every culture that exists, so how am I, how am I adjusting to that? And, and the second thing is that they were very accustomed in working in a specific environment in a specific way. And yes, I was hired to drive change, but one, the team doesn't have to accept it (laughs) because that's your job to, uh, to drive change. And also they're not necessarily on board with the fact that you are here to drive change, so how you take them with you through that journey was, was a big learning. And, and being able to communicate in a general way of communicating to drive clarity, that was a big learning.

    7. LRLenny Rachitsky44:48

       That's actually a recurring theme on Failure Corner, of the failure stories I'm thinking about, is just somebody's starting at a company and trying to change too much too quickly.

    8. ISInbal Shani44:55

       Right. It's, it's very common because we are human beings, we think that if we're coming, we have our experience, we're coming to a new, um, a new space, and you immediately see all the things that are not working or all the things that needs to be fixed. If you're working in the same spa- space for a long time in the same team, you often don't see these gaps anymore because you got used to living with them. So if someone's new, you're coming in, you immediately see all these holes and, and cracks and you're like, "Okay, here's all the things I need to fix." But you have a team that has been there for a while, so how you find that bridge to communicate between the things you see and the things that they think is fine. Like, "Why are you driving this change? It's been working. Don't fix it."

20. 45:34 – 46:19

    Inbal’s closing thoughts

    1. ISInbal Shani45:34

    2. LRLenny Rachitsky45:34

       Inbal, is there anything else you'd like to share before we get to our very exciting lightning round?

    3. ISInbal Shani45:40

       For me, it's an exciting moment in time. I've been celebrating my first year in GitHub and my first GitHub Universe, uh, on stage last week. It was amazing experience for me to meet our customers, our partners, and talk a lot about the importance of developer experience, our AI power platform, and how we're shaping the future. So I'm in a very exciting part of my journey with GitHub and, uh, you don't get here if you don't make some mistakes and you do some learnings and you have some successes and then you take the right turn or and sometimes you take the wrong turn, but you find your way to a place that makes you happy and feel fulfilled as a, as a product leader.

21. 46:19 – 50:03

    Lightning round

    1. ISInbal Shani46:19

    2. LRLenny Rachitsky46:19

       Awesome. We're going to link to that talk in the show notes if people want to watch it. With that, we've reached our very exciting lightning round. Are you ready?

    3. ISInbal Shani46:26

       I'm ready. (laughs)

    4. LRLenny Rachitsky46:28

       What are two or three books that you've recommended most to other people?

    5. ISInbal Shani46:32

       Failing Forward by John Maxwell, The Flywheel: From Good to Great by Jim Collins, and a recent book that I read that is called Dare to Lead Like a Girl by D- Dahlia Feldheim.

    6. LRLenny Rachitsky46:44

       Mm. That's a new mention on the show. Awesome. What is a favorite recent movie or TV show that you really enjoyed?

    7. ISInbal Shani46:51

       I like a lot of fantasy and I like a lot of history-based movies and series. The, there is a recent one that came up on Netflix, All the Lights We Cannot See.

    8. LRLenny Rachitsky47:01

       Hmm.

    9. ISInbal Shani47:01

       Amazing, so well-produced, really tells an interesting part of the history of World War II.

    10. LRLenny Rachitsky47:06

        Have you seen Wheel of Time on Amazon Prime if you like fantasy?

    11. ISInbal Shani47:09

        Yes. Yes.

    12. LRLenny Rachitsky47:09

        Okay.

    13. ISInbal Shani47:10

        I love that. It's ******* .It's a good one. (laughs)

    14. LRLenny Rachitsky47:13

        Yeah, especially season two is like, wow-

    15. ISInbal Shani47:14

        Yes. Yeah.

    16. LRLenny Rachitsky47:15

        ... it's a lot better.

    17. ISInbal Shani47:16

        I tell all.

    18. LRLenny Rachitsky47:16

        Okay, (laughs) next question. Do you have a favorite interview question you like to ask candidates when you're interviewing them?

    19. ISInbal Shani47:22

        I think the first question that I really like to ask is, "What is the most innovative thing you have done and why do you think it's innovative?" And it's really interesting that there are some people that think they need to answer about the biggest invention that they've done and there are some people that are very vulnerable and they talk about something very personal that they have innovated and it talks a lot, it, it, it shows a lot of their character. And I think the second question that I like to ask a lot is, "Give me an example and a time where you had a disagreement with your manager or you didn't align with your manager, and then what did you do about it and how did it go?" Because it showcase a lot about your character and are you willing to stand your ground and push up when you need to, and then what is that communication skills that you have, your influential skill that you have?

    20. LRLenny Rachitsky48:09

        Do you have a favorite life motto that you like to share with people, often come back to either in work or in life that you find useful?

    21. ISInbal Shani48:15

        I think it's, "If you don't take risks, you cannot create a future." I forgot who said it. I think it's from one of the animation monkey, Luffy if I remember. But it basically summarized, uh, the life motto that you have to take risks, you have to experiment. You, if you feel comfortable, (laughs) it's not a good thing. You always need to feel uncomfortable to stretch yourself to go to the next thing, so that's something that I've done. You, you just heard about my career story from applied scientist to chief product officer of GitHub. Like, who, who knew?

    22. LRLenny Rachitsky48:47

        I love that motto. Inbal, thank you so much for being here. Two final questions. Where can folks find you online if they want to reach out and follow up on anything, and then how can listeners be useful to you?

    23. ISInbal Shani48:57

        Lots of interesting sessions and talks from Universe, so if people want to learn more about what is it that we're doing, how we're thinking about AIs there, LinkedIn is likely the best, uh, social media platform to find me, uh, where I have a chance to talk to a lot of people. These are the best places.

    24. LRLenny Rachitsky49:12

        And then how can listeners be useful to you?

    25. ISInbal Shani49:14

        If you can share a similar experience or a story that you think will be helpful for me as I'm thinking about what does that mean being chief product officer for GitHub, any tips and tricks on how you use GitHub just so maybe I can learn or share your experience and story so I can learn from your experience as well.

    26. LRLenny Rachitsky49:31

        Amazing. Inbal, thank you so much for being here.

    27. ISInbal Shani49:35

        Thank you so much for having me.

    28. LRLenny Rachitsky49:36

        Bye, everyone. (instrumental music) Thank you so much for listening. If you found this valuable, you can subscribe to the show on Apple Podcasts, Spotify, or your favorite podcast app. Also, please consider giving us a rating or leaving a review, as that really helps other listeners find the podcast. You can find all past episodes or learn more about the show at lennyspodcast.com. See you in the next episode.

Episode duration: 50:03