No Priors Ep. 10 | With Copilot's Chief Architect and founder of Minion.AI Alex Graveley

Sarah Guo 0:06 So let's start with some background to how did you end up working in tech for AI?

Alex Graveley 0:10 Tech was earlier. I started, you know, really young and, uh, I got really into Linux when I was 14 or so. And, um, yeah, it was right around the time when the web was, like, a new thing, and you had to work to kind of get on the web. And then Linux is... I just sort of found it. It, um, popped into my life, and I really liked the... I read the GPL really early at 15, and it struck me. And, uh, just the idea of helping in the open and making it freely available, so other people could learn from things like I was learning from things seemed great. So I went and spent many years just working on open source stuff. And I don't know, I guess I didn't realize there was like... There was kind of a... There was, like, legacy technology, right? Like Windows was popular at that time. So people ran their, you know, data centers on Windows and networks on Windows and... Um, this Linux thing was, like, very strange and people didn't really know what it was. I spent many hours compiling kernels and, uh, hacking on stuff and, uh... Yeah.

Sarah Guo 1:14 And what was the thought process behind starting, like, Hackpad?

Alex Graveley 1:19 Oh, Hackpad. Uh, yeah. So I had just finished like four or five years at VMware, and, uh, I wanted to get into startups. I knew- I knew that. And then, uh... So I left VMware and I started working on an, an education startup like many of us do. Uh, I don't know if you know this, but like many, many founders start with like the idea of an education startup.

Elad Gil 1:41 It's like a rite of passage. Yeah.

Alex Graveley 1:43 It's like a rite of passage. Yeah. So I spent, I don't know, nine months working on- on- on- on that.

Sarah Guo 1:48 Most of us have done either that or consumer social

Sarah Guo 1:50 at the time.

Alex Graveley 1:50 Sure. Yeah. And, uh, it's... Uh, it turns out education's very hard. Um, yeah. You nod your heads knowingly. Yeah. And, uh, so after nine months I was like, "All right, this isn't going anywhere. Uh, I- I know... I don't know if there's a value prop here." I mean, that's... That was the value, is that I learned that, like, you have to make something that is both achievable and that people want to pay for or spend their time on. So, yeah, then I was just kind of fishing around. Um, I was living in like a- a warehouse in San Francisco with a bunch of Burning Man people, and, uh, we were having trouble organizing burning... Large-scale Burning Man projects. And so,

Alex Graveley 2:28 uh, I forked Etherpad and started hacking on it. Uh, recruited a friend to... In the community to start working on it with me. And yeah, just grew from there. We had... Um, you know, before we did YC we had most... Many of the large Burning Man camps using it to organize their- their builds.

Sarah Guo 2:46 Can you describe the product experience?

Alex Graveley 2:48 Oh, yeah. It was a... You know, it was a- a real-time text editor, uh, kind of like Google Docs. Um, Google Docs is the only other one that did that at that time. And, uh, it- it was kind of nice because, uh, it would highlight who said what, so you could go track down if somebody had a contribution and be like, "Oh, what did you mean?" Or, uh, "I heard you say something about this." So that's very useful in, like, um, large-scale anonymous, pseudo-anonymous groups where you don't really know who has ownership over what, like Burning Man camps. And let's see. And then we did YC, and then we got, uh, a bunch... Almost... Many of those people did it. It... My, like... My hack was to go do YC and try and get all the companies doing YC to use my product, which many of them did. I also took, like, very extensive notes of all the YC presentations using the product that everyone would then, like, look at. And, uh, we were able to get Stripe. And Stripe used us for many years actually. Um, they built for- for their... I think we were their first knowledge base. They used it for a long time. Um, a bunch of other big companies as well. Um, yeah. We were very lucky. Yeah.

Sarah Guo 3:54 And, um, post-Dropbox acquisition, you worked on what became Paper. How did you think about, like, what you wanted to go work on next?

Alex Graveley 4:04 After Paper? I kind of did... You know, I spent the next few years kind of poking around at stuff. Uh, I knew that I wanted to make a robot that does stuff for you. Yeah. So there was this company called Magic. So I worked at this company called Magic. They were doing this, like, uh, text-based personal assistance.

Sarah Guo 4:22 Mm-hmm.

Alex Graveley 4:23 Uh...

Sarah Guo 4:23 You remember this one?

Elad Gil 4:24 I- I- I think, uh, just like everybody who starts an education startup, Magic is one of those names that's cycling, and there's a really cool AI company right now called Magic as well.

Sarah Guo 4:31 Mm-hmm.

Elad Gil 4:32 So I feel like there's also these names that kind of persist from generation to generation, which I think is really cool.

Sarah Guo 4:36 I'm sure you know it's Codegen.

Alex Graveley 4:38 Yeah, yeah, yeah. No, yeah. Uh, I haven't seen a demo yet, but it sounds like they're doing the right thing. Um, there's a few people doing the kind of whole repository, uh, uh, changes. So it seems like a great direction.

Sarah Guo 4:51 Uh, what- what's like one learning from... Um, you know, Magic was, like, a- a company early in trying to do seamless

Narrator 4:59 Yeah. Yes.

Sarah Guo 5:00 ... sequence models and machine learning in general?

Alex Graveley 5:01 Uh, no, no. It wasn't trying to do machine learning. It was all op space. It was super heavy op space. So, uh, you know, they had teams of people and it was 24 hours, and they would cycle in and- and, you know, they'd lose context and, like, they're all busy because they're trying to... They're trying to, like, deal with lots of people with lots of requests all the time. And so really it was like a crash course in, like, human behavior, right? Like what do people do under stress? How do they act? What do they say? What can you train? What can't you train? Like, um, uh, can you bucket stuff? And the answer is no. Like humans are complicated, especially in text form. So the beauty of the web and, like, traditional UIs, right, is that you fill them in. And like if it... If you can't... If it doesn't do what you want, then you make the decision to either go forward or not go forward, right? Text has this, uh, annoying property of you can be all the way at the 99% mark and then change the goal entirely, right? And so... Yeah. So it's- it's complicated. It's hard to... It's hard to understand what, uh...... what people want. It's hard to understand the complexity involved, especially when you're dealing with the real world. Um, flights get delayed. Passwords get lost.

Elad Gil 6:09 Do you think we're the last generation to deal with that? In other words, it feels like we're about to hit a transition point in which agents can actually start doing some of these things for us for real. When before, I think all these products really started with these operations-heavy approaches. I remember, you know, there was a really early sort of, um, personalized search engine called Aardvark, where similarly if you kind of looked behind the hood, it was a lot of ops people, and there was a little bit of algorithm, right? And so-

Alex Graveley 6:32 That was a... Yeah. That was really with like, um... You would, like, sort of describe what you're good at or something, and then they would try and send questions to you, and you'd answer.

Elad Gil 6:39 Yeah. Yeah. Exactly. They'd kind of route things, but I think there was actually people doing some of the routing at least, or... I can't exactly remember, right? But I think it was... You know, I think a lot of people wanted to build these really complex sort of bots or agents that were doing really rich things, and the technology just wasn't there it feels like for a period of time. There was also some startups. I remember one company, um, that I got involved with that was, um, trying to do, like, virtual scheduling and assistance, you know?

Alex Graveley 7:01 Mm-hmm.

Elad Gil 7:02 Six, seven, eight years ago. And again-

Alex Graveley 7:03 Yeah.

Elad Gil 7:03 ... it felt like it was a little bit early maybe.

Alex Graveley 7:05 What were they called? Was that X dot... No, that wasn't X dot.

Elad Gil 7:08 Yeah. I don't know. There was, there was a couple of them, um-

Alex Graveley 7:10 Clara Labs was one. Clara. That's the one. Yeah. Yeah, yeah, yeah. Yeah. I remember this era. And then we'd... Like, we had Operator. Do you remember Operator? I remember Operator. Yeah, yeah.

Elad Gil 7:17 There was like three or four of them.

Alex Graveley 7:18 And then, like, on the question-answering stuff, we had Jelly. So there was a whole series of this

Elad Gil 7:22 Oh, right.

Alex Graveley 7:22 Yeah, yeah. Yeah. Yeah. It's, it's like, um... So, yeah. What we were able to do with Magic... As a, like an aside there, you know, I r- I started working there 'cause I wanted to work

Alex Graveley 7:32 on, I wanted to work on the AI part. Uh, I think somewhere in there, M, Facebook M started as well. Mm-hmm. Uh, and it was just... It was a fun place to try and learn everything I could about solving those problems. Before Transformers, there was, uh, Sequence to Sequence was kind of the previous, uh, iteration. And, um, we were able to, like, take all the, all the histories of the chats between assistants and, and people and train a little model on it and, uh... "A little" by today's metrics. Uh, um, and run it, and it would like... It would show some gray text in the little text bar, and people could edit it and hit Enter and- Like the operators, like... Yeah, yeah. Right. The, the, the operators. And, um, yeah. We would measure how much time they sense, uh, they spent, uh, typing with it and with it out of it, and so it would save, like, half an hour across 100 people per day, uh, across an eight-hour shift. Uh, so yeah. That was, like, my first shipping AI product, I guess.

Elad Gil 8:30 That's-

Alex Graveley 8:30 And then how'd you end up going to, um, Microsoft? Like, that's Oh, there was a bunch of other... Yeah. There was a bunch of other stuff along the way. There was... Um, so after that I did, uh, uh, I got into crypto. Uh, my friend was doing hCaptcha, which, uh, was like sort of a captcha marketplace, which is now, like, something like the number one or number two captcha service in the world, which is crazy. Yeah. So kind of launched that. That was fun. Uh, you know, annoyed people the world over for many-

Elad Gil 9:01 (laughs)

Alex Graveley 9:02 ... many man-hours in aggregate. And then worked on... Uh, left that to work with, uh, Moxie on, on a cryptocurrency for Signal. Um, so that was really fun. Um, complicated. And it all worked in a few seconds. Um, so, you know, we were sh- shooting for Venmo quality, which I think we pretty much hit. So.

Elad Gil 9:25 When you think about crypto in the context of AI... Because people talk about it in a few different contexts, right? One is, uh, you have programmatic, um, sort of money as code running. And so that could create all sorts of really interesting things from an agent-driven perspective. But then the other piece of it is identity. And some people think... I mean, Worldcoin would be one example, but there's other examples of effectively trying to secure identity cryptographically on the blockchain in an open way and then using that identity in the future to differentiate between AI-driven agents and people. Do you think that's gonna be important, or does that stuff not really matter in terms of the identity portion of not only crypto, but just, like, how we think about the future of agents?

Alex Graveley 10:03 The honest answer is, I think we're gonna go through a many-year period of extreme discomfort, where AIs pretend to be things, or-

Elad Gil 10:13 Right.

Alex Graveley 10:13 ... uh, or confuse people, or extract money, money from your grandparents, or, um, you know, drain people's life savings in ways that are, uh, that are scary. And, you know, OpenAI is trying to do their best, but for some reason, the focus has been on OpenAI doing everything. And instead of, like, "We should go build the systems that prevent that. We should go, uh, pass the legislation that, that drops the hammer on people doing that stuff. We should go..." All this kind of stuff that is, um... Unfortunately, it seems like we're gonna need some really bad things to happen before, um, we align correctly.

Elad Gil 10:49 Mm-hmm.

Alex Graveley 10:50 I don't... I'm not really scared about, uh, AIs killing us, although I'm very grateful that there are people that are thinking about it. Um, I'm more worried about bad people using new technology to hurt us.

Elad Gil 11:02 Yeah. Ilya from NEAR has some really interesting thoughts on this, 'cause he was one of the main author... Or he was the last author on the Transformer paper before he started NEAR.

Alex Graveley 11:09 Mm-hmm.

Elad Gil 11:09 And he's brought up these concepts of like, how do you stress-test society relative to the coming wave of AI, which I think is an interesting concept.

Alex Graveley 11:15 Yeah. It's a great, it's a great way to look at it. Like, yeah. It, it's not as bad as it could be, right? If you think about it, there, um... Most of the things that you want to spam, uh, are, uh, e- either have a spam blocker or are somewhat difficult to create an account on.

Elad Gil 11:32 Mm-hmm.

Alex Graveley 11:33 So, um, you know, doing a better job of, uh, of, uh, sock puppet account filtering is gonna be really important going forward. Um, you know, I like what Cloudflare is doing with their kind of fingerprinting, um, instead of visual captchas, which are not good enough anymore. You know, one thing that is, like, kind of a saving grace, grace here is, is that, uh, you know,

Alex Graveley 11:56 many of the things that, uh, you would want to do cost money.

Elad Gil 12:00 Mm-hmm.

Alex Graveley 12:01 So, calling everybody costs money.

Elad Gil 12:03 Sure.

Alex Graveley 12:03 Um-... uh, texting everyone should be hopefully illegal soon, but also costs money. Maybe not enough money to, to prevent these things, but-

Sarah Guo 12:13 Probably not enough as agents can make money, right? I mean, just look at the trade-offs of cost.

Alex Graveley 12:18 Yeah. I think it's interesting. Yeah, for sure. You know, the, the other thing, the other nice thing there is that, like, grandmas don't have crypto, right? They have bank accounts, and bank accounts can be traced. Um, I, I guess I would say it's not, you know, it's somewhere in the middle. It's like, uh, you know, it's imagine there's an, you know... North Korea has been trying to do this to us for a long time, right? Now there's a North Korea that has more resources or is more distributed or whatever. It's, you know, we have some mitigations. We need more. We need to be thinking about it a lot more.

Sarah Guo 12:50 Mm-hmm. How did you end up at GitHub, and how'd you end up working on Copilot?

Alex Graveley 12:54 Oh, yeah. Okay, uh, let's see. So while I was working on MobileCoin, I, uh, my dad's kidneys failed, and I tried to donate them and, um, donate, donate a kidney. And, uh, they found a lump in my chest as part of the, the scans they do. And, uh, I had to have my right, most of my right lung removed in 2018. And so that was a big deal. Um, and so took some time, um, yeah. Healing, it's weird. Healing from internal injuries takes a lot longer than you, than you think. Anyway, happy story is that, uh, I, I don't have cancer now for, uh, over four years. And, um, my dad's kidneys got... He got a transplant also, so things are good. And so after, uh, oof, I don't know. I guess I was recovering for, for quite a while, and then I went and, uh, begged my friend for a job. I figured I should start working again, so I worked on some random stuff at first. Uh, uh, I converted GitHub to using their own product to build GitHub, which was kind of fun. They weren't... Yeah, it's an... I think people still use Codespaces now to build GitHub, which is pretty cool. Um, yeah. Then this kind of opportunity to work with OpenAI came up, and, uh, yeah. Because I had been tracking AI in the past, um, and was pretty aware of what was going on, I jumped on it.

Sarah Guo 14:29 Was that proposed by OpenAI or by GitHub, or who kind of initiated it all?

Alex Graveley 14:33 So, I don't know the exact beginnings. I know that, uh, OpenAI and Microsoft were working on a deal for supercomputers. Um, so they wanted to build a big cluster for training, and there was a big deal that was being worked out. And there was some software kind of provisions thrown in. I think Office and Bing probably. And GitHub was like, "Oh, well, okay, well maybe we can... Let's like, uh, let's, um... May- maybe there's something GitHub can do here." Uh, I think OpenAI threw a small, threw a small fine tune over and was like, "Here's a, here's a small model trained on, um, on some code. See if this is interesting," you know? So we played around with it and-

Sarah Guo 15:09 What, what's small in that sense?

Alex Graveley 15:11 I don't know. This was, uh... I have to remember now. I think this is before I knew very much, so it was definitely not a DaVinci-sized model. That's for sure. Um, I don't know if it... I don't know what size it was. Probably... Yeah, I don't know what size it was. Um, probably less than 10, but I can't remember. I, I learned later that this was basically a training artifact. So they had wanted to see what introducing code into their, uh, into their base models would do. Um, I think it had positive effects on chain of thought reasoning. Um, code is kind of linear, so, um, you can imagine that, you know, you kind of do stuff one after another and the things before, uh, have an impact.

Sarah Guo 15:49 Mm-hmm.

Alex Graveley 15:50 And yeah, it was not that good. It was very bad. Uh, you know, they, it was, I think just like I said, just an artifact and just a small sample of, uh, GitHub data that they had crawled. And, um, yeah, we played around with it. I, this is before actually I, I joined. I was sort of, uh... Me and, uh, this guy Albert, uh, Ziegler were the first two after, uh, Uge, who, um, Uge got ahold of this model and started playing with it. He was able to say like, "Oh, well, you know, like, it doesn't work most of the time, but here it is doing something, you know. Here's, here's..." And it was only Python at that time. "Here it is, you know, generating something useful." We didn't really understand anything, you know. This just sort of lobbed over at us. So that was enough to like, "Okay, well, you know, go fetch a couple of people and start working. See what, see, see if there's anything there." We didn't really know what we had, so the first, you know, task was to go test it out, see what it did. Um, so we cr- we crowdsourced a bunch of, uh, Python problems in-house, stuff that we knew wouldn't be in the training set, and then we, we started work on, um, ph- fetching repositories and finding the tests in them so that we could, um, basically generate functions that were being tested and see if the tests still pass. There had been like a brand new, uh, PyTest feature introduced like recently that, that allowed you to see which functions were called by the PyTest. So you'll find that function, zero its body, ask the model to generate it, and then rerun the test and see if it passed. And, uh, I think it was less, I don't know, 10%, something like that, of, of those guys. And the dimensions are kind of like how many chances do you give it, um... How many chances do you give it to, to, to solve something? And, um, and then how do you test whether it's worked or not, right? So for the, the standalone tests that was w- we had people write test functions and then we would try to generate the body, and, uh, if the tests pass then you know it works. And for the in-the-wild, in-the-wild test harness we would download a repository, run all the tests, look at the ones that passed, find the functions that they called, make sure that they weren't trivial, generate the bodies for them, rerun the test, see what happens, see what... And then you get your percentage. And, uh, yeah. I mean, it was s- something like some very, very low percentage up front, but we knew that there was kind of a lot more juice to squeeze-So, like, getting, um, all of GitHub's code into the model, um, and then a bunch of other tricks that we hadn't, you know, we hadn't even thought of at that time. Eventually, you know, it, it went from, you know, less than 10% on, in the wild test to over 60%. So that's like one in two tests it can just generate code for, which is insane, right? Somewhere along the way, you know, there was like 10% to 20% to 35% to 45%. You know, these kind of like improvements along the way. Somewhere along the way, we did more prompting work so that the prompts got better. Um, somewhere along the way they used, you know, all the versions of the code as opposed to just the most recent version. They used diffs so that it could understand small changes, like, yeah, just it got better. And so but at the, when we first started, we were just, we didn't know what we had. We were just trying to figure it out. Um, and, uh, at the time they, they were thinking in terms of like, "Maybe you can replace stack overflow or something, you know, do a stack overflow competitor." Um...

Elad Gil 19:15 Was that the first, like, product idea-

Alex Graveley 19:18 I think-

Elad Gil 19:18 ... you guys had for it?

Alex Graveley 19:19 I, I, I don't know that we had that idea. I think that was kind of like a...

Elad Gil 19:22 Yeah, on high.

Alex Graveley 19:23 Yeah, yeah, yeah. There was more like a, um, it'd be nice, you know, it'd be nice if you could make something that competed with Stack Overflow because we have all this code, wouldn't it be nice to leverage it. Um, yeah. And so we made some UIs, but like early on, you know, it was like (laughs) y- early on it was bad. So it'd be like, you'd watch it and it'd, it would run and most of them would be bad and be like, there'd be like one success and you're like, "Oh, sweet, I got a success." But I had to wait, you know, some number of seconds for, for this test.

Elad Gil 19:51 Was the test user group just like the six of you, like some larger group?

Alex Graveley 19:56 We made a little... yeah, the first iteration was just like an internal tool that people would, that helped people write these tests.

Elad Gil 20:01 Mm-hmm.

Alex Graveley 20:01 And then, um, we wanted to see if maybe we could turn that into some UI that people would use where (laughs) if there was some way to cover up the fact that one in 10 things pass, right? So we tried a bu- we tried a few UI things there, and then, uh, it was actually OpenAI was like, "It would be nice if we could... we're testing these model fine tunes. It'd be nice if we could like test them more quickly." Can... what about doing like a VS Code extension? Um, just do autocomplete. So we did autocomplete at first, and that was kind of, that was a big jump, you know, because they, they were still thinking in terms of like stack overflow, you know, but this is, it's like, you know, I didn't have any ideas basically. I didn't know how to beat Stack Overflow with this thing. But we could come up with, we could play with some stuff in, in VS Code that was maybe closer to the code, you know? Um, and so we tried, uh, first we did autocomplete and that was kind of fun. It was useful, you know, um, it would, it would, it would show this little pop-up box like autocomplete does, and you could pick some strings. And so that format actually, you know, the, the usage was, you know, it was fine. Uh, it wasn't the right metaphor exactly, right? But you've got like this code generated mixed in with the, um, specific terms that are in the code, and it's a little... it's not exactly the same thing. Um, we tried things like, uh, uh, adding a little button over top of empty functions. So you would go generate them, or you could like hit a control key and it would create a big list on the side that you can choose from, or there's a little pop-up thing. So basically tried every single UI we could think of in VS Code, uh...

Elad Gil 21:37 And multiple generations, like the, the list, that didn't work.

Alex Graveley 21:40 Yeah, none of, none of, none of them like really worked, like lists were like, you know, maybe you'd get one generation per person per, per day. And this is just a small sample, it's just like a few people that were interested in, in, at GitHub, um, language nerds or, um, people that have written tests for us, uh, and OpenAI people. So, so then, you know, I had some... very early on, I had this idea that it should work like a Gmail, uh, free text autocomplete, which was... I was like enamored with that product. It's like, it was the first largest language, um, you know, "large language", um, model deployment in the wild. Um, it was fast, it was cool, like the paper's great, like they give you all sorts of details on how they do it. All the, all the workarounds they had to do. So that was always in the back of my head, you know? And it, it was bad also, it was like, you know, those completions are not good. Um, but it seemed like the right, the right thing. Um, anyway, somewhere along the way after I, after we've tried all the UI, you know, sort of come up with some idea for... VS Code didn't support this so I tried to hack it in and finally came up with a way to hack it in and, uh, enough to make a demo, um, like a little demo video. And, uh-

Elad Gil 22:58 Was there support to like build real support for it within the organization?

Alex Graveley 23:03 It's a little complicated. I, I guess, you know, we were pretty much a skunkworks project, so no one really knew, you know, no one knew about us. So we would go to like the... if you go to VS Code, people would be like, "Hey, we need you to go implement this very complex feature." Like, "I don't even know who you are. Like, what are you talking about?" Um, and, uh, yeah, there was, there was definitely some politicking that happened to, to get the VS Code people to, um, to dedicate some resources to that on a short, short time frame, like we were moving really fast. You know, it was less than a year before from beginning to ship public, public, public launch. Um...

Elad Gil 23:38 Was there a certain, um, metric where you're like, "This is good enough, like we need to actually put it in the public product?"

Alex Graveley 23:46 Yeah, there was... I mean, we had a long... we had a nice long window of public access before GA, so where it was free and you could use it and, um, and we did a bunch of optimizing for different groups of people that would be, you know, okay, well, you know, do we want more experienced people? Do we want more, uh, new people? Um, I want people from this area or this area and, uh, um, that gave us a bunch of really good stats. Uh, so we were able to learn that, for instance, like, uh, speed is the only thing that matters. Um, so-... uh, yeah, there's some crazy thing. Like, every 10 milliseconds is one percent fewer completions that people would do. That adds up. 10 milliseconds is pretty fast. Uh, we learned that because, uh, s- somewhere in our first few months of public release, we noticed that Indian completions were really low. Like, the- for whatever reason, they were just significantly lower than, than Europe.

Sarah Guo 24:42 So network latency to India?

Alex Graveley 24:45 Yeah.

Sarah Guo 24:45 (laughs) Wow.

Alex Graveley 24:45 And it turns out 'cause-

Sarah Guo 24:46 Yeah.

Alex Graveley 24:46 ... OpenAI only had one data center, so it was all in Texas, and so it was going... If you can imagine, you're like typing, and so that goes from India, through Europe, over the water, down to Texas, and back, and back, and back. And now if you've typed something that doesn't match the thing that you requested, then that's useless, right? So you don't get a completion.

Elad Gil 25:04 But by the way, that, uh, I know it's obvious, but that's happened on every single product I've ever worked on. You know, like when I was at Google, like, um, I worked on a variety of mobile products, same thing, you know, page load times. Obviously search in general, 100 millisecond difference, like, uh-

Alex Graveley 25:17 It's a big deal.

Elad Gil 25:18 ... makes a big shift in market share. So yeah, it's kind of nuts how much speed matters.

Alex Graveley 25:21 Yeah, and so we, that, once we figured that out, then we were able to... We knew that, maybe this is too far ahead, but we knew that we had something awesome, right? Like, people that were close to Texas were like, "This is freaking great." Like-

Sarah Guo 25:33 (laughs)

Alex Graveley 25:33 ... they were like, you know, we had a s- a Slack channel and people were posting in it all the time. And, um, yeah. There, and the fun, the most fun stuff was like, these people that would pop up and be like, "I don't program, but I just like learned how to write this 100 line of, 100 line script that like does this thing that I need." It's like, oh my god.

Elad Gil 25:50 (laughs)

Sarah Guo 25:51 I definitely feel like I now speak different languages that I don't actually know the syntax of-

Alex Graveley 25:55 Yeah.

Sarah Guo 25:55 ... which is ex- very exciting.

Alex Graveley 25:56 Yeah, it turns out these models are really great at, uh, accop- like, finding patterns. So like once, after we figured out that, you know, once we knew that... So the, once we had the, the UI mechanism that worked, um, which was this like gray text, so you hit tab, it inserts. And once we knew... And then also like a bunch of work to get the multiline working 'cause we knew that... The problem with the auto-complete was that there's only one line. You get this one line to like tell people what, what you wanna... But we could, we knew the model could do more than that. It could do, you know, a few, like a, 10 lines realistically. Now it can do way more but... And so we needed something that would show that off and, uh, and so yeah. We tried this ghost text thing and it would just... The key insight was realizing that y- you, programmers kind of think in blocks, so if you can just auto-complete the re- the rest of the current block, then that's what people want. People can, eh, they're kind of like judging what this block is gonna do, and if it does the right thing, they know quickly. Um, so yeah. So if you can complete an if statement and it does the right thing, then it's easy to judge. So we knew, we knew we wanted something like that and then it was just this like, then it was just this, uh, squeezing as much performance as we could get out. Like, we basically never found the bottom of like,

Alex Graveley 27:11 you know, we, we hit, we made it as fast as we could and it was still improving on, um, you know, completions. But yeah, so that was like... And then that perpetuated like, "Okay. Okay. Uh, I know there's this plan for like Azure to run OpenAI in six months. We need you to do that in the next month. So like can we, let's figure out how to make this happen." So for, so 'cause we wanted to run, uh, a bunch of GPUs in Europe, so, so we could, um, hit Asia. You know, there was no, there was no other place that we could, we could run them in Europe, we could run them on the West Coast, we could run them in Texas, um, at the time. So yeah, w- so that was... And then, and Microsoft stepped up there, we got it running. Um, yeah. And then pretty much after that we launched. And yeah.

Elad Gil 27:58 Were you surprised by the uptake post-launch?

Alex Graveley 28:02 No. Yeah. No, I mean, our retention rate was 50%. Like, it never went, like, months later it was still above 50% by like weekly cohort, which is like-

Elad Gil 28:13 Yeah.

Alex Graveley 28:13 ... insane.

Elad Gil 28:14 Yeah.

Alex Graveley 28:15 Right? Um, and we didn't, we didn't know if people would pay for it. That was one thing. Um, I lobbied pretty hard for, uh, for going cheap and capturing the market.

Sarah Guo 28:27 How did you guys think about inference cost for this thing at the beginning?

Alex Graveley 28:31 Oh yeah, we were... Our estimates were wildly off. Wildly off. Yeah, so we got estimates that were like, you know, it'll be 30 bucks a month for ev- for your a- on average, right? And then, um, we were able to like... Once we, once we were able to, once Microsoft was able to like do some, y- kind of do their Azure infrastructure, we were able to then like fork off little bits so we could do more accurate, uh, projections. And, uh, yeah, there was a bunch of like, there was a bunch of like moments where like, "How much is it gonna cost?" You'd be like, "Wait for these results." And, uh, the first big one was 10 bucks a month. It'll cost 10 bucks a month. And I was like, "Oh my god, so much cheaper than we expected." Um, and then, and then we could optimize it but now it's like we hadn't even optimized on price yet, right? And then we optimized on price a bunch and yeah, now it's less than that. So like, it was very fortuitous, right? Like, we were thinking like, "Okay, well maybe it's, maybe it's enterprise only because that's the only people who are gonna be willing to pay for this." Like, um, you know, 30 bucks a month is not... That's a, it's a lot. And that's like with no margins, right? But yeah, so-

Sarah Guo 29:42 For 40% of your code it's not a lot.

Alex Graveley 29:44 Yeah, that's the thing. We know that, we know that now. And not only that, it's like, uh, the crazy thing is there's, uh, um, there's some whales out there. Like, there's people who it writes 80% of their code, um, which was insane. Uh-

Elad Gil 29:58 Where do you extrapolate all this like three to five years out? Are there basically gonna be just like agents writing code for us in certain ways? Is it 95% of code is written by copilots and, you know, humans are kind of directing it? Like, what, what do you kind of view the, the world evolving into in the next couple years? And next couple I mean like three to five, not 20.

Alex Graveley 30:21 Yeah, it's hard. Uh, I don't know. (laughs)

Elad Gil 30:23 (laughs)

Alex Graveley 30:24 I think it's hard for me to, you know, it- it- i- it's hard for me to imagine what that world

Alex Graveley 30:30 looks like, 'cause it's such a shift from, like, I have my hands on something and I know that it's right to, um, where we are now, which is I mostly know what's right or I have a sense that it's right, but I have to, I have to test it and, and see it run, um, to know that it's right to then just write this and I'll, I'll trust that it's gonna work. Like, those are pretty crazy transitions, right? Um, they, they exist. Like...

Elad Gil 30:54 Sure. But you could also imagine, like, certain ways to do, like, a code review post, some chunk or some- some other sort of quality check to...

Alex Graveley 31:00 Yeah. I think every... If that's, if that's the goal we wanna get to as a people, like, uh, I think every barrier to that is achievable.

Elad Gil 31:09 Mm-hmm.

Alex Graveley 31:09 Um, so we can code review only the s- the dangerous parts-

Elad Gil 31:13 Yeah.

Alex Graveley 31:13 ... or only the scary or the confusing parts, or...

Elad Gil 31:15 Mm-hmm.

Alex Graveley 31:16 Um, uh, we can do things like train a model on functions before and after changes-

Elad Gil 31:22 Yeah.

Alex Graveley 31:22 ... to say like, "Okay, this looks like your... Looks like a more polished version of this function would be this." Or...

Elad Gil 31:28 Mm-hmm.

Alex Graveley 31:29 Um, yeah, we can do things like, you know, just start at the very basic, you know, very basic main loop and then add everything-

Elad Gil 31:36 Mm-hmm.

Alex Graveley 31:36 ... piece by piece, um, with tests so that it's, you know, what, what works and what doesn't, and then just have the, uh, the AI keep generating what's the logical...

Elad Gil 31:45 Yeah.

Alex Graveley 31:45 ... next feature. Like, um, all these things will get figured out. So if-

Elad Gil 31:49 Yeah.

Alex Graveley 31:49 ... if that's what we wanna do, that's what's gonna happen.

Elad Gil 31:52 So what's the, what's the idea behind Minion?

Alex Graveley 31:56 Yeah. I think I mentioned, uh, making bots that do stuff for you. It's a, um, broad topic. And, um, I think that's where we see AI going. You know, the next few years are, are, um, in the AIs taking action, not just, uh, answering questions or, um, writing copy, but actually, um, helping us in our daily lives. Um, things like, um, organizing my schedule, or booking flights, or finding a trip for me to take, or, uh, doing my taxes, or, um, uh, telling me which contacts I haven't talked to in a long time and should reach out to, you know? Um, there's a lot of stuff that we can do by giving, uh, AIs access to information and letting them act on that information in a controlled way that, um, checks to make sure that we're, that we're aligned. And yeah, I think that'll be a really fun future. I think, you know, almost like you can imagine Copilot applied to everyday activities, right? Like, Copilot gives you a little bit of help. So I want Minion to give you a little bit of help, uh, outside of your code editor.

Elad Gil 33:17 How did you decide to, to work on Minion specifically?

Alex Graveley 33:21 Minion specifically. So basically I s- I stopped working... At Magic I quit 'cause I, I couldn't figure out how to hook up the AI to data. I was like, "If in order to... In, in order to improve the quality I need a PhD in math, like, I don't know what to do now." And it just sort of... The models got better enough where that specific problem seemed solvable. Yeah, so the, the tech got better. And so that specific problem is where interacting with the real world broke down in the past. You know, like I said, flights get delayed, prices change. Um, uh, not just like a little bit, all the time, you know. Um, you know, they might not have the seat you want at the, at the concert that you wanna go to. Uh, and so, you know, AIs are this kind of compression of everything that you... In their training set, um, but they're not a real-time mechanism yet. Anyway, so that, that was the idea. I was like, "Okay, well, uh, I think we can, I think we can work on this old problem of how to make a, a bot do stuff for people." Like, yeah. That's what, that's what we want. Let's go make it. I don't know. It's like, maybe I can use the excitement from Copilot to launch into something which is incredibly hard, um, and, but which I believe the technology is around for.

Elad Gil 34:42 Mm-hmm.

Alex Graveley 34:43 Um, if we can figure it out. So yeah, that's it. That- I-

Elad Gil 34:46 Yeah.

Alex Graveley 34:46 That's the only reason I've ever gotten into startups actually, is like, uh, "Okay, I wanna do a startup so that I can do a harder startup."

Elad Gil 34:52 Yeah.

Alex Graveley 34:52 You know? Or, "I wanna do a project so I can do a harder project." Um...

Elad Gil 34:56 Is this one sufficiently hard?

Alex Graveley 34:57 This one's hard. Yeah. It's getting hard. There are these fun things that you kind of learn along the way that keep you, that keep you, like, engaged, right? Um, like the thing with code, with Copilot, is like, it turns out code is pretty special, right? Like, you can run it. So if an AI generates some code and it runs and you know something about that code that you wouldn't know necessarily with text. Um, and, and accomplish... Like, doing agent-like work on the internet, uh, or on apps is, uh, has a lot of these similar kind of properties where, um, it's like, oh yeah, y- you can, you can learn something here. We can learn what... We can learn from what people do or we can, um, we can see if it's a success or a failure and then learn from that. Or, um, we can see what... We can optimize what works based on what doesn't work or what, uh, you know, figure out what's annoying and, and, and try to improve it. So I think these things all compound in some really interesting way. The goal, right, is, is, you know, straight out of sci-fi, right? You wanna make a thing where you say, "Hey, computer file my taxes." And it does the right thing, you know. I think we can get there, certainly in the next few years. Um, you know? But it's also fun to, like, think about how to break these things down. And, um, it turns out, turns out breaking down tasks in the same way that humans do... Take a complex task, you figure out... You know, you write a list of things to go through it. Same kind of thing works for AIs as well. So you break 'em... You break down complex tasks, figure out if there's any information you need, um, maybe you have to write some code, maybe you have to ask some questions, maybe you have to query some data. Um, same thing a human do.... humans don't usually write code to do their taxes, but sometimes it's kind of the same thing. You go through a list of your, uh, a, a list of your pay stubs. Um, you know, that's executing a for loop. Also, I like, I like data sets that don't exist, so, uh, people clicking on stuff on the web is a h- gigantic data set which is currently unowned. Uh, and so I think that's pretty exciting. I think there's a lot of stuff to learn from that.

Sarah Guo 37:08 So, you're hiring, and one thing we've talked about is, like, it's kind of a funny thing to try to hire for building products with this new set of technologies, in that, like, working in machine learning for the last decade may not help you that much. Um, how do you... W- how do you think about the people you need?

Alex Graveley 37:27 Yeah, it's been strange.

Sarah Guo 37:29 Okay.

Alex Graveley 37:30 (laughs) It's been, it's been o- it's been odd, actually. You know, o- one thing... I don't know if other people have noticed this, but it seems very bimodal. Like, very senior people get it, uh, and very young people get it. And so, I don't know that the middle has really caught up yet, or realizes that, that, um-

Sarah Guo 37:49 Are we on the other side? Are we in the middle? (laughs)

Alex Graveley 37:52 No, I just mean in the middle in terms of, like-

Sarah Guo 37:54 Oh, no, no. No.

Alex Graveley 37:54 Yeah, no, no, I just mean in the middle in terms of, like, uh, it's like-

Sarah Guo 37:58 I'm very young.

Alex Graveley 37:59 It's, like, super... No, no, it's not an age thing. It's not an age thing. It's more like an a- there's like, there's like, um, uh-

Elad Gil 38:04 It's kind of like a experiential Midway curve, you know, the Midway IQ thing?

Alex Graveley 38:08 Yeah, sure, sure. Yeah.

Elad Gil 38:09 It's, it's kind of like that for experience and some tests, yeah.

Alex Graveley 38:10 A little bit, yeah.

Elad Gil 38:11 Yeah.

Alex Graveley 38:11 It's like, it's almost like a... It's like you tell someone who's very naive, and you tell someone who's very experienced, like, "Uh, um, I can make magic. Here's how I can do it." And then the very naive person is like, "That's awesome." And then the very experienced person is like, "Ah, maybe."

Elad Gil 38:28 Yeah.

Alex Graveley 38:28 You know? Um, but the response from almost everybody else is, "That's not possible," or, "I don't see it."

Elad Gil 38:36 A number of companies right now are trying to build that intuition internally now, and often, this is the first time in a really long time that I've seen certain founders come back and start coding again. You know, they'll have a multi-hundred person company, and they'll get so enamored by what's happening that they'll just dive in. And it's almost... Uh, so a number of them have told me that they feel like the... some of their team members just don't have that intuition for what this can actually do, and so they don't even know what to build, or where to start, or how interesting or important it is. And so, it's almost like this founder mindset is needed, to your point, in terms of this is a really interesting new set of capabilities, and how to actually learn what these are, and then how do I apply them? And I think people lack natural intuition for what, what this does right now.

Alex Graveley 39:16 It's definitely not intuitive. Yeah. I mean, I, I would say I have some intuition now, and people that are, again, on that experience spectrum have some intuition. Um, but it's not intuitive. You know, it's, uh... There's many kinds of different learners, and there's many kinds of different programmers, right? Um, it takes a certain kind of programmer to be comfortable with this idea of, like, "I don't know what's gonna work. Let me try some stuff." Uh, um, it turns out it's, uh, it's similar, similar attributes to the web or product, product-making. Um, you know, being able to test stuff, look at the right metrics, find the right metrics. Um, uh, yeah, those are, those are useful skills, I think, that are reusable.

Elad Gil 39:59 Mm-hmm.

Alex Graveley 40:00 Um, but the, the intuition and the tenacity that it takes to, like, you know, "This doesn't work at all, what do I do?" Um, that's still rare, I think, in that field, 'cause there's st- there's so much uncertainty that it's, it's, it's easy to go, like, "Uh, I tried 10 things and it didn't work, so, like, this is impossible. Like, okay." Uh...

Sarah Guo 40:17 But I think the natural reaction of somebody looking at something that has 10% performance at the beginning is like, "Why even bother? We're not gonna get there." Right?

Alex Graveley 40:26 Yeah. The crazy thing is that these things work at all, right? Um, and not only that, but they scale, and they improve the scale, right? Most things break at scale. Almost everything breaks at scale. And that's what, you know, that's why you hire people to deal with. You know, every doubling, u- usually something breaks, you gotta fix it. Um, and, uh, it's a pretty crazy thing to think of what kind of emergent properties might still i- be out there. Um, you know, if we can get these things 10 times bigger. So I'm always, I'm always in favor of people pushing those limits. You know, like, it doesn't make sense. Like, even, even the best people can't explain what's gonna happen when you, you know, go bigger. But, like, that's... We built the Large Hadron Collider for that reason too, you know? We didn't know, "Oh, we think maybe we'll find some stuff." Um, these endeavors are valuable. And, uh, yeah, I'm, I'm grateful to be alive during, during this time, for sure.

Sarah Guo 41:26 (instrumental music) It's a great note to end on. Thank you for joining us for the podcast, Alex.

Elad Gil 41:30 Yeah, thanks so much.

Alex Graveley 41:31 Sure.