No PriorsNo Priors Ep. 10 | With Copilot's Chief Architect and founder of Minion.AI Alex Graveley
EVERY SPOKEN WORD
85 min read · 16,676 words- 0:00 – 1:50
Introduction
- SGSarah Guo
So let's start with some background to how did you end up working in tech for AI?
- AGAlex Graveley
Tech was earlier. I started, you know, really young and, uh, I got really into Linux when I was 14 or so. And, um, yeah, it was right around the time when the web was, like, a new thing, and you had to work to kind of get on the web. And then Linux is... I just sort of found it. It, um, popped into my life, and I really liked the... I read the GPL really early at 15, and it struck me. And, uh, just the idea of helping in the open and making it freely available, so other people could learn from things like I was learning from things seemed great. So I went and spent many years just working on open source stuff. And I don't know, I guess I didn't realize there was like... There was kind of a... There was, like, legacy technology, right? Like Windows was popular at that time. So people ran their, you know, data centers on Windows and networks on Windows and... Um, this Linux thing was, like, very strange and people didn't really know what it was. I spent many hours compiling kernels and, uh, hacking on stuff and, uh... Yeah.
- SGSarah Guo
And what was the thought process behind starting, like, Hackpad?
- AGAlex Graveley
Oh, Hackpad. Uh, yeah. So I had just finished like four or five years at VMware, and, uh, I wanted to get into startups. I knew- I knew that. And then, uh... So I left VMware and I started working on an, an education startup like many of us do. Uh, I don't know if you know this, but like many, many founders start with like the idea of an education startup.
- EGElad Gil
It's like a rite of passage. Yeah.
- AGAlex Graveley
It's like a rite of passage. Yeah. So I spent, I don't know, nine months working on- on- on- on that.
- SGSarah Guo
Most of us have done either that or consumer social
- 1:50 – 2:28
How Alex got started in technology
- SGSarah Guo
at the time.
- AGAlex Graveley
Sure. Yeah. And, uh, it's... Uh, it turns out education's very hard. Um, yeah. You nod your heads knowingly. Yeah. And, uh, so after nine months I was like, "All right, this isn't going anywhere. Uh, I- I know... I don't know if there's a value prop here." I mean, that's... That was the value, is that I learned that, like, you have to make something that is both achievable and that people want to pay for or spend their time on. So, yeah, then I was just kind of fishing around. Um, I was living in like a- a warehouse in San Francisco with a bunch of Burning Man people, and, uh, we were having trouble organizing burning... Large-scale Burning Man projects. And so,
- 2:28 – 7:32
Alex’s earlier projects with Hack Pad and Dropbox Paper
- AGAlex Graveley
uh, I forked Etherpad and started hacking on it. Uh, recruited a friend to... In the community to start working on it with me. And yeah, just grew from there. We had... Um, you know, before we did YC we had most... Many of the large Burning Man camps using it to organize their- their builds.
- SGSarah Guo
Can you describe the product experience?
- AGAlex Graveley
Oh, yeah. It was a... You know, it was a- a real-time text editor, uh, kind of like Google Docs. Um, Google Docs is the only other one that did that at that time. And, uh, it- it was kind of nice because, uh, it would highlight who said what, so you could go track down if somebody had a contribution and be like, "Oh, what did you mean?" Or, uh, "I heard you say something about this." So that's very useful in, like, um, large-scale anonymous, pseudo-anonymous groups where you don't really know who has ownership over what, like Burning Man camps. And let's see. And then we did YC, and then we got, uh, a bunch... Almost... Many of those people did it. It... My, like... My hack was to go do YC and try and get all the companies doing YC to use my product, which many of them did. I also took, like, very extensive notes of all the YC presentations using the product that everyone would then, like, look at. And, uh, we were able to get Stripe. And Stripe used us for many years actually. Um, they built for- for their... I think we were their first knowledge base. They used it for a long time. Um, a bunch of other big companies as well. Um, yeah. We were very lucky. Yeah.
- SGSarah Guo
And, um, post-Dropbox acquisition, you worked on what became Paper. How did you think about, like, what you wanted to go work on next?
- AGAlex Graveley
After Paper? I kind of did... You know, I spent the next few years kind of poking around at stuff. Uh, I knew that I wanted to make a robot that does stuff for you. Yeah. So there was this company called Magic. So I worked at this company called Magic. They were doing this, like, uh, text-based personal assistance.
- SGSarah Guo
Mm-hmm.
- AGAlex Graveley
Uh...
- SGSarah Guo
You remember this one?
- EGElad Gil
I- I- I think, uh, just like everybody who starts an education startup, Magic is one of those names that's cycling, and there's a really cool AI company right now called Magic as well.
- SGSarah Guo
Mm-hmm.
- EGElad Gil
So I feel like there's also these names that kind of persist from generation to generation, which I think is really cool.
- SGSarah Guo
I'm sure you know it's Codegen.
- AGAlex Graveley
Yeah, yeah, yeah. No, yeah. Uh, I haven't seen a demo yet, but it sounds like they're doing the right thing. Um, there's a few people doing the kind of whole repository, uh, uh, changes. So it seems like a great direction.
- SGSarah Guo
Uh, what- what's like one learning from... Um, you know, Magic was, like, a- a company early in trying to do seamless
- NANarrator
Yeah. Yes.
- SGSarah Guo
... sequence models and machine learning in general?
- AGAlex Graveley
Uh, no, no. It wasn't trying to do machine learning. It was all op space. It was super heavy op space. So, uh, you know, they had teams of people and it was 24 hours, and they would cycle in and- and, you know, they'd lose context and, like, they're all busy because they're trying to... They're trying to, like, deal with lots of people with lots of requests all the time. And so really it was like a crash course in, like, human behavior, right? Like what do people do under stress? How do they act? What do they say? What can you train? What can't you train? Like, um, uh, can you bucket stuff? And the answer is no. Like humans are complicated, especially in text form. So the beauty of the web and, like, traditional UIs, right, is that you fill them in. And like if it... If you can't... If it doesn't do what you want, then you make the decision to either go forward or not go forward, right? Text has this, uh, annoying property of you can be all the way at the 99% mark and then change the goal entirely, right? And so... Yeah. So it's- it's complicated. It's hard to... It's hard to understand what, uh...... what people want. It's hard to understand the complexity involved, especially when you're dealing with the real world. Um, flights get delayed. Passwords get lost.
- EGElad Gil
Do you think we're the last generation to deal with that? In other words, it feels like we're about to hit a transition point in which agents can actually start doing some of these things for us for real. When before, I think all these products really started with these operations-heavy approaches. I remember, you know, there was a really early sort of, um, personalized search engine called Aardvark, where similarly if you kind of looked behind the hood, it was a lot of ops people, and there was a little bit of algorithm, right? And so-
- AGAlex Graveley
That was a... Yeah. That was really with like, um... You would, like, sort of describe what you're good at or something, and then they would try and send questions to you, and you'd answer.
- EGElad Gil
Yeah. Yeah. Exactly. They'd kind of route things, but I think there was actually people doing some of the routing at least, or... I can't exactly remember, right? But I think it was... You know, I think a lot of people wanted to build these really complex sort of bots or agents that were doing really rich things, and the technology just wasn't there it feels like for a period of time. There was also some startups. I remember one company, um, that I got involved with that was, um, trying to do, like, virtual scheduling and assistance, you know?
- AGAlex Graveley
Mm-hmm.
- EGElad Gil
Six, seven, eight years ago. And again-
- AGAlex Graveley
Yeah.
- EGElad Gil
... it felt like it was a little bit early maybe.
- AGAlex Graveley
What were they called? Was that X dot... No, that wasn't X dot.
- EGElad Gil
Yeah. I don't know. There was, there was a couple of them, um-
- AGAlex Graveley
Clara Labs was one. Clara. That's the one. Yeah. Yeah, yeah, yeah. Yeah. I remember this era. And then we'd... Like, we had Operator. Do you remember Operator? I remember Operator. Yeah, yeah.
- EGElad Gil
There was like three or four of them.
- AGAlex Graveley
And then, like, on the question-answering stuff, we had Jelly. So there was a whole series of this
- EGElad Gil
Oh, right.
- 7:32 – 11:56
Why Alex always wanted to make bots that did stuff for people
- AGAlex Graveley
on, I wanted to work on the AI part. Uh, I think somewhere in there, M, Facebook M started as well. Mm-hmm. Uh, and it was just... It was a fun place to try and learn everything I could about solving those problems. Before Transformers, there was, uh, Sequence to Sequence was kind of the previous, uh, iteration. And, um, we were able to, like, take all the, all the histories of the chats between assistants and, and people and train a little model on it and, uh... "A little" by today's metrics. Uh, um, and run it, and it would like... It would show some gray text in the little text bar, and people could edit it and hit Enter and- Like the operators, like... Yeah, yeah. Right. The, the, the operators. And, um, yeah. We would measure how much time they sense, uh, they spent, uh, typing with it and with it out of it, and so it would save, like, half an hour across 100 people per day, uh, across an eight-hour shift. Uh, so yeah. That was, like, my first shipping AI product, I guess.
- EGElad Gil
That's-
- AGAlex Graveley
And then how'd you end up going to, um, Microsoft? Like, that's Oh, there was a bunch of other... Yeah. There was a bunch of other stuff along the way. There was... Um, so after that I did, uh, uh, I got into crypto. Uh, my friend was doing hCaptcha, which, uh, was like sort of a captcha marketplace, which is now, like, something like the number one or number two captcha service in the world, which is crazy. Yeah. So kind of launched that. That was fun. Uh, you know, annoyed people the world over for many-
- EGElad Gil
(laughs)
- AGAlex Graveley
... many man-hours in aggregate. And then worked on... Uh, left that to work with, uh, Moxie on, on a cryptocurrency for Signal. Um, so that was really fun. Um, complicated. And it all worked in a few seconds. Um, so, you know, we were sh- shooting for Venmo quality, which I think we pretty much hit. So.
- EGElad Gil
When you think about crypto in the context of AI... Because people talk about it in a few different contexts, right? One is, uh, you have programmatic, um, sort of money as code running. And so that could create all sorts of really interesting things from an agent-driven perspective. But then the other piece of it is identity. And some people think... I mean, Worldcoin would be one example, but there's other examples of effectively trying to secure identity cryptographically on the blockchain in an open way and then using that identity in the future to differentiate between AI-driven agents and people. Do you think that's gonna be important, or does that stuff not really matter in terms of the identity portion of not only crypto, but just, like, how we think about the future of agents?
- AGAlex Graveley
The honest answer is, I think we're gonna go through a many-year period of extreme discomfort, where AIs pretend to be things, or-
- EGElad Gil
Right.
- AGAlex Graveley
... uh, or confuse people, or extract money, money from your grandparents, or, um, you know, drain people's life savings in ways that are, uh, that are scary. And, you know, OpenAI is trying to do their best, but for some reason, the focus has been on OpenAI doing everything. And instead of, like, "We should go build the systems that prevent that. We should go, uh, pass the legislation that, that drops the hammer on people doing that stuff. We should go..." All this kind of stuff that is, um... Unfortunately, it seems like we're gonna need some really bad things to happen before, um, we align correctly.
- EGElad Gil
Mm-hmm.
- AGAlex Graveley
I don't... I'm not really scared about, uh, AIs killing us, although I'm very grateful that there are people that are thinking about it. Um, I'm more worried about bad people using new technology to hurt us.
- EGElad Gil
Yeah. Ilya from NEAR has some really interesting thoughts on this, 'cause he was one of the main author... Or he was the last author on the Transformer paper before he started NEAR.
- AGAlex Graveley
Mm-hmm.
- EGElad Gil
And he's brought up these concepts of like, how do you stress-test society relative to the coming wave of AI, which I think is an interesting concept.
- AGAlex Graveley
Yeah. It's a great, it's a great way to look at it. Like, yeah. It, it's not as bad as it could be, right? If you think about it, there, um... Most of the things that you want to spam, uh, are, uh, e- either have a spam blocker or are somewhat difficult to create an account on.
- EGElad Gil
Mm-hmm.
- AGAlex Graveley
So, um, you know, doing a better job of, uh, of, uh, sock puppet account filtering is gonna be really important going forward. Um, you know, I like what Cloudflare is doing with their kind of fingerprinting, um, instead of visual captchas, which are not good enough anymore. You know, one thing that is, like, kind of a saving grace, grace here is, is that, uh, you know,
- 11:56 – 27:11
How Alex started working at Github and Copilot
- AGAlex Graveley
many of the things that, uh, you would want to do cost money.
- EGElad Gil
Mm-hmm.
- AGAlex Graveley
So, calling everybody costs money.
- EGElad Gil
Sure.
- AGAlex Graveley
Um-... uh, texting everyone should be hopefully illegal soon, but also costs money. Maybe not enough money to, to prevent these things, but-
- SGSarah Guo
Probably not enough as agents can make money, right? I mean, just look at the trade-offs of cost.
- AGAlex Graveley
Yeah. I think it's interesting. Yeah, for sure. You know, the, the other thing, the other nice thing there is that, like, grandmas don't have crypto, right? They have bank accounts, and bank accounts can be traced. Um, I, I guess I would say it's not, you know, it's somewhere in the middle. It's like, uh, you know, it's imagine there's an, you know... North Korea has been trying to do this to us for a long time, right? Now there's a North Korea that has more resources or is more distributed or whatever. It's, you know, we have some mitigations. We need more. We need to be thinking about it a lot more.
- SGSarah Guo
Mm-hmm. How did you end up at GitHub, and how'd you end up working on Copilot?
- AGAlex Graveley
Oh, yeah. Okay, uh, let's see. So while I was working on MobileCoin, I, uh, my dad's kidneys failed, and I tried to donate them and, um, donate, donate a kidney. And, uh, they found a lump in my chest as part of the, the scans they do. And, uh, I had to have my right, most of my right lung removed in 2018. And so that was a big deal. Um, and so took some time, um, yeah. Healing, it's weird. Healing from internal injuries takes a lot longer than you, than you think. Anyway, happy story is that, uh, I, I don't have cancer now for, uh, over four years. And, um, my dad's kidneys got... He got a transplant also, so things are good. And so after, uh, oof, I don't know. I guess I was recovering for, for quite a while, and then I went and, uh, begged my friend for a job. I figured I should start working again, so I worked on some random stuff at first. Uh, uh, I converted GitHub to using their own product to build GitHub, which was kind of fun. They weren't... Yeah, it's an... I think people still use Codespaces now to build GitHub, which is pretty cool. Um, yeah. Then this kind of opportunity to work with OpenAI came up, and, uh, yeah. Because I had been tracking AI in the past, um, and was pretty aware of what was going on, I jumped on it.
- SGSarah Guo
Was that proposed by OpenAI or by GitHub, or who kind of initiated it all?
- AGAlex Graveley
So, I don't know the exact beginnings. I know that, uh, OpenAI and Microsoft were working on a deal for supercomputers. Um, so they wanted to build a big cluster for training, and there was a big deal that was being worked out. And there was some software kind of provisions thrown in. I think Office and Bing probably. And GitHub was like, "Oh, well, okay, well maybe we can... Let's like, uh, let's, um... May- maybe there's something GitHub can do here." Uh, I think OpenAI threw a small, threw a small fine tune over and was like, "Here's a, here's a small model trained on, um, on some code. See if this is interesting," you know? So we played around with it and-
- SGSarah Guo
What, what's small in that sense?
- AGAlex Graveley
I don't know. This was, uh... I have to remember now. I think this is before I knew very much, so it was definitely not a DaVinci-sized model. That's for sure. Um, I don't know if it... I don't know what size it was. Probably... Yeah, I don't know what size it was. Um, probably less than 10, but I can't remember. I, I learned later that this was basically a training artifact. So they had wanted to see what introducing code into their, uh, into their base models would do. Um, I think it had positive effects on chain of thought reasoning. Um, code is kind of linear, so, um, you can imagine that, you know, you kind of do stuff one after another and the things before, uh, have an impact.
- SGSarah Guo
Mm-hmm.
- AGAlex Graveley
And yeah, it was not that good. It was very bad. Uh, you know, they, it was, I think just like I said, just an artifact and just a small sample of, uh, GitHub data that they had crawled. And, um, yeah, we played around with it. I, this is before actually I, I joined. I was sort of, uh... Me and, uh, this guy Albert, uh, Ziegler were the first two after, uh, Uge, who, um, Uge got ahold of this model and started playing with it. He was able to say like, "Oh, well, you know, like, it doesn't work most of the time, but here it is doing something, you know. Here's, here's..." And it was only Python at that time. "Here it is, you know, generating something useful." We didn't really understand anything, you know. This just sort of lobbed over at us. So that was enough to like, "Okay, well, you know, go fetch a couple of people and start working. See what, see, see if there's anything there." We didn't really know what we had, so the first, you know, task was to go test it out, see what it did. Um, so we cr- we crowdsourced a bunch of, uh, Python problems in-house, stuff that we knew wouldn't be in the training set, and then we, we started work on, um, ph- fetching repositories and finding the tests in them so that we could, um, basically generate functions that were being tested and see if the tests still pass. There had been like a brand new, uh, PyTest feature introduced like recently that, that allowed you to see which functions were called by the PyTest. So you'll find that function, zero its body, ask the model to generate it, and then rerun the test and see if it passed. And, uh, I think it was less, I don't know, 10%, something like that, of, of those guys. And the dimensions are kind of like how many chances do you give it, um... How many chances do you give it to, to, to solve something? And, um, and then how do you test whether it's worked or not, right? So for the, the standalone tests that was w- we had people write test functions and then we would try to generate the body, and, uh, if the tests pass then you know it works. And for the in-the-wild, in-the-wild test harness we would download a repository, run all the tests, look at the ones that passed, find the functions that they called, make sure that they weren't trivial, generate the bodies for them, rerun the test, see what happens, see what... And then you get your percentage. And, uh, yeah. I mean, it was s- something like some very, very low percentage up front, but we knew that there was kind of a lot more juice to squeeze-So, like, getting, um, all of GitHub's code into the model, um, and then a bunch of other tricks that we hadn't, you know, we hadn't even thought of at that time. Eventually, you know, it, it went from, you know, less than 10% on, in the wild test to over 60%. So that's like one in two tests it can just generate code for, which is insane, right? Somewhere along the way, you know, there was like 10% to 20% to 35% to 45%. You know, these kind of like improvements along the way. Somewhere along the way, we did more prompting work so that the prompts got better. Um, somewhere along the way they used, you know, all the versions of the code as opposed to just the most recent version. They used diffs so that it could understand small changes, like, yeah, just it got better. And so but at the, when we first started, we were just, we didn't know what we had. We were just trying to figure it out. Um, and, uh, at the time they, they were thinking in terms of like, "Maybe you can replace stack overflow or something, you know, do a stack overflow competitor." Um...
- EGElad Gil
Was that the first, like, product idea-
- AGAlex Graveley
I think-
- EGElad Gil
... you guys had for it?
- AGAlex Graveley
I, I, I don't know that we had that idea. I think that was kind of like a...
- EGElad Gil
Yeah, on high.
- AGAlex Graveley
Yeah, yeah, yeah. There was more like a, um, it'd be nice, you know, it'd be nice if you could make something that competed with Stack Overflow because we have all this code, wouldn't it be nice to leverage it. Um, yeah. And so we made some UIs, but like early on, you know, it was like (laughs) y- early on it was bad. So it'd be like, you'd watch it and it'd, it would run and most of them would be bad and be like, there'd be like one success and you're like, "Oh, sweet, I got a success." But I had to wait, you know, some number of seconds for, for this test.
- EGElad Gil
Was the test user group just like the six of you, like some larger group?
- AGAlex Graveley
We made a little... yeah, the first iteration was just like an internal tool that people would, that helped people write these tests.
- EGElad Gil
Mm-hmm.
- AGAlex Graveley
And then, um, we wanted to see if maybe we could turn that into some UI that people would use where (laughs) if there was some way to cover up the fact that one in 10 things pass, right? So we tried a bu- we tried a few UI things there, and then, uh, it was actually OpenAI was like, "It would be nice if we could... we're testing these model fine tunes. It'd be nice if we could like test them more quickly." Can... what about doing like a VS Code extension? Um, just do autocomplete. So we did autocomplete at first, and that was kind of, that was a big jump, you know, because they, they were still thinking in terms of like stack overflow, you know, but this is, it's like, you know, I didn't have any ideas basically. I didn't know how to beat Stack Overflow with this thing. But we could come up with, we could play with some stuff in, in VS Code that was maybe closer to the code, you know? Um, and so we tried, uh, first we did autocomplete and that was kind of fun. It was useful, you know, um, it would, it would, it would show this little pop-up box like autocomplete does, and you could pick some strings. And so that format actually, you know, the, the usage was, you know, it was fine. Uh, it wasn't the right metaphor exactly, right? But you've got like this code generated mixed in with the, um, specific terms that are in the code, and it's a little... it's not exactly the same thing. Um, we tried things like, uh, uh, adding a little button over top of empty functions. So you would go generate them, or you could like hit a control key and it would create a big list on the side that you can choose from, or there's a little pop-up thing. So basically tried every single UI we could think of in VS Code, uh...
- EGElad Gil
And multiple generations, like the, the list, that didn't work.
- AGAlex Graveley
Yeah, none of, none of, none of them like really worked, like lists were like, you know, maybe you'd get one generation per person per, per day. And this is just a small sample, it's just like a few people that were interested in, in, at GitHub, um, language nerds or, um, people that have written tests for us, uh, and OpenAI people. So, so then, you know, I had some... very early on, I had this idea that it should work like a Gmail, uh, free text autocomplete, which was... I was like enamored with that product. It's like, it was the first largest language, um, you know, "large language", um, model deployment in the wild. Um, it was fast, it was cool, like the paper's great, like they give you all sorts of details on how they do it. All the, all the workarounds they had to do. So that was always in the back of my head, you know? And it, it was bad also, it was like, you know, those completions are not good. Um, but it seemed like the right, the right thing. Um, anyway, somewhere along the way after I, after we've tried all the UI, you know, sort of come up with some idea for... VS Code didn't support this so I tried to hack it in and finally came up with a way to hack it in and, uh, enough to make a demo, um, like a little demo video. And, uh-
- EGElad Gil
Was there support to like build real support for it within the organization?
- AGAlex Graveley
It's a little complicated. I, I guess, you know, we were pretty much a skunkworks project, so no one really knew, you know, no one knew about us. So we would go to like the... if you go to VS Code, people would be like, "Hey, we need you to go implement this very complex feature." Like, "I don't even know who you are. Like, what are you talking about?" Um, and, uh, yeah, there was, there was definitely some politicking that happened to, to get the VS Code people to, um, to dedicate some resources to that on a short, short time frame, like we were moving really fast. You know, it was less than a year before from beginning to ship public, public, public launch. Um...
- EGElad Gil
Was there a certain, um, metric where you're like, "This is good enough, like we need to actually put it in the public product?"
- 27:11 – 30:30
What is Minion AI
- AGAlex Graveley
you know, we, we hit, we made it as fast as we could and it was still improving on, um, you know, completions. But yeah, so that was like... And then that perpetuated like, "Okay. Okay. Uh, I know there's this plan for like Azure to run OpenAI in six months. We need you to do that in the next month. So like can we, let's figure out how to make this happen." So for, so 'cause we wanted to run, uh, a bunch of GPUs in Europe, so, so we could, um, hit Asia. You know, there was no, there was no other place that we could, we could run them in Europe, we could run them on the West Coast, we could run them in Texas, um, at the time. So yeah, w- so that was... And then, and Microsoft stepped up there, we got it running. Um, yeah. And then pretty much after that we launched. And yeah.
- EGElad Gil
Were you surprised by the uptake post-launch?
- AGAlex Graveley
No. Yeah. No, I mean, our retention rate was 50%. Like, it never went, like, months later it was still above 50% by like weekly cohort, which is like-
- EGElad Gil
Yeah.
- AGAlex Graveley
... insane.
- EGElad Gil
Yeah.
- AGAlex Graveley
Right? Um, and we didn't, we didn't know if people would pay for it. That was one thing. Um, I lobbied pretty hard for, uh, for going cheap and capturing the market.
- SGSarah Guo
How did you guys think about inference cost for this thing at the beginning?
- AGAlex Graveley
Oh yeah, we were... Our estimates were wildly off. Wildly off. Yeah, so we got estimates that were like, you know, it'll be 30 bucks a month for ev- for your a- on average, right? And then, um, we were able to like... Once we, once we were able to, once Microsoft was able to like do some, y- kind of do their Azure infrastructure, we were able to then like fork off little bits so we could do more accurate, uh, projections. And, uh, yeah, there was a bunch of like, there was a bunch of like moments where like, "How much is it gonna cost?" You'd be like, "Wait for these results." And, uh, the first big one was 10 bucks a month. It'll cost 10 bucks a month. And I was like, "Oh my god, so much cheaper than we expected." Um, and then, and then we could optimize it but now it's like we hadn't even optimized on price yet, right? And then we optimized on price a bunch and yeah, now it's less than that. So like, it was very fortuitous, right? Like, we were thinking like, "Okay, well maybe it's, maybe it's enterprise only because that's the only people who are gonna be willing to pay for this." Like, um, you know, 30 bucks a month is not... That's a, it's a lot. And that's like with no margins, right? But yeah, so-
- SGSarah Guo
For 40% of your code it's not a lot.
- AGAlex Graveley
Yeah, that's the thing. We know that, we know that now. And not only that, it's like, uh, the crazy thing is there's, uh, um, there's some whales out there. Like, there's people who it writes 80% of their code, um, which was insane. Uh-
- EGElad Gil
Where do you extrapolate all this like three to five years out? Are there basically gonna be just like agents writing code for us in certain ways? Is it 95% of code is written by copilots and, you know, humans are kind of directing it? Like, what, what do you kind of view the, the world evolving into in the next couple years? And next couple I mean like three to five, not 20.
- AGAlex Graveley
Yeah, it's hard. Uh, I don't know. (laughs)
- EGElad Gil
(laughs)
- AGAlex Graveley
I think it's hard for me to, you know, it- it- i- it's hard for me to imagine what that world
- 30:30 – 41:39
What’s possible on the horizon of AI
- AGAlex Graveley
looks like, 'cause it's such a shift from, like, I have my hands on something and I know that it's right to, um, where we are now, which is I mostly know what's right or I have a sense that it's right, but I have to, I have to test it and, and see it run, um, to know that it's right to then just write this and I'll, I'll trust that it's gonna work. Like, those are pretty crazy transitions, right? Um, they, they exist. Like...
- EGElad Gil
Sure. But you could also imagine, like, certain ways to do, like, a code review post, some chunk or some- some other sort of quality check to...
- AGAlex Graveley
Yeah. I think every... If that's, if that's the goal we wanna get to as a people, like, uh, I think every barrier to that is achievable.
- EGElad Gil
Mm-hmm.
- AGAlex Graveley
Um, so we can code review only the s- the dangerous parts-
- EGElad Gil
Yeah.
- AGAlex Graveley
... or only the scary or the confusing parts, or...
- EGElad Gil
Mm-hmm.
- AGAlex Graveley
Um, uh, we can do things like train a model on functions before and after changes-
- EGElad Gil
Yeah.
- AGAlex Graveley
... to say like, "Okay, this looks like your... Looks like a more polished version of this function would be this." Or...
- EGElad Gil
Mm-hmm.
- AGAlex Graveley
Um, yeah, we can do things like, you know, just start at the very basic, you know, very basic main loop and then add everything-
- EGElad Gil
Mm-hmm.
- AGAlex Graveley
... piece by piece, um, with tests so that it's, you know, what, what works and what doesn't, and then just have the, uh, the AI keep generating what's the logical...
- EGElad Gil
Yeah.
- AGAlex Graveley
... next feature. Like, um, all these things will get figured out. So if-
- EGElad Gil
Yeah.
- AGAlex Graveley
... if that's what we wanna do, that's what's gonna happen.
- EGElad Gil
So what's the, what's the idea behind Minion?
- AGAlex Graveley
Yeah. I think I mentioned, uh, making bots that do stuff for you. It's a, um, broad topic. And, um, I think that's where we see AI going. You know, the next few years are, are, um, in the AIs taking action, not just, uh, answering questions or, um, writing copy, but actually, um, helping us in our daily lives. Um, things like, um, organizing my schedule, or booking flights, or finding a trip for me to take, or, uh, doing my taxes, or, um, uh, telling me which contacts I haven't talked to in a long time and should reach out to, you know? Um, there's a lot of stuff that we can do by giving, uh, AIs access to information and letting them act on that information in a controlled way that, um, checks to make sure that we're, that we're aligned. And yeah, I think that'll be a really fun future. I think, you know, almost like you can imagine Copilot applied to everyday activities, right? Like, Copilot gives you a little bit of help. So I want Minion to give you a little bit of help, uh, outside of your code editor.
- EGElad Gil
How did you decide to, to work on Minion specifically?
- AGAlex Graveley
Minion specifically. So basically I s- I stopped working... At Magic I quit 'cause I, I couldn't figure out how to hook up the AI to data. I was like, "If in order to... In, in order to improve the quality I need a PhD in math, like, I don't know what to do now." And it just sort of... The models got better enough where that specific problem seemed solvable. Yeah, so the, the tech got better. And so that specific problem is where interacting with the real world broke down in the past. You know, like I said, flights get delayed, prices change. Um, uh, not just like a little bit, all the time, you know. Um, you know, they might not have the seat you want at the, at the concert that you wanna go to. Uh, and so, you know, AIs are this kind of compression of everything that you... In their training set, um, but they're not a real-time mechanism yet. Anyway, so that, that was the idea. I was like, "Okay, well, uh, I think we can, I think we can work on this old problem of how to make a, a bot do stuff for people." Like, yeah. That's what, that's what we want. Let's go make it. I don't know. It's like, maybe I can use the excitement from Copilot to launch into something which is incredibly hard, um, and, but which I believe the technology is around for.
- EGElad Gil
Mm-hmm.
- AGAlex Graveley
Um, if we can figure it out. So yeah, that's it. That- I-
- EGElad Gil
Yeah.
- AGAlex Graveley
That's the only reason I've ever gotten into startups actually, is like, uh, "Okay, I wanna do a startup so that I can do a harder startup."
- EGElad Gil
Yeah.
- AGAlex Graveley
You know? Or, "I wanna do a project so I can do a harder project." Um...
- EGElad Gil
Is this one sufficiently hard?
Episode duration: 41:40
Install uListen for AI-powered chat & search across the full episode — Get Full Transcript
Transcript of episode wdfuOA2PxJg
Get more out of YouTube videos.
High quality summaries for YouTube videos. Accurate transcripts to search & find moments. Powered by ChatGPT & Claude AI.
Add to Chrome